Building Rich CLI Applications with Go’s Built-in Templating

Barak Amar
October 20, 2021

Overview The templating package text/template implements data-driven templates for generating textual output. Although we do not benefit from executing the template output more than once, we found it easy to…

lakeFS – Data Versioning at Scale

Paul Singman
October 12, 2021

If you think about it, lakeFS is about two things — version control and big data. We see ourselves as bringing version control to big data. This bridges a workflow gap that…

Data Engineering

Introduction to Building Data Products

Oz Katz
October 4, 2021

This is the first post of a series (well, at least 2) about Data Products. In Part I, we’ll cover what data products are, how they differ from traditional services…

Data Engineering

3 Strategies For Late-Arriving Data

Ariel Shaqed (Scolnicov)
September 25, 2021

This week I’m pleased to share “Tardy Data”, a chapter I wrote for the new book 97 Things Every Data Engineer Should Know, adapted to blog format. It’s about what…

Thought Leadership

lakeFS in Search of a Role Model

Einat Orr, PhD
September 20, 2021

Who needs a role model? When we first launched lakeFS in August of 2020, we asked ourselves a simple question: What does success look like? And how will we know…

Data Engineering

Thoughts on the Future of the Databricks Ecosystem

Paul Singman
September 8, 2021

Databricks has come a long way since growing out of a Berkeley Lab in 2013 with an open-source distributed computing framework called Spark. Fast forward eight years and in addition…

Product

Building an Auto-Upgrading lakeFS Environment

Nirav Adunuthula
September 1, 2021

One look at the cheerful aquamarine axolotl resting atop the lakeFS homepage was all it took to assure me that an internship here would make this summer different than any…

Data Engineering

The Docker Everything Bagel™ – Spin Up A Local Data Stack

Paul Singman
August 25, 2021

logo 10 Everything Bagel is now lakeFS-samples You can find Everything Bagel, along with lots of amazing hands-on examples for using lakeFS, in the lakeFS-samples repository. Let's Go! Introduction An…

Data Engineering

Hive Metastore – It Didn’t Age Well

Oz Katz
August 16, 2021

This post is the second in a series about Hive Metastore. In the last post, Einat covered its history, the problems that it solves – and questioned whether or not…

Data Engineering

Hive Metastore (HMS): What it is & What Can Replace it

Einat Orr, PhD
August 9, 2021

What is Hive Metastore? Hive Metastore (HMS) provides a single repository of metadata that you can quickly analyze to make educated, data-driven decisions. It’s an important component of many data…

Product

Seamlessly Sync Data Into Your lakeFS Repos With Airbyte

Itai Admi
August 5, 2021

New features in Airbyte and lakeFS make it easy to send data replicated by Airbyte into a lakeFS repo. See how to leverage this integration in your data pipelines! If…

The lakeFS Blog