dbt Tests – Create Staging Environments for Flawless Data Write-Audit-Publish

Guy Hardonag, Paul Singman
November 3, 2021

Recently, we’ve heard from several community members experimenting with new development workflows using lakeFS and dbt. The timing isn’t surprising given dbt’s more recent support of big data compute tools…

Data Engineering

lakeFS Community Call Recap – Oct. 2021

Paul Singman
November 1, 2021

Last week we held another lakeFS Community Call! We believe these calls are invaluable opportunities to have direct dialogue with our users on all things lakeFS. Oz covered important new…

Data Engineering

3 Ways to Add Data to lakeFS

Paul Singman
October 26, 2021

Few people start using lakeFS without first having some data collected. Consequently, it is common that after getting it up and running, one of the first things people do is…

Building Rich CLI Applications with Go’s Built-in Templating

Barak Amar
October 20, 2021

Overview The templating package text/template implements data-driven templates for generating textual output. Although we do not benefit from executing the template output more than once, we found it easy to…

lakeFS – Data Versioning at Scale

Paul Singman
October 12, 2021

If you think about it, lakeFS is about two things — version control and big data. We see ourselves as bringing version control to big data. This bridges a workflow gap that…

Data Engineering

Introduction to Building Data Products

Oz Katz
October 4, 2021

This is the first post of a series (well, at least 2) about Data Products. In Part I, we’ll cover what data products are, how they differ from traditional services…

Data Engineering

3 Strategies For Late-Arriving Data

Ariel Shaqed (Scolnicov)
September 25, 2021

This week I’m pleased to share “Tardy Data”, a chapter I wrote for the new book 97 Things Every Data Engineer Should Know, adapted to blog format. It’s about what…

Thought Leadership

lakeFS in Search of a Role Model

Einat Orr, PhD
September 20, 2021

Who needs a role model? When we first launched lakeFS in August of 2020, we asked ourselves a simple question: What does success look like? And how will we know…

Data Engineering

Thoughts on the Future of the Databricks Ecosystem

Paul Singman
September 8, 2021

Databricks has come a long way since growing out of a Berkeley Lab in 2013 with an open-source distributed computing framework called Spark. Fast forward eight years and in addition…

Product

Building an Auto-Upgrading lakeFS Environment

Nirav Adunuthula
September 1, 2021

One look at the cheerful aquamarine axolotl resting atop the lakeFS homepage was all it took to assure me that an internship here would make this summer different than any…

Data Engineering

The Docker Everything Bagel™ – Spin Up A Local Data Stack

Paul Singman
August 25, 2021

logo 10 Everything Bagel is now lakeFS-samples You can find Everything Bagel, along with lots of amazing hands-on examples for using lakeFS, in the lakeFS-samples repository. Let's Go! Introduction An…

The lakeFS Blog