Data Engineering

How to Level Up Your Data Lake Architecture

Paul Singman
February 22, 2022

What is the Basic Data Lake? A data lake is primarily two things: an object store and the objects being stored. It might look something like this: Even with this basic…

Data Engineering

How Easy It Is to Re-use Old Pandas Code in Spark 3.2?

Paul Singman
January 31, 2022

In October, it was announced that the Pandas API was being integrated with Spark. This was particularly exciting news for a Pandas-baby like myself, whose first exposure to data analytics…

Thought Leadership

In the Realm of the New Open Data Stack: Joining the lakeFS Adventure

Adi Polak
January 18, 2022

Trends in the data industry, how lakeFS fits into the new data stack, and a personal story of why I chose to join lakeFS. Open Source Is Everywhere! For the…

Data Engineering

lakeFS Community Analytics: 2021 Summary

Noy Davidson
January 6, 2022

The lakeFS open source project saw tremendous growth in 2021, our first full year of existence. We started the year with high expectations and many questions: Will we see engagement…

The Everything Bagel II: Versioned Data Lake Tables with lakeFS and Trino

Paul Singman, Guy Hardonag
December 15, 2021

logo 10 Everything Bagel is now lakeFS-samples You can find Everything Bagel, along with lots of amazing hands-on examples for using lakeFS, in the lakeFS-samples repository. Let's Go! Introduction: Dockerize…

Data Engineering

Data Versioning Explained: Guide, Examples & Best Practices

Paul Singman
November 23, 2021

“I have never lied to you, I have always told you some version of the truth.” “The truth doesn’t have versions, okay?” — Something’s Gotta Give (2003) Jack Nicholson and Diane Keaton discuss data…

Data Engineering

Data Versioning – Does It Mean What You Think It Means?

Einat Orr, PhD
November 22, 2021

Introduction When we first thought about a tagline for our open source project lakeFS, we instinctively gravitated to terms like “Data versioning”, “Manage data the way you manage code”, “Git…

Data Engineering

Takeaways From the Future of Metadata After Hive Metastore Roundtable

Paul Singman
November 16, 2021

Overview of Hive’s Metastore Let’s get right into it. This is not an objective recap of every topic covered at the Future of Metadata After Hive Roundtable last week. But…

dbt Tests – Create Staging Environments for Flawless Data Write-Audit-Publish

Guy Hardonag, Paul Singman
November 3, 2021

Recently, we’ve heard from several community members experimenting with new development workflows using lakeFS and dbt. The timing isn’t surprising given dbt’s more recent support of big data compute tools…

Data Engineering

lakeFS Community Call Recap – Oct. 2021

Paul Singman
November 1, 2021

Last week we held another lakeFS Community Call! We believe these calls are invaluable opportunities to have direct dialogue with our users on all things lakeFS. Oz covered important new…

Data Engineering

3 Ways to Add Data to lakeFS

Paul Singman
October 26, 2021

Few people start using lakeFS without first having some data collected. Consequently, it is common that after getting it up and running, one of the first things people do is…

The lakeFS Blog