Product

Air & Water: The Airflow and lakeFS Integration

Itai Admi
May 25, 2021

Today we are excited to announce the official release of the lakeFS Airflow provider! What this package does is allow you to easily integrate lakeFS functionality to your Airflow DAGs….

Data Engineering

Solving Data Reproducibility

Paul Singman
May 19, 2021

Debugging an issue is never fun, but why make it harder? In this post, we show how reproducing data is possible whether interacting with a single file, entire table, or data…

Go Versions: Manage Multiple Go Versions with Go

Barak Amar
May 12, 2021

Updated on April 5, 2022 As a user of the Go programming language, I’ve found it useful to enable the running multiple versions within a single project. If this is…

The State of Data Engineering in 2021

Einat Orr, PhD
May 5, 2021

Let’s start with the obvious: the lakeFS project doesn’t exist in isolation. It belongs to a larger ecosystem of data engineering tools and technologies adjacent and complementary to the problems…

Concrete Graveler: Splitting for Reuse

Ariel Shaqed (Scolnicov)
April 27, 2021

Welcome to another episode “Concrete Graveler”, our deep-dive into the implementation of Graveler, the committed object storage for lakeFS. Graveler is our versioned object store, inspired by Git. It is…

Data Engineering

Messing with AWS Endpoint URLs

Paul Singman
April 20, 2021

It makes perfect sense that if you type aws s3 ls s3://my-bucket to list the contents of an S3 bucket, you would expect to connect to the genuine bucket and…

Data Engineering

Hudi vs Iceberg vs Delta Lake: Data Lake Table Formats Compared

Oz Katz
April 12, 2021

Introduction to Data Lakehouse When building a data lake, there is perhaps no more consequential decision than the format data will be stored in. The outcome will have a direct…

Data Engineering

Why I’m Joining lakeFS

Paul Singman
April 6, 2021

Thoughts on a personal journey into the world of developer advocacy at an open-source data project. In March of 2021, I chose to leave the data team at Equinox Media and…

Data Engineering

3 Data Lake Anti-Patterns to Avoid

Paul Singman
March 30, 2021

Rid yourself of these troubling habits and start the journey towards data lake mastery! Introduction Data lakes offer tantalizing performance upside, which is a major reason for their high rate of…

Data Engineering

What is a Data Lake? Data Lake vs Data Warehouse

Paul Singman
March 22, 2021

What is a Data Lake? A data lake is a system of technologies that allow for the querying of data in file or blob objects. When employed effectively, they enable…

Power Amazon EMR Applications with Git-like Operations Using lakeFS

Itai Admi
March 9, 2021

This article will provide a detailed explanation of how to use lakeFS with Amazon EMR. Today, it’s common to manage a data lake using cloud object stores like AWS S3,…

The lakeFS Blog