How Data Version Control Provides Data Lineage for Data Lakes

Iddo Avneri
September 4, 2023

One of the reasons behind the rise in data lakes’ adoption is their ability to handle massive amounts of data coming from diverse data sources, transform it at scale, and…

Commit Graph – A Data Version Control Visualization

Oz Katz
August 31, 2023

In the world of data management and data version control, understanding the relationships between different versions of your data is crucial. Just like in software development, where version control systems…

Product

Data Garbage Collection: How lakeFS Keeps a Clean House (Lake)

Nir Ozeri
August 28, 2023

In today’s data-driven world, managing vast amounts of data efficiently is crucial for organizations of all sizes. As data lakes and object storage systems popularity is on the rise, the…

Prefect + lakeFS: How to Troubleshoot Data Pipelines and Reproduce Data

Amit Kesarwani
August 24, 2023

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines. It’s the easiest way to transform any Python function into a unit of work…

Scalable Data Version Control – Getting the Best of Both Worlds with lakeFS

Oz Katz
August 18, 2023

There are several tools in the data version control space, all looking to solve similar problems. Two of the leaders are lakeFS and DVC. In this post, I am going…

Product

Mixing Metadata, Air and Water: Use the lakeFS Airflow Provider to Link Airflow Execution to lakeFS Data

Ariel Shaqed (Scolnicov)
August 14, 2023

Introduction “How do I integrate X with lakeFS” is an ever-green question on lakeFS Slack. lakeFS takes a “tooling-first” strategy to data management: it slots into your existing lineup of…

Metadata Management In Data Lakes

Einat Orr, PhD
August 3, 2023

4 Metadata Management Challenges And How To Solve Them Modern data architectures support an increasing number and variety of business use cases. Product creation, tailored customer experiences, fraud detection, regulatory…

Best 17 Vector Databases for 2026 [Top Picks]

Einat Orr, PhD
July 31, 2023

In the real world, teams may have to deal with data that isn’t neatly organized into rows and columns. That’s especially true when you’re working with complex, unstructured data such…

What is Metadata? Ultimate Guide For Data Engineers

Oz Katz
July 27, 2023

What is metadata? Why is it so important? Keep reading to learn more about modern practices in metadata management….

Delta-rs, Apache Arrow, Polars, WASM: Is Rust the Future of Analytics?

Oz Katz
July 25, 2023

This post is a recap of a talk I gave at this year’s Data + AI Summit about why I believe the Rust Programming Language and related novel technologies such…

What is a Vector Database? Top 12 Use Cases

Idan Novogroder
July 19, 2023

Find out what are vector databases and why you need them as a data practitioner…

The lakeFS Blog