Product

New in lakeFS Version 0.100.0

The lakeFS Team
May 10, 2023

lakeFS is celebrating its version release 0.100.0! We’re excited to announce the 0.100.0 version release! It’s thanks to you, the community that we’ve reached this milestone and: Managed to run…

Version Control Data Pipelines Using the Medallion Architecture

Iddo Avneri
May 8, 2023

A step by step guide to running pipelines on Bronze, Silver and Gold layers with lakeFS Introduction The Medallion Architecture is a software design pattern that organizes a data pipeline…

Applying Engineering Best Practices to Data Lakes

Einat Orr, PhD
May 3, 2023

In the last 30 years, agile development methodology played a significant part in the digital transformation the world is undergoing. What stands as the basis of the methodology is the…

Data Engineering

Open Table Formats vs. Data Version Control Systems

Iddo Avneri
April 25, 2023

I often get asked what is the difference between lakeFS and open table formats (OTF), namely Apache Iceberg, DeltaLake and Apache Hudi. The short answer is, “Those are different technologies…

Building an ML Experimentation Platform for Easy Reproducibility Using lakeFS

Vino SD
April 21, 2023

MLOps is mostly data engineering. As organizations ride past the hype cycle of MLOps, we realize there is significant overlap between MLOps and data engineering. As ML engineers, we spend…

Thought Leadership

MLOps Is Overfitting: Here’s Why

Einat Orr, PhD
April 5, 2023

VC surveys show that the MLOps category has significantly expanded in the past few years, with hundreds of companies defining themselves as part of this dynamically evolving niche. MLOps systems…

Data Engineering

Managing Structured and Unstructured Data – a Guide for an Effective Synergy

Michal Wosk
March 28, 2023

No time for the full article now? Read the abbreviated version here Many organizations and companies are rapidly moving from managing only structured data sets to managing both structured and…

Product

How Windward Leverages lakeFS for Resilient Data Ingestion

Lior Resisi
March 27, 2023

Implementing CI/CD-inspired workflows built atop lakeFS operations prevents inconsistent data and brings increased reliability to our analytics platform. This article was originally published in Lior’s Medium Blog. At Windward, our…

Security in lakeFS: Understanding Role-Based Access Control (RBAC)

Robin Moffatt
March 20, 2023

A discussion of how Role-Based Access Control works in lakeFS with code examples for three different types of user. …

Product

Improving Our Research Velocity With lakeFS

The lakeFS Team
March 20, 2023

This post was originally published in the Enigma blog. In every software engineering problem I’ve worked on, I’ve noticed a recurring tension between two highly desirable properties: flexibility and robustness….

Product

Testing Data Pipelines: Overview, Challenges & Importance

The lakeFS Team
March 15, 2023

What’s not to love about data pipeline testing? Adding user acceptance testing and contract acceptance testing to your data pipelines makes them less likely to cause errors and ensures that…

The lakeFS Blog