Data Engineering

Open Table Formats vs. Data Version Control Systems

Iddo Avneri
April 25, 2023

I often get asked what is the difference between lakeFS and open table formats (OTF), namely Apache Iceberg, DeltaLake and Apache Hudi. The short answer is, “Those are different technologies…

Building an ML Experimentation Platform for Easy Reproducibility Using lakeFS

Vino SD
April 21, 2023

MLOps is mostly data engineering. As organizations ride past the hype cycle of MLOps, we realize there is significant overlap between MLOps and data engineering. As ML engineers, we spend…

Thought Leadership

MLOps Is Overfitting: Here’s Why

Einat Orr, PhD
April 5, 2023

VC surveys show that the MLOps category has significantly expanded in the past few years, with hundreds of companies defining themselves as part of this dynamically evolving niche. MLOps systems…

Data Engineering

Managing Structured and Unstructured Data – a Guide for an Effective Synergy

Michal Wosk
March 28, 2023

No time for the full article now? Read the abbreviated version here Many organizations and companies are rapidly moving from managing only structured data sets to managing both structured and…

Case Studies

How Windward Leverages lakeFS for Resilient Data Ingestion

Lior Resisi
March 27, 2023

Implementing CI/CD-inspired workflows built atop lakeFS operations prevents inconsistent data and brings increased reliability to our analytics platform. This article was originally published in Lior’s Medium Blog. At Windward, our…

Security in lakeFS: Understanding Role-Based Access Control (RBAC)

Robin Moffatt
March 20, 2023

A discussion of how Role-Based Access Control works in lakeFS with code examples for three different types of user. …

Case Studies

Improving Our Research Velocity With lakeFS

The lakeFS Team
March 20, 2023

This post was originally published in the Enigma blog. In every software engineering problem I’ve worked on, I’ve noticed a recurring tension between two highly desirable properties: flexibility and robustness….

Product

Testing Data Pipelines: Overview, Challenges & Importance

The lakeFS Team
March 15, 2023

What’s not to love about data pipeline testing? Adding user acceptance testing and contract acceptance testing to your data pipelines makes them less likely to cause errors and ensures that…

9 Best Practices For Handling Late-Arriving Data

Adi Polak
March 7, 2023

Processing what you’d call the “latest” data may sound simple, but in reality, it’s complex and challenging. When you gather time-based data, you’ll quickly notice that some are born late,…

Product

lakeFS Product Offerings Overview

The lakeFS Team
February 28, 2023

What is lakeFS? lakeFS is a platform that helps data engineers build scalable and resilient data lakes running on object storage. It provides version control, branching, and merging capabilities for…

Tutorials

Authorization (RBAC) in lakeFS: Step-by-Step Configuration Tutorial

Amit Kesarwani
February 22, 2023

Introduction Last month, the lakeFS team decided to move from the decoupled security authentication and access control features to enable you to plug your own authentication and security mechanism. Consequently,…

The lakeFS Blog