How to Build an Isolated Testing Environment for Data with lakeFS

Barak Amar
October 27, 2022

Overview Our routine work with data includes developing code, choosing and upgrading compute infrastructure, and testing new and changed data pipelines. Usually, this requires running our tested pipelines in parallel…

How to Develop Spark ETL Pipelines in Isolation

Amit Kesarwani, Vino SD, Iddo Avneri
October 26, 2022

You’re bound to ask yourself this question at some point: Do I need to test the Spark ETLs I’m developing? The answer is yes; you certainly should – and not…

lakeFS with DynamoDB – How Key Value Store is Used by lakeFS

Itai David
September 20, 2022

This blog discusses advanced topics within lakeFS. If you are new to lakeFS, or would like to expand your knowledge of how lakeFS works, make sure to check out our…

EPCOR Implements Write-Audit-Publish Framework for Data Pipelines with lakeFS

Stephen Seewald, Raghvendra Verma, Cory Matheson
September 14, 2022

It is no secret that modern businesses run on big data. If your business was a car, big data would be the engine that powers it. All businesses want to…

Top Data Engineering Conferences & Events You Don’t Want to Miss in the Fall of 2022!

The lakeFS Team
August 23, 2022

As the Covid-19 pandemic loosens its grip on the world, we’re all eager to start travelling and meeting in person again. The great news is that in persone conferences are…

Data Manageability: The revolution that is turning Data Trust into the New North Star

Einat Orr, PhD
August 17, 2022

A few weeks ago, I was looking at a dashboard in our internal BI system. It’s a simple system. Redash over PostgreSQL that has just a few hundreds of thousands…

One Spark job, Many Data Sources – How to Easily Use lakeFS with Spark

Jonathan Rosenberg, Tal Sofer
August 15, 2022

lakeFS is an interface to the data lake, or the parts of the data lake one chooses to version control. The lakeFS interface is S3 compatible, and hence easily used…

Data Mesh: What is it and What Does it Mean for Data Engineers?

The lakeFS Team
August 10, 2022

Organizations have practically always needed data analytics, and they jumped on the analytics bandwagon as soon as the first computers appeared on the scene. In the 80s, businesses built data…

Case Studies

How Karius Used lakeFS to Meet FDA Compliance in Disease Diagnostic Research

Vino SD
August 10, 2022

A brief study, in collaboration with Karius team, on how lakeFS enabled Karius to run their experimental studies effectively. Karius is a Forbes AI-50 listed life sciences company based in California,…

Data Engineering

What is a Data Product? Examples & How to Build It

The lakeFS Team
August 1, 2022

Have you ever heard the phrase “data is the new oil” before? Of course, you have. As more businesses use and benefit from data, it becomes a valuable currency. Using…

Data versioning as your ‘Get out of jail’ card – DVC vs. Git-LFS vs. dolt vs. lakeFS

Einat Orr, PhD
July 25, 2022

Data Versioning at Scale: Solutions Overview Back when I was a 23-year-old student, I worked at an Israeli networking company as a BI analyst in the Operations department. My job…

The lakeFS Blog