lakeFS ❤️ DuckDB: Embedding an OLAP database in the lakeFS UI

Oz Katz
December 22, 2022

We’re happy to introduce experimental support for running SQL queries, directly on objects (your files!) in lakeFS. All through the lakeFS UI. No need to install or configure anything. TLDR…

Introducing Lua hooks for Write-Audit-Publish for Data

Oz Katz
December 20, 2022

TLDR – lakeFS hooks just became so much easier to use.As of v0.87.1, lakeFS includes a Lua Virtual Machine that allows you to run hooks directly in lakeFS, without relying…

Thought Leadership

4 New Year Resolutions for Data Engineering Leaders and How You Can Achieve Them

Michal Wosk
December 11, 2022

Introduction As 2022 is about to end, many engineering leaders use this time to reflect on the year that has passed and start planning ahead on the year upcoming. Data…

Troubleshoot and Reproduce Data with Apache Airflow

Iddo Avneri
December 6, 2022

Apache airflow enables you to build multistep workflows across multiple technologies. The programmatic approach, allowing you to schedule and monitor workflows, helps users build complicated ETLs on their data that…

Write-Audit-Publish for Data Pipelines: The Shortest Path to Your Destination with lakeFS

The lakeFS Team
December 4, 2022

Overview Continuous integration (CI) of data is the process of exposing data to consumers only after ensuring it adheres to best practices such as format, schema, and PII governance. Continuous…

Data Version Control – A Data Engineering Best Practice You Must Adopt

Einat Orr, PhD
December 1, 2022

Imagine the software engineering world before distributed version control systems like Git became widespread. This is where the data world is currently at. The explosion in the volume of generated…

Git for Data – What, How and Why Now?

Einat Orr, PhD
November 24, 2022

Git, the Source Control, aka Code Version Control When we wish for “Git for Data”, we already know what code version control is, and that Git is the standard tool…

Data Engineering

Data Reproducibility and other Data Lake Best Practices

The lakeFS Team
November 20, 2022

Overview Data changes frequently, making the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state. Data lake…

Machine Learning

lakeFS Community: Leonard Aukea nominated for Machine Learning Professional of the year!

Adi Polak
November 16, 2022

Our community is full of people with incredible skills and know-how. And this nomination proves us right! Our community member @Leonard Aukea has been nominated for Machine Learning Professional of…

Thought Leadership

What a Year With lakeFS Taught Me

Michal Wosk
November 10, 2022

A year ago, in November 2021, I took a bold decision to leave my role at Microsoft and join team lakeFS as VP Marketing. Upon joining I wrote an article…

4 Ways to Reduce Cloud Data Storage Costs

Oz Katz
November 4, 2022

In the past year, words like recession, business slowdown and monetary cuttings are being heard more and more often. Not just in the economic press and in the media, these…

The lakeFS Blog