Data Engineering

Analytical Data: Guide to Enterprise Data Architecture

Iddo Avneri
June 5, 2023

Organizations can accomplish more with their data than ever before thanks to advances in analytical data processing and data democratization initiatives led by the spread of visualization tools, low-code and…

Putting the Write-Audit-Publish Pattern into Practice with lakeFS

Robin Moffatt
June 1, 2023

Step-by-step tutorial on implementing Write-Audit-Publish with lakeFS, including support for multiple data formats….

Data Engineering

How to Implement Write-Audit-Publish (WAP)

Robin Moffatt
May 31, 2023

Comparison of tools and their suitability for Write-Audit-Publish including Apache Iceberg, Apache Hudi, Delta Lake, Project Nessie,and lakeFS…

Data Engineering Patterns: Write-Audit-Publish (WAP)

Robin Moffatt
May 30, 2023

This blog explains the concept of Write-Audit-Publish, which is a pattern in data engineering to enforce data quality in data pipelines….

What is OLTP? OLTP Database Examples, Key Features & Tools

Iddo Avneri
May 28, 2023

Data is a goldmine for every organization, no matter the industry. But to make the most of it, businesses need technology to maintain and manage transactional data like payments, inventory…

The State of Data Engineering 2023

Einat Orr, PhD
May 21, 2023

A lot has happened since 2022, from the rise of Generative AI to the economic slowdown and job losses impacting data practitioners in 2023. In fact, it’s safe to say…

15 Data Engineering Best Practices to Follow in 2025

Einat Orr, PhD
May 18, 2023

The software engineering world has profoundly transformed in the past decades. This was possible thanks to the emergence of methodologies and tools that helped establish and apply new successful engineering…

Backfilling Data: A Foolproof Guide to Managing Historical Data

Iddo Avneri
May 15, 2023

If you work with a smaller dataset or do one-off jobs, the way you manage backfills isn’t that crucial. But what if you face constantly growing datasets with billions to…

Product

New in lakeFS Version 0.100.0

The lakeFS Team
May 10, 2023

lakeFS is celebrating its version release 0.100.0! We’re excited to announce the 0.100.0 version release! It’s thanks to you, the community that we’ve reached this milestone and: Managed to run…

Version Control Data Pipelines Using the Medallion Architecture

Iddo Avneri
May 8, 2023

A step by step guide to running pipelines on Bronze, Silver and Gold layers with lakeFS Introduction The Medallion Architecture is a software design pattern that organizes a data pipeline…

Applying Engineering Best Practices to Data Lakes

Einat Orr, PhD
May 3, 2023

In the last 30 years, agile development methodology played a significant part in the digital transformation the world is undergoing. What stands as the basis of the methodology is the…

The lakeFS Blog