Product

Building Reproducible Data Pipelines with Airflow and lakeFS

Guy Hardonag
February 3, 2021

Update (May 26th, 2021): We officially released the lakeFS Airflow provider. Read all about it in the latest blog post. In this post, we’ll see how easy it is to…

Working with Embed in Go 1.16 Version

Barak Amar
January 26, 2021

The new Golang v1.16 embed directive helps us keep a single binary and bundle out static content. This post will cover how to work with embed directive by applying it…

Data Engineering

Ensuring Data Quality in a Data Lake Environment

Einat Orr, PhD
January 19, 2021

The quality of the data we introduce determines the overall reliability of our data lake. And the ingestion stage is a critical point for ensuring the soundnes of our service…

Data Engineering

Getting Insights from Amazon S3 Inventory

Yoni Augarten
January 13, 2021

Lists are everywhere. And there’s a good reason for that: lists have an order to them. They reduce our stress by assuring us that everything is written down on paper…

Product

Git-like Operations Over MinIO with lakeFS

Yoni Augarten
January 5, 2021

lakeFS is an open source tool that delivers resilience and manageability to object-storage based data lakes. lakeFS provides Git-like operations over your MinIO storage environment and works seamlessly with all…

Why Data Versioning as an Infrastructure Matters

Einat Orr, PhD
December 15, 2020

The demand for infrastructure that contributes to the collection, storage, and analysis of data is growing with the increasing amounts of data managed by organizations. Every organization that manages data…

Loosely Coupled Monolith vs Tightly Coupled Microservices

Barak Amar
December 9, 2020

TL;DR With some thoughtful engineering, we can achieve a lot of the benefits that come with a microservice oriented architecture, while retaining the simplicity and low operating cost of being…

Data Mesh Applied: How to Move Beyond the Data Lake with lakeFS

Einat Orr, PhD
December 1, 2020

The data mesh paradigm The Data Mesh paradigm was first introduced by Zhamak Dehghani in her article How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. …

Data Engineering

Object Storage: Everything You Need to Know

Yael Rivkind
November 25, 2020

While Object Storage is not novel technology, it can still be overwhelming when getting started. Here’s a definitive guide to object-based storage with everything you need to know. What…

Data Engineering

Chaos Data Engineering

Oz Katz
November 17, 2020

Modern Data Lakes are a complexity tar pit. They involve many moving parts: distributed computation engines, running on virtualized servers connected by a software defined network, running on top of…

System Tests: Lessons Learned From Developing For OSS Project

Itai Admi
November 10, 2020

Overview In this article, I will try to cover some do’s and don’ts for system testing from the perspective of an open-source project. To keep things simple, it all boils…

The lakeFS Blog