Data Engineering
Filter by
Write-Audit-Publish (continuous integration/continuous deployment of data) is the process of exposing data to consumers only after ensuring it adheres to best practices such as format,
- Oz Katz
Introduction In our recent version of lakeFS, we switched to base metadata storage on immutable files stored on S3 and other common object stores.
- Ariel Shaqed (Scolnicov)
Introduction We recently released the first version of lakeFS supported by Pebble’s sstable library – RocksDB. The release introduced a new data model which is
- Itai Admi
The quality of the data we introduce determines the overall reliability of our data lake. And the ingestion stage is a critical point for ensuring
- Einat Orr, PhD
Lists are everywhere. And there’s a good reason for that: lists have an order to them. They reduce our stress by assuring us that everything
- Yoni Augarten
The demand for infrastructure that contributes to the collection, storage, and analysis of data is growing with the increasing amounts of data managed by organizations.
- Einat Orr, PhD
TL;DR With some thoughtful engineering, we can achieve a lot of the benefits that come with a microservice oriented architecture, while retaining the simplicity and
- Barak Amar
The data mesh paradigm The Data Mesh paradigm was first introduced by Zhamak Dehghani in her article How to Move Beyond a Monolithic Data Lake
- Einat Orr, PhD
While Object Storage is not novel technology, it can still be overwhelming when getting started. Here’s a definitive guide to object-based storage with everything you
- Yael Rivkind
Modern Data Lakes are a complexity tar pit. They involve many moving parts: distributed computation engines, running on virtualized servers connected by a software defined
- Oz Katz
Overview In this article, I will try to cover some do’s and don’ts for system testing from the perspective of an open-source project. To keep
- Itai Admi
Overview As part of our routine work with data we develop code, choose and upgrade compute infrastructure, and test new data. Usually, this requires running
- Barak Amar
