Data Engineering

Object Storage: Everything You Need to Know

Yael Rivkind
November 25, 2020

While Object Storage is not novel technology, it can still be overwhelming when getting started. Here’s a definitive guide to object-based storage with everything you need to know. What…

Data Engineering

Chaos Data Engineering

Oz Katz
November 17, 2020

Modern Data Lakes are a complexity tar pit. They involve many moving parts: distributed computation engines, running on virtualized servers connected by a software defined network, running on top of…

System Tests: Lessons Learned From Developing For OSS Project

Itai Admi
November 10, 2020

Overview In this article, I will try to cover some do’s and don’ts for system testing from the perspective of an open-source project. To keep things simple, it all boils…

Building A Data Development Environment with lakeFS

Barak Amar
October 27, 2020

Overview As part of our routine work with data we develop code, choose and upgrade compute infrastructure, and test new data. Usually, this requires running parts of our production pipelines…

Tutorials

The lakeFS Katacoda Sandbox Environment – Interactive Data Versioning Learning

Guy Hardonag
October 20, 2020

If you’re interested in playing around and exploring lakeFS, you can now easily get started using the Katacoda demo which provides a personalized sandboxed environment – all from your browser,…

Data Engineering

Introducing lakeview: A Visibility Tool for AWS S3 Based Data Lakes

Oz Katz
October 12, 2020

Lakeview is a new open source visibility tool for AWS S3 based data lakes. Think of it as ncdu, but for Petabyte-scale data. It’s goal is to provide you with…

How to Manage Your Data the Way You Manage Your Code

Einat Orr, PhD
October 5, 2020

50 years ago it was very hard to collaborate over code. When developing large scale software projects it was difficult to manage changes to source code over time, as revision…

Best Practices

Improving Postgres Performance Tenfold Using Go Concurrency

Tzahi Yaacobovicz
September 29, 2020

In this article I will show how Go concurrency enabled us to cut through a daunting DB performance barrier. This blog post continues our journey to big data performance. The…

In-process Caching In Go: Scaling lakeFS to 100k Requests/Second

Barak Amar
September 23, 2020

This is a first in a series of posts describing our journey of scaling lakeFS. In this post we describe how adding an in-process cache to our Go server speed…

Data Engineering

Diary of a Data Engineer

Oz Katz
September 14, 2020

A glimpse into the life of a data engineer. Day 1: Finally, an easy one Got a pretty simple task for a change – read a new type of event…

Data Engineering

How to Pick the Right Postgres for your Application

Ariel Shaqed (Scolnicov)
September 7, 2020

Lots of applications require a Postgres database. Before you can install them, you will need a Postgres database. How do you pick the right Postgres for your application? There are…

The lakeFS Blog