Building A Data Development Environment with lakeFS

Barak Amar
October 27, 2020

Overview As part of our routine work with data we develop code, choose and upgrade compute infrastructure, and test new data. Usually, this requires running parts of our production pipelines…

Tutorials

The lakeFS Katacoda Sandbox Environment – Interactive Data Versioning Learning

Guy Hardonag
October 20, 2020

If you’re interested in playing around and exploring lakeFS, you can now easily get started using the Katacoda demo which provides a personalized sandboxed environment – all from your browser,…

Data Engineering

Introducing lakeview: A Visibility Tool for AWS S3 Based Data Lakes

Oz Katz
October 12, 2020

Lakeview is a new open source visibility tool for AWS S3 based data lakes. Think of it as ncdu, but for Petabyte-scale data. It’s goal is to provide you with…

How to Manage Your Data the Way You Manage Your Code

Einat Orr, PhD
October 5, 2020

50 years ago it was very hard to collaborate over code. When developing large scale software projects it was difficult to manage changes to source code over time, as revision…

Best Practices

Improving Postgres Performance Tenfold Using Go Concurrency

Tzahi Yaacobovicz
September 29, 2020

In this article I will show how Go concurrency enabled us to cut through a daunting DB performance barrier. This blog post continues our journey to big data performance. The…

In-process Caching In Go: Scaling lakeFS to 100k Requests/Second

Barak Amar
September 23, 2020

This is a first in a series of posts describing our journey of scaling lakeFS. In this post we describe how adding an in-process cache to our Go server speed…

Data Engineering

Diary of a Data Engineer

Oz Katz
September 14, 2020

A glimpse into the life of a data engineer. Day 1: Finally, an easy one Got a pretty simple task for a change – read a new type of event…

Data Engineering

How to Pick the Right Postgres for your Application

Ariel Shaqed (Scolnicov)
September 7, 2020

Lots of applications require a Postgres database. Before you can install them, you will need a Postgres database. How do you pick the right Postgres for your application? There are…

Product

The Quick Guide for Running Presto Locally on S3

Guy Hardonag
August 31, 2020

This post aims to cover our experience running Presto in a local environment with the ability to query Amazon S3 and other S3 Compatible Systems. We will: TL;DR: If you…

Data Engineering

Metadata Management: Hive Metastore vs AWS Glue

Yoni Augarten
August 24, 2020

Introducing the concept of metadata catalogs and explaining the benefits and pains of using Hive Metastore and AWS Glue …

From Zero to Versioned Data in Spark

Guy Hardonag
August 18, 2020

This tutorial aims to give you a fast start with lakeFS and use its git-like terminology in Spark. It covers the following: This simple flow gives a sneak peak to…

The lakeFS Blog