Barak Amar

Data Engineering Project

Building A Data Development Environment with lakeFS

Barak Amar
October 27, 2020

Overview As part of our routine work with data we develop code, choose and upgrade compute infrastructure, and test new data. Usually, this requires running parts of our production pipelines in parallel to production, testing the changes we wish to apply. Every data engineer knows that this convoluted process requires copying data, manually updating configuration, …

Building A Data Development Environment with lakeFS Read More »

Go Project

In-process caching in Go: scaling lakeFS to 100k requests/second

Barak Amar
October 5, 2020

This is a first in a series of posts describing our journey of scaling lakeFS. In this post we describe how adding an in-process cache to our Go server speed up our authorization flow. Background lakeFS is an open-source layer that delivers resilience and manageability to object-storage based data lakes. With lakeFS you can build …

In-process caching in Go: scaling lakeFS to 100k requests/second Read More »

LakeFS

  • Get Started
    Get Started