For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!
In this article I will show how Go concurrency enabled us to cut through a daunting DB performance barrier. This blog post continues our journey to big data performance. The first post on this issue discussed in-process caching in Go. The Pain lakeFS is a versioned directory over objects stores like AWS S3 and GCS …
Improving Postgres Performance Tenfold Using Go Concurrency Read More »
This is a first in a series of posts describing our journey of scaling lakeFS. In this post we describe how adding an in-process cache to our Go server speed up our authorization flow. Background lakeFS is an open-source layer that delivers resilience and manageability to object-storage based data lakes. With lakeFS you can build …
In-process Caching In Go: Scaling lakeFS to 100k Requests/Second Read More »
A glimpse into the life of a data engineer. Day 1: Finally, an easy one Got a pretty simple task for a change – read a new type of event stream generated by sales, and publish it to the data lake. Sounds like a straightforward ETL. I estimate this as one day of work. I …
Lots of applications require a Postgres database. Before you can install them, you will need a Postgres database. How do you pick the right Postgres for your application? There are a bewildering variety of possible ways to acquire a database running on a Postgres instance, but the biggest choice is “build or buy”: whether to …
How to Pick the Right Postgres for your Application Read More »
This post aims to cover our experience running Presto in a local environment with the ability to query Amazon S3 and other S3 Compatible Systems. We will: Describe the components needed and how to configure them. Provide a dockerized environment you could run. Show an example of running the provided environment and querying a publicly …
The Quick Guide for Running Presto Locally on S3 Read More »
Introducing the concept of metadata catalogs and explaining the benefits and pains of using Hive Metastore and AWS Glue
This tutorial aims to give you a fast start with lakeFS and use it’s git-like terminology in spark. It covers the following: Quick start to install lakeFS using Docker Compose. How to create a repository, add files to it, create a branch and make changes to the repository using spark jobs. How to review changes …
lakeFS is an open source platform that delivers resilience and manageability to your existing object-storage based data lake. With lakeFS you can build repeatable, atomic and versioned data lake operations – from complex ETL jobs to data science and analytics.
© Copyright 2022 | All Rights Reserved | Privacy Policy | Terms of Use
