What is lakeFS? As data practitioners, we use many different terms to talk about what we do – we call it business intelligence, analytics, data pipelines, or insights. But there’s one term that captures what we do really well: delivering products. When we were leading a large R&D organization, we couldn’t help but wonder about …
What if you could manage your data lake just like you manage code? With rollback, versioning, and branching capabilities on top of your existing data lake? lakeFS is an open-source project that provides a Git-like version control interface for data lakes, with seamless integration to most data tools and frameworks. lakeFS enables you to easily …
The lakeFS playground is now live and everybody can play! Read More »
This is the first post of a series (well, at least 2) about Data Products. In Part I, we’ll cover what data products are, how they differ from traditional services built by product teams, and conclude with a few requirements that help create successful data products. What Are Data Products? Data products are carefully curated …
This post is the second in a series about Hive Metastore. In the last post, Einat covered its history, the problems that it solves – and questioned whether or not it will survive the next revolution. For this post, I’d like to take a closer look at Hive Metastore at present time. Specifically how it …
The idea About two years ago, we left SimilarWeb, where we had the privilege of managing some of the most interesting and complex data architecture projects in the world. The architecture was centered around a Data Lake – 7 Petabytes of data on Amazon S3. On top, hundreds of hourly, daily and monthly jobs running, …
How to use lakeFS to keep multiple Delta tables consistent and guarantee data quality.
Introduction When building a data lake, there is perhaps no more consequential decision than the format data will be stored in. The outcome will have a direct effect on its performance, usability, and compatibility. It is inspiring that by simply changing the format data is stored in, we can unlock new functionality and improve the …
Hudi, Iceberg and Delta Lake: Data Lake Table Formats Compared Read More »
Continuous integration of data is the process of exposing data to consumers only after ensuring it adheres to best practices such as format, schema, and PII governance. Continuous deployment of data ensures the quality of data at each step of a production pipeline. In this blog, I will present lakeFS’s web hooks, and showcase a …
lakeFS Hooks: Implementing CI/CD for Data using Pre-merge Hooks Read More »
Modern Data Lakes are a complexity tar pit. They involve many moving parts: distributed computation engines, running on virtualized servers connected by a software defined network, running on top of distributed object stores, orchestrated by a distributed stream processor or pipeline execution engine. These moving parts fail. All the time. Handling these failures is not …
Lakeview is a new open source visibility tool for AWS S3 based data lakes. Think of it as ncdu, but for Petabyte-scale data. It’s goal is to provide you with an easy way to see the total size of your S3 bucket (prefix) storage. Instead of scanning billions of objects using the S3 API, which …
Introducing lakeview: A Visibility Tool for AWS S3 Based Data Lakes Read More »
© Copyright 2022 | All Rights Reserved | Privacy Policy | Terms of Use
Design by hello.
lakeFS Cloud is live!
Read the announcement
