For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!
In the world of data management and data version control, understanding the relationships between different versions of your data is crucial. Just like in software development, where version control systems like Git help developers track changes in their codebase, data versioning tools such as lakeFS are indispensable for tracking changes in data lakes and object …
Commit Graph – A Data Version Control Visualization Read More »
In today’s data-driven world, managing vast amounts of data efficiently is crucial for organizations of all sizes. As data lakes and object storage systems popularity is on the rise, the need for robust data versioning and governance solutions has grown. Data retention and storage optimization has become an increasingly complex task, even more so when …
Data Garbage Collection: How lakeFS Keeps a Clean House (Lake) Read More »
There are several tools in the data version control space, all looking to solve similar problems. Two of the leaders are lakeFS and DVC. In this post, I am going to give an overview of how each has been designed so as to provide a basis for understanding their relative abilities to scale. Being a …
Scalable Data Version Control – Getting the Best of Both Worlds with lakeFS Read More »
Introduction “How do I integrate X with lakeFS” is an ever-green question on lakeFS Slack. lakeFS takes a “tooling-first” strategy to data management: it slots into your existing lineup of tools. So a significant part of our work on lakeFS is devoted to leveraging lakeFS and these other tools to improve these integrations. Our latest …
Dagster is a cloud-native data pipeline orchestration tool for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. It is designed for developing and maintaining data assets. With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at …
Dagster + lakeFS: How to Troubleshoot and Reproduce Data Read More »
A step by step guide to the lakeFS Cloud playground environment In this document, you will learn the quickest way to get started with lakeFS, utilizing the playground experience in lakeFS Cloud. Then I will cover how to connect your own storage to lakeFS, so you can run lakeFS against your own data. Step 1: …
Metadata Management for the Modern Data Lakehouse Efficient data management is a critical component of any modern organization. As data volumes grow and data sources become more diverse, the need for robust data catalog solutions becomes increasingly evident. Recognizing this need, lakeFS, an open-source data lake management platform, has recently integrated with Unity Data Catalog, …
lakeFS now directly supports Apache Iceberg tables. Using straightforward table identifiers you can switch between branches when reading and writing data: lakeFS itself remains format agnostic, happily providing version control for data whether binary, tabular, or just a lump of text. The benefit of this new functionality is that the same table that you create …
We are pleased to announce that lakeFS Cloud is now available as a self service on Azure. lakeFS Cloud is a fully-managed lakeFS platform, providing version control for your data lake. As well as being secure and scalable, it includes enterprise features such as Single Sign On (SSO), managed garbage collection, and role-based access control …
lakeFS Cloud is Now Self-Service on Microsoft Azure Read More »
If you’re deploying lakeFS on Azure you’ll be keen to ensure that you’re using the optimal set of components in doing so. One of the pieces that lakeFS needs to run is somewhere to store its metadata, and in the latest v0.103 release of lakeFS we now support CosmosDB for this. In this blog post, …
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
