For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!
There are several tools in the data version control space, all looking to solve similar problems. Two of the leaders are lakeFS and DVC. In this post, I am going to give an overview of how each has been designed so as to provide a basis for understanding their relative abilities to scale. Being a …
Scalable Data Version Control – Getting the Best of Both Worlds with lakeFS Read More »
Introduction “How do I integrate X with lakeFS” is an ever-green question on lakeFS Slack. lakeFS takes a “tooling-first” strategy to data management: it slots into your existing lineup of tools. So a significant part of our work on lakeFS is devoted to leveraging lakeFS and these other tools to improve these integrations. Our latest …
Find out what are vector databases and why you need them as a data practitioner
How and why you should use lakeFS to provide data version control for your data lake[house] when working with R. Hands-on examples and code snippets.
Dagster is a cloud-native data pipeline orchestration tool for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability. It is designed for developing and maintaining data assets. With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at …
Dagster + lakeFS: How to Troubleshoot and Reproduce Data Read More »
In the traditional setup, organizations had a centralized infrastructure team responsible for managing data ownership across domains. But product-led companies started to approach this matter a little differently. Instead, they distribute the data ownership directly among producers (subject matter experts) using a data mesh architecture. This is a concept originally presented by Zhamak Dehghani in …
Data Mesh Architecture: Guide to Enterprise Data Architecture Read More »
If you’re deploying lakeFS on Azure you’ll be keen to ensure that you’re using the optimal set of components in doing so. One of the pieces that lakeFS needs to run is somewhere to store its metadata, and in the latest v0.103 release of lakeFS we now support CosmosDB for this. In this blog post, …
Guide to Enterprise Data Architecture Part 3 If you look at where companies keep their analytical data, you’ll quickly see that this space has split into two major architectural and technology stacks: data warehouses and data lakes. What are their defining characteristics? What factors should you take into account when choosing a data warehouse vs. …
Organizations can accomplish more with their data than ever before thanks to advances in analytical data processing and data democratization initiatives led by the spread of visualization tools, low-code and no-code solutions, and innovations like data mesh. Advances in compute power, innovative data processing methods, and broader cloud adoption have accelerated these trends, placing data …
Analytical Data: Guide to Enterprise Data Architecture Read More »
Step-by-step tutorial on implementing Write-Audit-Publish with lakeFS, including support for multiple data formats.
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
