Menu
Increase data quality and reduce the painful cost of errors
Data engineering best practices
using git-like operations on data
lakeFS is an open source data version control for data lakes.
It enables zero copy Dev \ Test isolated environments, continuous quality validation, atomic rollback on bad data, reproducibility, and more.
Trusted by
The cloud never warned us about the data getting clouded. As the blessing of infinite storage quickly became an unmanageable mess, there is a need for technologies like lakeFS to make data accessible again
Sivan Bercovici,
CTO
CTO
Since introducing lakeFS to our production data environment, we’ve enjoyed the benefits of atomic and isolated operations in our data pipelines. This has allowed us to spend more time improving other aspects of our data platform, and less time dealing with the fallout from race conditions and partially failed operations
Lior Resisi,
Data Platform Team Lead
Data Platform Team Lead
By using lakeFS we produce a commit history on the production branch that easily allows for rollbacks. In the case of data quality issues in production, this allows us to simply revert to the previous high quality snapshot of our data.
Doron Grinzaig
Big Data R&D Team Lead
Big Data R&D Team Lead
With lakeFS we can easily achieve advanced use cases with data, such as running parallel pipelines with different logic to experiment or conduct what-if analysis, compare large result sets for data science and machine learning, and more
Stephen Seewald,
Raghvendra Verma,
Cory Matheson
Raghvendra Verma,
Cory Matheson
Moving to a data branching solution has paid off quickly for us. A few days after completing the migration, we’ve already reduced testing time by 80% on two different projects. And we’re excited to see how data branching increases our product velocity.
Ryan Green,
CTO
CTO
Why do you need Data Versioning?
Data is dynamic, it changes over time. Dealing with that without a data version control system is error prone and labor intensive. With lakeFS, your data lake is version controlled and you can easily time-travel between consistent snapshots of the lake.
Develop on top of production data, in isolation, without copying anything
- Safely experiment and test on full production data
- Easily Collaborate on production data with your team
Promote only high quality data to production
- Automate data quality checks within data pipelines
- Bad data is not promoted, yet available in Isolation for debugging
Atomic rollback on bad data in production
- Rapidly recover from data quality issues in production by reverting the entire data lake and not just a specific table
- Troubleshoot on an isolated version of production data at the time of the failure
Data Version Control that works seamlessly with today’s data stack
lakeFS is fully compatible with a wide ecosystem of data engineering tools and technologies
Works seamlessly with today’s data stack
lakeFS is fully compatible with a wide ecosystem of data engineering tools and technologies
3000+
Installations
2.2K
GitHub Stars
1800+
Community members
Trusted by
Manage your data like code with data version control
Your data stays in place while lakeFS provides highly scalable, format agnostic and zero copy data version control over it
20%-80%
Storage Cost Reduction
X2
Double Data Engineering Efficiency
2 Seconds
Average time to rollback bad data
Stay updated
Best Practices, Data Engineering
Data Version Control – A Data Engineering Best Practice You Must Adopt
Imagine the software engineering world before distributed version control systems like Git became widespread. This is where the data world is currently...
Best Practices, Data Engineering
Git for Data – What, How and Why Now?
Git, the Source Control, aka Code Version Control When we wish for “Git for Data”, we already know what code version control...
Case Studies
Improving Our Research Velocity With lakeFS
Overview Data changes frequently, making the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one...
