For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!
Data is a goldmine for every organization, no matter the industry. But to make the most of it, businesses need technology to maintain and manage transactional data like payments, inventory updates, and customer records. This is where OLTP databases come in. Online Transaction Processing (OLTP) databases are used to store and process large numbers of …
OLTP: Guide to Enterprise Data Architecture Part 1 Read More »
A lot has happened since 2022, from the rise of Generative AI to the economic slowdown and job losses impacting data practitioners in 2023. In fact, it’s safe to say that the GenAI hype doesn’t seem to be high enough to counterbalance the influence of the struggling economy, as the demand for data practitioners is …
The world of software engineering underwent a huge acceleration in the past decades. This was possible thanks to the emergence of methodologies and tools that helped establish and apply new successful engineering best practices. The leading example is the move from a waterfall software development process into the concept of DevOps: The idea that at …
If you work with a smaller dataset or do one-off jobs, the way you manage backfills isn’t that crucial. But what if you face constantly growing datasets with billions to trillions of records? Your backfilling data strategy will have a much bigger impact. When dealing with modern data pipelines on such a scale, it’s key …
Backfilling Data: A Foolproof Guide to Managing Historical Data Read More »
lakeFS is celebrating its version release 0.100.0! We’re excited to announce the 0.100.0 version release! It’s thanks to you, the community that we’ve reached this milestone and: Managed to run 4,197 commits From 73 contributors, including (to name a few): Let’s celebrate this 0.100.0 release in a BIG way. Install the 100th lakeFS v0.100.0 and …
A step by step guide to running pipelines on Bronze, Silver and Gold layers with lakeFS Introduction The Medallion Architecture is a software design pattern that organizes a data pipeline into three distinct tiers based on functionality: bronze, silver, and gold. The bronze tier represents the core functionality of the system, while the silver and …
Version Control Data Pipelines Using the Medallion Architecture Read More »
In the last 30 years, agile development methodology played a significant part in the digital transformation the world is undergoing. What stands as the basis of the methodology is the ability to iterate fast on product features, using the shortest possible feedback loop from ideation to user feedback. This short feedback loop allows us to …
Applying Engineering Best Practices to Data Lakes Read More »
I often get asked what is the difference between lakeFS and open table formats (OTF), namely Apache Iceberg, DeltaLake and Apache Hudi. The short answer is, “Those are different technologies solving for different use cases.” But if this answer is so clear cut, why does the question keep coming up? It’s time for a detailed …
Open Table Formats vs. Data Version Control Systems Read More »
MLOps is mostly data engineering. As organizations ride past the hype cycle of MLOps, we realize there is significant overlap between MLOps and data engineering. As ML engineers, we spend most of our time collecting, verifying, pre-processing, and engineering features from data before we can even begin training models. Only 5% of developing and deploying …
Building an ML Experimentation Platform for Easy Reproducibility Using lakeFS Read More »
Delta Lake is one of the three new open-source table formats gaining wide adoption in the data engineering community, along with Apache Iceberg and Apache Hudi. In the recent release of lakeFS we’ve added support for comparing the state of Delta Lake tables so that you can see metadata for what has changed. Delta Diff …
Here’s Something Diff-erent: lakeFS adds support for diff of Delta tables Read More »
© Copyright 2022 | All Rights Reserved | Privacy Policy | Terms of Use
The annual State of Data Engineering Report is now available. Find out what’s new in 2023 -
Get the report