Who needs a role model? When we first launched lakeFS in August of 2020, we asked ourselves a simple question: What does success look like? And how will we know we’re doing the right things to get there? Of course with thousands of installations, a thriving user community, active developer contributions, and exponential growth on …
Databricks has come a long way since growing out of a Berkeley Lab in 2013 with an open-source distributed computing framework called Spark. Fast forward eight years and in addition to the core Spark product, there are a dizzying number of new features in various stages of public preview within the Databricks platform. In case …
Thoughts on the Future of the Databricks Ecosystem Read More »
One look at the cheerful aquamarine axolotl resting atop the lakeFS homepage was all it took to assure me that an internship here would make this summer different than any other… This summer I had the opportunity to intern with the amazing developer team at Treeverse and work on the lakeFS project. This opportunity represents an important …
Introduction An important part of developing an open source project like lakeFS is assisting and advising our users. When they run into an issue and feel pain, we want to feel that pain, too. Quite literally. This means recreating the environment, running the same code, and raising the same error. In complex, modern data stacks …
The Docker Everything Bagel™ – Spin Up A Local Data Stack Read More »
This post is the second in a series about Hive Metastore. In the last post, Einat covered its history, the problems that it solves – and questioned whether or not it will survive the next revolution. For this post, I’d like to take a closer look at Hive Metastore at present time. Specifically how it …
Hive & Hadoop — A Brief History Apache Hive burst onto the scene in 2010 as a component of the Hadoop ecosystem, when Hadoop was the novel and innovative way of doing big data analytics. What Hive did was implement a SQL interface to Hadoop. Its architecture consisted of two main services: A Query Engine …
Hive Metastore – Why It’s Still Here and What Can Replace It? Read More »
New features in Airbyte and lakeFS make it easy to send data replicated by Airbyte into a lakeFS repo. See how to leverage this integration in your data pipelines! If you work in data, chances are you rely on replicating data between different systems to centralize it for analysis. Modern companies produce data from all …
Seamlessly Sync Data Into Your lakeFS Repos With Airbyte Read More »
The idea About two years ago, we left SimilarWeb, where we had the privilege of managing some of the most interesting and complex data architecture projects in the world. The architecture was centered around a Data Lake – 7 Petabytes of data on Amazon S3. On top, hundreds of hourly, daily and monthly jobs running, …
“I can remember everything. That’s my curse, young man. It’s the greatest curse that’s ever been inflicted on the human race: memory.” — Jedediah Leland, Citizen Kane (1941) lakeFS makes data corruptions easy to avoid and fix by allowing you to travel back in time to any state of your data. This new capability has …
What is Data Lifecycle Management Datasets are the foundational output of a data team. They do not appear out of thin air. No one has ever snapped their fingers and created an orders_history table. Instead, useful sets of data are created and maintained through a process that involves several predictable steps. Managing this process is …
Making Sure Your Data Lifecycle Management Makes Sense Read More »