Databricks has come a long way since growing out of a Berkeley Lab in 2013 with an open-source distributed computing framework called Spark. Fast forward eight years and in addition to the core Spark product, there are a dizzying number of new features in various stages of public preview within the Databricks platform. In case …
Thoughts on the Future of the Databricks Ecosystem Read More »
Introduction An important part of developing an open source project like lakeFS is assisting and advising our users. When they run into an issue and feel pain, we want to feel that pain, too. Quite literally. This means recreating the environment, running the same code, and raising the same error. In complex, modern data stacks …
The Docker Everything Bagel™ – Spin Up A Local Data Stack Read More »
This post is the second in a series about Hive Metastore. In the last post, Einat covered its history, the problems that it solves – and questioned whether or not it will survive the next revolution. For this post, I’d like to take a closer look at Hive Metastore at present time. Specifically how it …
Hive & Hadoop — A Brief History Apache Hive burst onto the scene in 2010 as a component of the Hadoop ecosystem, when Hadoop was the novel and innovative way of doing big data analytics. What Hive did was implement a SQL interface to Hadoop. Its architecture consisted of two main services: A Query Engine …
Hive Metastore – Why It’s Still Here and What Can Replace It? Read More »
What is Data Lifecycle Management Datasets are the foundational output of a data team. They do not appear out of thin air. No one has ever snapped their fingers and created an orders_history table. Instead, useful sets of data are created and maintained through a process that involves several predictable steps. Managing this process is …
Making Sure Your Data Lifecycle Management Makes Sense Read More »
Combining lakeFS and Spark provides a new standard for scale and elasticity to distributed data pipelines. When integrating two technologies, the aim should be to expose the strengths of each as much as possible. With this philosophy in mind, we are excited to announce the beta release of the lakeFS FileSystem! This native Hadoop FileSystem …
Advancing lakeFS: Version Data At Scale With Spark Read More »
Data teams love calculating and tracking everything with metrics. We already have the infrastructure in place to do so… yet often fail to apply the same strategy for our own work. To fix that, let’s take a look at a few metrics useful for measuring our own performance. In my old office, there was a …
How to use lakeFS to keep multiple Delta tables consistent and guarantee data quality.
Today we are excited to announce the official release of the lakeFS Airflow provider! What this package does is allow you to easily integrate lakeFS functionality to your Airflow DAGs. The library is published on PyPI so it can easily be installed in your project via the command: pip install airflow-provider-lakefs Once installed, you are …
Debugging an issue is never fun, but why make it harder? In this post, we show how reproducing data is possible whether interacting with a single file, entire table, or data repository. Introducing Data Reproducibility There are two types of issues in the world — reproducible and unreproducible. A reproducible issue is one where the original conditions for …