Building Reproducible Data Pipelines with Airflow and lakeFS
In this post, we’ll see how easy it is to use lakeFS with an existing Airflow DAG, to make every step in a pipeline completely reproducible in both code and data. This is done without modifying the actual code and logic of our jobs – by wrapping these operations with lakeFS commits. An example data …
Building Reproducible Data Pipelines with Airflow and lakeFS Read More »