Ready to dive into the lake?

lakeFS is currently only
available on desktop.

For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!

lakeFS Community

Subsurface – Rethinking Ingestion: CI/CD for Data Lakes

What we propose and will cover in this talk, is a new strategy for data lake ingestion. One where new data can be added in isolation, then tested and validated, before “going live” in a production table.

Subsurface – Rethinking Ingestion: CI/CD for Data Lakes

What we propose and will cover in this talk, is a new strategy for data lake ingestion. One where new data can be added in isolation, then tested and validated, before “going live” in a production table.

Description

At first glance, ingesting data into a data lake may seem like a one-step process — you simply add files to an object store. What else is there to do?!

It turns out that there is more you can do, and blindly writing new data introduces a host of potential problems. For example, how do you know the data you write is accurate and conforms to schema? The truth is, once you’ve written it to the lake, in a sense, it’s already too late.

What we propose and will cover in this talk, is a new strategy for data lake ingestion. One where new data can be added in isolation, then tested and validated, before “going live” in a production table. Finally, we’ll show how git-for-data tools like lakeFS and Nessie enable this ingestion paradigm in a seamless way.

Git for Data – lakeFS

  • Get Started
    Get Started
  • The annual State of Data Engineering Report is now available. Find out what’s new in 2023 -

    +