Itai Admi

Integrations

Seamlessly Sync Data Into Your lakeFS Repos With Airbyte

Itai Admi
May 17, 2022

New features in Airbyte and lakeFS make it easy to send data replicated by Airbyte into a lakeFS repo. See how to leverage this integration in your data pipelines! If you work in data, chances are you rely on replicating data between different systems to centralize it for analysis. Modern companies produce data from all …

Seamlessly Sync Data Into Your lakeFS Repos With Airbyte Read More »

Data Engineering Integrations

Air & Water: The Airflow and lakeFS Integration

Itai Admi
May 27, 2021

Today we are excited to announce the official release of the lakeFS Airflow provider! What this package does is allow you to easily integrate lakeFS functionality to your Airflow DAGs. The library is published on PyPI so it can easily be installed in your project via the command: pip install airflow-provider-lakefs Once installed, you are …

Air & Water: The Airflow and lakeFS Integration Read More »

Project

Power Amazon EMR Applications with Git-like Operations Using lakeFS

Itai Admi
May 19, 2021

This article will provide a detailed explanation on how to use lakeFS with Amazon EMR. Today it’s common to manage a data lake using cloud object stores like AWS S3, Azure Blob Storage, or Google Cloud Storage as the underlying storage service. Each cloud provider offers a set of managed services to simplify the way …

Power Amazon EMR Applications with Git-like Operations Using lakeFS Read More »

Project

Tiers in the Cloud: How lakeFS caches immutable data on local-disk

Itai Admi
May 19, 2021

Introduction We recently released the first version of lakeFS supported by Pebble’s sstable library – RocksDB. The release introduced a new data model which is now much closer to Git. Instead of using a PostgreSQL server that quickly becomes a bottleneck, committed metadata now lives on the object store itself. Early on we realized that …

Tiers in the Cloud: How lakeFS caches immutable data on local-disk Read More »

Project

System Tests: Lessons Learned From Developing For OSS Project

Itai Admi
March 8, 2021

Overview In this article, I will try to cover some do’s and don’ts for system testing from the perspective of an open-source project. To keep things simple, it all boils down to running the system as our customers would: think of the different use-cases of your system, the environment where it runs, the configuration options, …

System Tests: Lessons Learned From Developing For OSS Project Read More »

LakeFS

  • Get Started
    Get Started
  • lakeFS Cloud is live!

    Read the announcement
    +

    lakeFS Cloud
    is live!

    annopp-img