Data Engineering

Data Engineering Go

lakeFS with DynamoDB – How Key Value Store is Used by lakeFS

Itai David
September 20, 2022

This blog discusses advanced topics within lakeFS. If you are new to lakeFS, or would like to expand your knowledge of how lakeFS works, make sure to check out our documents section. In the Beginning There Was Postgres Up until recently, lakeFS was using a strongly consistent SQL DB, namely PostgreSQL, where all metadata was …

lakeFS with DynamoDB – How Key Value Store is Used by lakeFS Read More »

Case Studies Data Engineering

How Epcor Built CI/CD for Data Pipelines

Stephen Seewald, Raghvendra Verma, Cory Matheson
September 14, 2022

It is no secret that modern businesses run on big data. If your business was a car, big data would be the engine that powers it. All businesses want to leverage their data to the hilt to make better-informed decisions that accelerate their success. But with the volume, velocity, and variety of data growing exponentially, …

How Epcor Built CI/CD for Data Pipelines Read More »

Data Engineering

Top Data Engineering Conferences & Events You Don’t Want to Miss in the Fall of 2022!

The lakeFS team
August 23, 2022

As the Covid-19 pandemic loosens its grip on the world, we’re all eager to start travelling and meeting in person again. The great news is that in persone conferences are back, so it’s time to make up for lost time! Attending conferences is a great way to learn, network, and engage with like-minded people.  Not …

Top Data Engineering Conferences & Events You Don’t Want to Miss in the Fall of 2022! Read More »

Data Engineering

Data Manageability: The revolution that is turning Data Trust into the New North Star

Einat Orr, PhD.
August 21, 2022

A few weeks ago, I was looking at a dashboard in our internal BI system. It’s a simple system. Redash over PostgreSQL that has just a few hundreds of thousands of rows.  I noticed a change in one of my favourite metrics that calculates the number of new installations since the beginning of the quarter. …

Data Manageability: The revolution that is turning Data Trust into the New North Star Read More »

Data Engineering Integrations

One Spark job, Many Data Sources – How to Easily Use lakeFS with Spark

Jonathan Rosenberg, Tal Sofer
August 15, 2022

lakeFS is an interface to the data lake, or the parts of the data lake one chooses to version control. The lakeFS interface is S3 compatible, and hence easily used with all common data applications, including Spark. In some cases, lakeFS is first adopted by the teams responsible for the data ingested to the lake, …

One Spark job, Many Data Sources – How to Easily Use lakeFS with Spark Read More »

Data Engineering

Data Mesh: What is it and What Does it Mean for Data Engineers?

The lakeFS team
August 14, 2022

Organizations have practically always needed data analytics, and they jumped on the analytics bandwagon as soon as the first computers appeared on the scene. In the 80s, businesses built data warehouses using relational databases as their decision-support systems (DSS). However, as companies generated more diverse data at high velocity, relational databases showed their limitations.  This …

Data Mesh: What is it and What Does it Mean for Data Engineers? Read More »

Data Engineering

Data versioning as your ‘Get out of jail’ card – DVC vs. Git-LFS vs. dolt vs. lakeFS

Einat Orr, PhD.
July 31, 2022

Data Versioning at Scale: Solutions Overview Back when I was a 23-year-old student, I worked at an Israeli networking company as a BI analyst in the Operations department. My job revolved around modeling the company’s inventory which was quite costly and needed optimization.  At some point, I attended a meeting of the company’s management. When …

Data versioning as your ‘Get out of jail’ card – DVC vs. Git-LFS vs. dolt vs. lakeFS Read More »

Data Engineering Machine Learning

Data+AI Summit 2022 Recap: Top 6 Industry trends and 9 major announcements!

Vino SD
July 25, 2022

It was 27th June 2022. San Francisco was bustling with 5000+ data folks from around the world to attend the Data & AI summit live after two years. Four days packed with tons of information from Keynotes, Speakers, Panels, Expo booths and Databricks trainings. Flurry of new product announcements followed. lakeFS cloud launch, Delta lake …

Data+AI Summit 2022 Recap: Top 6 Industry trends and 9 major announcements! Read More »

Data Engineering

Proudly announcing lakeFS Cloud

Einat Orr, PhD., Oz Katz
June 27, 2022

What is lakeFS? As data practitioners, we use many different terms to talk about what we do – we call it business intelligence, analytics, data pipelines, or insights. But there’s one term that captures what we do really well: delivering products.  When we were leading a large R&D organization, we couldn’t help but wonder about …

Proudly announcing lakeFS Cloud Read More »

LakeFS

  • Get Started
    Get Started
  • Join our live webinar on October 12th:

    Troubleshoot and Reproduce Data with Apache Airflow
    +