Data Engineering

Data Engineering Go

lakeFS with DynamoDB – How Key Value Store is Used by lakeFS

Itai David
October 26, 2022

This blog discusses advanced topics within lakeFS. If you are new to lakeFS, or would like to expand your knowledge of how lakeFS works, make sure to check out our documents section. In the Beginning There Was Postgres Up until recently, lakeFS was using a strongly consistent SQL DB, namely PostgreSQL, where all metadata was …

lakeFS with DynamoDB – How Key Value Store is Used by lakeFS Read More »

Data Engineering Thought Leadership

Top Data Engineering Conferences & Events You Don’t Want to Miss in the Fall of 2022!

The lakeFS team
November 7, 2022

As the Covid-19 pandemic loosens its grip on the world, we’re all eager to start travelling and meeting in person again. The great news is that in persone conferences are back, so it’s time to make up for lost time! Attending conferences is a great way to learn, network, and engage with like-minded people.  Not …

Top Data Engineering Conferences & Events You Don’t Want to Miss in the Fall of 2022! Read More »

Data Engineering Thought Leadership

Data Mesh: What is it and What Does it Mean for Data Engineers?

The lakeFS team
November 7, 2022

Organizations have practically always needed data analytics, and they jumped on the analytics bandwagon as soon as the first computers appeared on the scene. In the 80s, businesses built data warehouses using relational databases as their decision-support systems (DSS). However, as companies generated more diverse data at high velocity, relational databases showed their limitations.  This …

Data Mesh: What is it and What Does it Mean for Data Engineers? Read More »

Data Engineering

Clearing the mess – How to ensure data quality with versioning

The lakeFS team
May 11, 2022

The last decade saw an unprecedented rise in the number of organizations that base their decisions and operations on data. The number of digital products that collect and process data and use it to fuel decision-making algorithms for enhancing future services is also growing at a very fast pace. That’s why data and data quality …

Clearing the mess – How to ensure data quality with versioning Read More »

Data Engineering

5 Painful mistakes data engineers make, and how to avoid them

The lakeFS team
June 6, 2022

Modern data engineering practices lead more and more organizations to a broader use of object stores. This happens due to the rising scale and complexity of the data that they manage – along with the growing variety of use cases that these data warehouses need to cater: from machine learning and algorithm development, to analytics …

5 Painful mistakes data engineers make, and how to avoid them Read More »

Data Engineering Hive Metastore

Takeaways From the Future of Metadata After Hive Metastore Roundtable

Paul Singman
May 11, 2022

Overview of Hive’s Metastore Let’s get right into it. This is not an objective recap of every topic covered at the Future of Metadata After Hive Roundtable last week. But it is a summary of what I found most interesting from the discussion between panelists Lior Ebel, Ryan Blue, Seshu Adunuthula and host Oz Katz. Watch the full talk below! …

Takeaways From the Future of Metadata After Hive Metastore Roundtable Read More »

Data Engineering

The Docker Everything Bagel™ – Spin Up A Local Data Stack

Paul Singman
May 24, 2022

Update Dec 16, 2021: Part II of the Everything Bagel series is published! Click here to read.  Introduction An important part of developing an open source project like lakeFS is assisting and advising our users. When they run into an issue and feel pain, we want to feel that pain, too. Quite literally. This means recreating …

The Docker Everything Bagel™ – Spin Up A Local Data Stack Read More »

LakeFS

  • Get Started
    Get Started
  • Git for Data - What, How and Why Now?

    Read the article
    +