Data Engineering

Towards Effective DataOps

Paul Singman
May 5, 2022

Gain the confidence to mess with your datawithout making a mess of your data. “If it hurts, do it more often.” is a wise piece of advice that DevOps engineers…

Data Engineering

How to Level Up Your Data Lake Architecture

Paul Singman
February 22, 2022

What is the Basic Data Lake? A data lake is primarily two things: an object store and the objects being stored. It might look something like this: Even with this basic…

Data Engineering

How Easy It Is to Re-use Old Pandas Code in Spark 3.2?

Paul Singman
January 31, 2022

In October, it was announced that the Pandas API was being integrated with Spark. This was particularly exciting news for a Pandas-baby like myself, whose first exposure to data analytics…

The Everything Bagel II: Versioned Data Lake Tables with lakeFS and Trino

Paul Singman, Guy Hardonag
December 15, 2021

logo 10 Everything Bagel is now lakeFS-samples You can find Everything Bagel, along with lots of amazing hands-on examples for using lakeFS, in the lakeFS-samples repository. Let's Go! Introduction: Dockerize…

Data Engineering

Data Versioning Explained: Guide, Examples & Best Practices

Paul Singman
November 23, 2021

“I have never lied to you, I have always told you some version of the truth.” “The truth doesn’t have versions, okay?” — Something’s Gotta Give (2003) Jack Nicholson and Diane Keaton discuss data…

Data Engineering

Takeaways From the Future of Metadata After Hive Metastore Roundtable

Paul Singman
November 16, 2021

Overview of Hive’s Metastore Let’s get right into it. This is not an objective recap of every topic covered at the Future of Metadata After Hive Roundtable last week. But…

dbt Tests – Create Staging Environments for Flawless Data Write-Audit-Publish

Guy Hardonag, Paul Singman
November 3, 2021

Recently, we’ve heard from several community members experimenting with new development workflows using lakeFS and dbt. The timing isn’t surprising given dbt’s more recent support of big data compute tools…

Data Engineering

lakeFS Community Call Recap – Oct. 2021

Paul Singman
November 1, 2021

Last week we held another lakeFS Community Call! We believe these calls are invaluable opportunities to have direct dialogue with our users on all things lakeFS. Oz covered important new…

Data Engineering

3 Ways to Add Data to lakeFS

Paul Singman
October 26, 2021

Few people start using lakeFS without first having some data collected. Consequently, it is common that after getting it up and running, one of the first things people do is…

lakeFS – Data Versioning at Scale

Paul Singman
October 12, 2021

If you think about it, lakeFS is about two things — version control and big data. We see ourselves as bringing version control to big data. This bridges a workflow gap that…

Data Engineering

Thoughts on the Future of the Databricks Ecosystem

Paul Singman
September 8, 2021

Databricks has come a long way since growing out of a Berkeley Lab in 2013 with an open-source distributed computing framework called Spark. Fast forward eight years and in addition…

Data Engineering

The Docker Everything Bagel™ – Spin Up A Local Data Stack

Paul Singman
August 25, 2021

Pick up the Slack with lakeFS