Best Practices

What is Snowflake Data Catalog? Its Benefits & How to Set It Up

Idan Novogroder
August 8, 2024

Snowflake has many advantages, but its security and scalability are arguably the leading magnets for data practitioners. More and more businesses are migrating their data to Snowflake from big data…

Power Up Your Lakehouse with Git Semantics and Delta Lake

Oz Katz
August 5, 2024

The lakehouse architecture has become the backbone of modern big data operations, but it comes with specific issues. The challenge of data versioning arises in various DataOps areas, including: Fortunately,…

Product

Why We’re Moving ACLs Out of Core lakeFS

Oz Katz, Einat Orr, PhD
July 30, 2024

An upcoming release of lakeFS will no longer include user management with ACLs. The code currently used to manage users and groups will remain open source, but will move into…

Best Practices

Building A Management Layer For Your Data Lake: 3 Practical Examples with Databricks, AWS, and Snowflake

Einat Orr, PhD
July 30, 2024

This article is the continuation of Building A Management Layer For Your Data Lake: 3 Architecture Components. In this part, we explore open table formats, metastores, and data version control…

Best Practices

Building A Management Layer For Your Data Lake: 3 Architecture Components

Einat Orr, PhD
July 29, 2024

The growth in amounts of data was the catalyst for replacing traditional analytics databases with data lakes. While data lakes were able to handle large amounts of data, they did…

lakeFS Mount: Revolutionizing Data Access for Data Scientists and ML Practitioners

Oz Katz
July 25, 2024

We are excited to announce the launch of lakeFS Mount, a powerful new lakeFS client designed to simplify your data workflows. lakeFS Mount allows you to mount a lakeFS repository…

MLflow on Databricks: Benefits, Capabilities & Quick Tutorial

Amit Kesarwani
July 25, 2024

Machine learning teams face many hurdles, from data sources with missing values to experiment reproducibility issues. MLflow is a tool that makes this easier. And Databricks makes working with it…

RAG Pipeline: Example, Tools & How to Build It

Idan Novogroder
July 15, 2024

It may be tempting to think large language models (LLMs) can provide commercial value without any additional work, but this is a rare case. Businesses can make the most of…

Data Collaboration: What Is It And Why Do Teams Need It?

Tal Sofer
July 10, 2024

One common problem data teams face today is how to avoid stepping on teammates’ toes when working in data environments. Data assets are often handled like a shared folder that…

CI/CD Data Pipeline: Benefits, Challenges & Best Practices

Idan Novogroder
July 9, 2024

Continuous integration/continuous delivery (CI/CD) helps software developers adhere to security and consistency standards in their code while meeting business requirements. Today, CI/CD is also one of the data engineering best…

Unit Testing for Notebooks: Best Practices, Tools & Examples

Idan Novogroder
July 2, 2024

Quality can start from the moment you write your code in a notebook. Unit testing is a great approach to making the code in your notebooks more consistent and of…

The lakeFS Blog