lakeFS Blog

Announcements

Audit Logs are Now Available in lakeFS Cloud

Adi Polak, Guy Hardonag
February 8, 2023

TL;DR lakeFS Cloud offers Audit Logs for compliance, operational stability,  monitoring access, activities and security analysis.  In the latest version of lakeFS Cloud, we introduced our new Audit Logs feature, providing detailed information on all user actions across all regions. Audit Logs – what is it and why should you care? Using the lakeFS Audit …

Audit Logs are Now Available in lakeFS Cloud Read More »

Use Cases

Data Lake Governance at Scale with lakeFS

Iddo Avneri
January 30, 2023

Introduction Often, data lake platforms lack simple ways to enforce data governance. This is especially challenging since data governance requirements are complicated to begin with, even without the added complexities of managing data in a data lake. Therefore, enforcing them is an expensive, time-consuming ongoing effort, requiring continuous management. Typically, at the expense of data …

Data Lake Governance at Scale with lakeFS Read More »

Best Practices Data Engineering

Big Data Testing: How To Test Data Pipelines In The ETL World

The lakeFS team
January 23, 2023

When testing ETLs for big data applications, data engineers usually face a challenge that originates in the very nature of data lakes. Since we’re writing or streaming huge volumes of data to a central location, it only makes sense to carry out data testing against equally massive amounts of data. You need to test with …

Big Data Testing: How To Test Data Pipelines In The ETL World Read More »

Data Engineering People

Data Engineering Conferences 2023

Ankit Srinivas
January 17, 2023

Conferences are back in full steam! 2023 is looking to be another great year for data conferences. This is a great time to learn, network, and engage with like-minded people.  Let’s kick off this list with some of the top Data Engineering Conferences that you will want to attend! Developer Week 2023 Website: https://www.developerweek.com/  When: …

Data Engineering Conferences 2023 Read More »

Integrations Tutorials

Databricks and lakeFS Integration: Step-by-Step Configuration Tutorial

Iddo Avneri
January 11, 2023

Introduction This tutorial will review all steps needed to configure lakeFS on Databricks.  This tutorial assumes that lakeFS is already set up and running against your storage (in this example AWS s3), and is focused on setting up the Databricks and lakeFS integration. Prerequisites Step 1 – Acquire lakeFS Key and Secret In this step, …

Databricks and lakeFS Integration: Step-by-Step Configuration Tutorial Read More »

Community People

Year in Review: Thanks A-LOTL for an Outstanding 2022

Adi Polak
January 4, 2023

2022 has been an incredible year, in the same way that roller coasters are thrilling. Our industry has seen many shifts and rapid changes – which we experienced together. In 2022, we witnessed how data engineering teams became increasingly central to any data-driven organization. And the growth of more significant roles of Analytics Engineers, ML …

Year in Review: Thanks A-LOTL for an Outstanding 2022 Read More »

Data Engineering

ETL Testing: A Practical Guide

Iddo Avneri
January 16, 2023

What is ETL Testing? ETL testing is the process of evaluating and verifying that the ETL (Extract, Transform, Load) processes work correctly.  What is ETL? An ETL process Extracts data of potentially many different structure or unstructured formats from multiple sources into a centralized repository. Then, an ETL process Transforms the data to a format …

ETL Testing: A Practical Guide Read More »

Git for Data – lakeFS

  • Get Started
    Get Started
  • LIVE: Develop Spark pipelines against production data on February 15 -

    Register Now
    +