TL;DR lakeFS Cloud offers Audit Logs for compliance, operational stability, monitoring access, activities and security analysis. In the latest version of lakeFS Cloud, we introduced our new Audit Logs feature, providing detailed information on all user actions across all regions. Audit Logs – what is it and why should you care? Using the lakeFS Audit …
Introduction Often, data lake platforms lack simple ways to enforce data governance. This is especially challenging since data governance requirements are complicated to begin with, even without the added complexities of managing data in a data lake. Therefore, enforcing them is an expensive, time-consuming ongoing effort, requiring continuous management. Typically, at the expense of data …
lakeFS enabled us to streamline and run 200+ dbt models in production, increase data deployment velocity, efficiently reproduce ML experiments, increase productivity of the data teams and adhere to FDA compliance requirements
When testing ETLs for big data applications, data engineers usually face a challenge that originates in the very nature of data lakes. Since we’re writing or streaming huge volumes of data to a central location, it only makes sense to carry out data testing against equally massive amounts of data. You need to test with …
Big Data Testing: How To Test Data Pipelines In The ETL World Read More »
Conferences are back in full steam! 2023 is looking to be another great year for data conferences. This is a great time to learn, network, and engage with like-minded people. Let’s kick off this list with some of the top Data Engineering Conferences that you will want to attend! Developer Week 2023 Website: https://www.developerweek.com/ When: …
Introduction This tutorial will review all steps needed to configure lakeFS on Databricks. This tutorial assumes that lakeFS is already set up and running against your storage (in this example AWS s3), and is focused on setting up the Databricks and lakeFS integration. Prerequisites Step 1 – Acquire lakeFS Key and Secret In this step, …
Databricks and lakeFS Integration: Step-by-Step Configuration Tutorial Read More »
2022 has been an incredible year, in the same way that roller coasters are thrilling. Our industry has seen many shifts and rapid changes – which we experienced together. In 2022, we witnessed how data engineering teams became increasingly central to any data-driven organization. And the growth of more significant roles of Analytics Engineers, ML …
Year in Review: Thanks A-LOTL for an Outstanding 2022 Read More »
lakeFS is gaining momentum as a solution for data versioning on top of an object store, and more and more data driven organizations adopt lakeFS as their data version control. Once you start using lakeFS, the files on your object store will form in a new structure. Other solutions, such as Iceberg, also create a …
What is ETL Testing? ETL testing is the process of evaluating and verifying that the ETL (Extract, Transform, Load) processes work correctly. What is ETL? An ETL process Extracts data of potentially many different structure or unstructured formats from multiple sources into a centralized repository. Then, an ETL process Transforms the data to a format …
As a new year is just around the corner, it is time to look ahead to the year that is coming and make some new year resolutions. It is a chance to adopt new habits that will make you more successful and more impactful, but it is also a great opportunity to stop some bad …
© Copyright 2022 | All Rights Reserved | Privacy Policy | Terms of Use
LIVE: Develop Spark pipelines against production data on February 15 -
Register Now