For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!
What’s not to love about data pipeline testing? Adding acceptance tests to your data pipelines makes them less likely to make mistakes and makes sure that enough quality checks are done on the data before it is sent to end users. Testing data pipelines involves two components of any data pipeline: data and code used …
Acceptance Testing For Data Pipelines: Expert Guide Read More »
What is lakeFS? lakeFS is a platform that helps data engineers build scalable and resilient data lakes running on object storage. It provides version control, branching, and merging capabilities for data at petabyte scale, on or off premises. lakeFS enables teams to collaborate and manage data effectively by applying engineering best practices to data management. …
lakeFS Product Offerings Overview: Open Source vs. Enterprise vs. Cloud Read More »
Enterprises use more and more data as the foundation for their decisions and operations. The sheer number of digital goods that collect, analyze, and use data to feed decision-making algorithms in order to improve future services is also rapidly increasing. Because of this, data quality has become the most important asset for businesses in almost …
When testing ETLs for big data applications, data engineers usually face a challenge that originates in the very nature of data lakes. Since we’re writing or streaming huge volumes of data to a central location, it only makes sense to carry out data testing against equally massive amounts of data. You need to test with …
Big Data Testing: How To Test Data Pipelines In The ETL World Read More »
Overview Continuous integration (CI) of data is the process of exposing data to consumers only after ensuring it adheres to best practices such as format, schema, and PII governance. Continuous deployment (CD) of data ensures the quality of data at each step of a production pipeline. These approaches are commonly used by application developers of …
CI/CD for data pipelines – The Shortest Path to Your Destination with lakeFS Read More »
Overview Data changes frequently, making the task of keeping track of its exact state over time difficult. Oftentimes, people maintain only one state of their data––its current state. Data lake best practices require reproducibility that lets us time travel between different versions of the data, enabling us a snapshot at the data at different times …
Data Reproducibility and other Data Lake Best Practices Read More »
As the Covid-19 pandemic loosens its grip on the world, we’re all eager to start travelling and meeting in person again. The great news is that in persone conferences are back, so it’s time to make up for lost time! Attending conferences is a great way to learn, network, and engage with like-minded people. Not …
Top Data Engineering Conferences & Events You Don’t Want to Miss in the Fall of 2022! Read More »
Organizations have practically always needed data analytics, and they jumped on the analytics bandwagon as soon as the first computers appeared on the scene. In the 80s, businesses built data warehouses using relational databases as their decision-support systems (DSS). However, as companies generated more diverse data at high velocity, relational databases showed their limitations. This …
Data Mesh: What is it and What Does it Mean for Data Engineers? Read More »
Table of Contents Introduction Have you ever heard the phrase “data is the new oil” before? Of course you have. In this age, data is becoming a valued currency of its own, as more and more businesses take advantage of and derive value from data. Using data, a company can plan ahead and create strategies …
lakeFS Cloud provides a Git-like repository for data lakes in a hosted version available in AWS Marketplace NEW YORK and TEL AVIV, June 22, 2022–lakeFS, the technology that brings streamlined data lifecycle management and version control to data lakes, is announcing lakeFS Cloud, a fully managed SaaS version of its open-source technology. The new hosted …
LakeFS Introduces a New Approach to Data Manageability and Reliability Read More »
© Copyright 2022 | All Rights Reserved | Privacy Policy | Terms of Use
