For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!
A lot has happened since 2022, from the rise of Generative AI to the economic slowdown and job losses impacting data practitioners in 2023. In fact, it’s safe to say that the GenAI hype doesn’t seem to be high enough to counterbalance the influence of the struggling economy, as the demand for data practitioners is …
The world of software engineering underwent a huge acceleration in the past decades. This was possible thanks to the emergence of methodologies and tools that helped establish and apply new successful engineering best practices. The leading example is the move from a waterfall software development process into the concept of DevOps: The idea that at …
If you work with a smaller dataset or do one-off jobs, the way you manage backfills isn’t that crucial. But what if you face constantly growing datasets with billions to trillions of records? Your backfilling data strategy will have a much bigger impact. When dealing with modern data pipelines on such a scale, it’s key …
Backfilling Data: A Foolproof Guide to Managing Historical Data Read More »
In the last 30 years, agile development methodology played a significant part in the digital transformation the world is undergoing. What stands as the basis of the methodology is the ability to iterate fast on product features, using the shortest possible feedback loop from ideation to user feedback. This short feedback loop allows us to …
Applying Engineering Best Practices to Data Lakes Read More »
I often get asked what is the difference between lakeFS and open table formats (OTF), namely Apache Iceberg, DeltaLake and Apache Hudi. The short answer is, “Those are different technologies solving for different use cases.” But if this answer is so clear cut, why does the question keep coming up? It’s time for a detailed …
Open Table Formats vs. Data Version Control Systems Read More »
No time for the full article now? Read the abbreviated version here Many organizations and companies are rapidly moving from managing only structured data sets to managing both structured and unstructured data. This is due to the growth in the number of sources and data types, which are rooted in the new variety of use …
Managing Structured and Unstructured Data – a Guide for an Effective Synergy Read More »
A discussion of how Role-Based Access Control works in lakeFS with code examples for three different types of user.
When testing ETLs for big data applications, data engineers usually face a challenge that originates in the very nature of data lakes. Since we’re writing or streaming huge volumes of data to a central location, it only makes sense to carry out data testing against equally massive amounts of data. You need to test with …
Big Data Testing: How To Test Data Pipelines In The ETL World Read More »
Conferences are back in full steam! 2023 is looking to be another great year for data conferences. This is a great time to learn, network, and engage with like-minded people. Let’s kick off this list with some of the top Data Engineering Conferences that you will want to attend! Developer Week 2023 Website: https://www.developerweek.com/ When: …
What is ETL Testing? In a rush? Read the condensed version of this article here ETL testing is the process of evaluating and verifying that the ETL (Extract, Transform, Load) processes work correctly. What is ETL? An ETL process Extracts data of potentially many different structure or unstructured formats from multiple sources into a centralized …
© Copyright 2022 | All Rights Reserved | Privacy Policy | Terms of Use
The annual State of Data Engineering Report is now available. Find out what’s new in 2023 -
Get the report