Data Engineering
Filter by
Key Takeaways The software engineering world has profoundly transformed in the past decades. This was possible thanks to the emergence of methodologies and tools that
- Einat Orr, PhD
If you work with a smaller dataset or do one-off jobs, the way you manage backfills isn’t that crucial. But what if you face constantly
- Iddo Avneri
In the last 30 years, agile development methodology played a significant part in the digital transformation the world is undergoing. What stands as the basis
- Einat Orr, PhD
I often get asked what is the difference between lakeFS and open table formats (OTF), namely Apache Iceberg, DeltaLake and Apache Hudi. The short answer
- Iddo Avneri
No time for the full article now? Read the abbreviated version here Many organizations and companies are rapidly moving from managing only structured data sets
- Michal Wosk
Processing what you’d call the “latest” data may sound simple, but in reality, it’s complex and challenging. When you gather time-based data, you’ll quickly notice
- Adi Polak
When testing ETLs for big data applications, data engineers usually face a challenge that originates in the very nature of data lakes. Since we’re writing
- The lakeFS Team
Conferences are back in full steam! 2023 is looking to be another great year for data conferences. This is a great time to learn, network,
- Ankit Srinivas
2022 has been an incredible year, in the same way that roller coasters are thrilling. Our industry has seen many shifts and rapid changes –
- Adi Polak
Imagine spending significant time and money migrating your data to a new system only to discover that the data is partial or incorrect. Not only
- Iddo Avneri
As a new year is just around the corner, it is time to look ahead to the year that is coming and make some new
- Michal Wosk
Overview Continuous integration (CI) of data is the process of exposing data to consumers only after ensuring it adheres to best practices such as format,
- The lakeFS Team
