Headless agents are coming for your data. Be ready with lakeFS
It usually starts the same way. A late night message pops up in the channel: “Which v5_final dataset did we use to train the model that just failed in production?” You sigh, open three S3 buckets, and begin the forensic treasure hunt.If that sounds familiar, you’ve already built a portion of an AI Factory. And […]
Since 2021, we’ve published the annual State of Data Engineering Report, which includes a summary of all key categories that directly impact data engineering infrastructure. In 2025, we see five primary trends that influence the categories that will be covered in this report. Trend #1: MLOps space is slowly diminishing The MLOps space is slowly
In today’s data-driven world, businesses don’t just rely on data – they are built on it. But as data infrastructure sprawls across on-prem systems, multiple cloud providers, and third-party platforms, a new challenge is taking center stage: distributed data management. It’s a silent bottleneck with loud consequences. Challenges in Distributed Data Management Managing data across
When I first joined lakeFS by Treeverse in 2020, we were just four engineers building an open-source solution for data versioning. It was exhilarating—being part of something from the ground up, shaping the product, and seeing it grow. But after four years, something changed. The excitement faded, and I felt like I was running in
Originally presented at Big Data LDN 2024. More than two decades ago, data warehouses outgrew the capacity of single machines, and scaling them started to become costly or inefficient. This prompted the tech industry to rethink the architecture and start to use distributed systems. If we wanted to store more data, we just bought more
Since 2021 we’ve been releasing the annual State of Data Engineering Report, a compilation of all the relevant categories that have a direct impact on data engineering infrastructure. In 2024, we see 3 primary trends that influence the categories which will be covered in this report. Trend #1: GenAI influence on software infrastructure As predicted
TL;DR: DataOps complexity arises from unclear R&Rs, a lack of standardization in interfaces, distributed technology complexities, and difficulties in implementing engineering best practices. The solution is to define clear responsibilities, address missing requirements, and manage data pipelines efficiently using emerging solutions that enhance the manageability and resilience of DataOps. What makes DataOps so hard is,
Git for data may sound odd at first. But using the Git logic and mechanisms for data lakes makes a lot of sense. After all, software developers figured out long ago how to efficiently collaborate on constantly changing source code with Git. We created lakeFS to provide similar capabilities to data folks. It provides data
This article was originally published on Datanami and is republished here with permission. As we officially kick off 2024, I realized I have a few thoughts on the direction of the data landscape that might be of interest to others. This is a recap of my “predictions.” I will admit that it’s a mix of what I
A data lake is often implemented using object storage, and our data resides in objects (files) that we can access through the storage API. Data professionals in analytics and ML are used to consuming data from a database, where there is an abstraction of a table one can access using SQL. They will find it
October 24, 2024 — TL;DR lakeFS 1.0 is generally available. You can upgrade to this version via the Assets here. Just over three years ago we publicly released the initial lakeFS. Our mission when we first launched lakeFS was to provide data practitioners with a scalable data version control system that would bring order to
This post is a recap of a talk I gave at this year’s Data + AI Summit about why I believe the Rust Programming Language and related novel technologies such as WebAssembly will play a large part in the data ecosystem in the coming years. The talk covers: The “present” of analytics (from my perspective,
We use cookies to improve your experience and understand how our site is used.
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
