lakeFS Acquires DVC, Uniting Data Version Control Pioneers to Accelerate AI-Ready Data
Since 2021 we’ve been releasing the annual State of Data Engineering Report, a compilation of all the relevant categories that have a direct impact on data engineering infrastructure. In 2024, we see 3 primary trends that influence the categories which will be covered in this report. Trend #1: GenAI influence on software infrastructure As predicted […]
TL;DR: DataOps complexity arises from unclear R&Rs, a lack of standardization in interfaces, distributed technology complexities, and difficulties in implementing engineering best practices. The solution is to define clear responsibilities, address missing requirements, and manage data pipelines efficiently using emerging solutions that enhance the manageability and resilience of DataOps. What makes DataOps so hard is,
Git for data may sound odd at first. But using the Git logic and mechanisms for data lakes makes a lot of sense. After all, software developers figured out long ago how to efficiently collaborate on constantly changing source code with Git. We created lakeFS to provide similar capabilities to data folks. It provides data
This article was originally published on Datanami and is republished here with permission. As we officially kick off 2024, I realized I have a few thoughts on the direction of the data landscape that might be of interest to others. This is a recap of my “predictions.” I will admit that it’s a mix of what I
A data lake is often implemented using object storage, and our data resides in objects (files) that we can access through the storage API. Data professionals in analytics and ML are used to consuming data from a database, where there is an abstraction of a table one can access using SQL. They will find it
October 24, 2024 — TL;DR lakeFS 1.0 is generally available. You can upgrade to this version via the Assets here. Just over three years ago we publicly released the initial lakeFS. Our mission when we first launched lakeFS was to provide data practitioners with a scalable data version control system that would bring order to
This post is a recap of a talk I gave at this year’s Data + AI Summit about why I believe the Rust Programming Language and related novel technologies such as WebAssembly will play a large part in the data ecosystem in the coming years. The talk covers: The “present” of analytics (from my perspective,
A lot has happened since 2022, from the rise of Generative AI to the economic slowdown and job losses impacting data practitioners in 2023. In fact, it’s safe to say that the GenAI hype doesn’t seem to be high enough to counterbalance the influence of the struggling economy, as the demand for data practitioners is
VC surveys show that the MLOps category has significantly expanded in the past few years, with hundreds of companies defining themselves as part of this dynamically evolving niche. MLOps systems provide the infrastructure allowing ML practitioners to manage the lifecycle of their work from development to production in a robust and reproducible manner. An MLOps
2022 has been an incredible year, in the same way that roller coasters are thrilling. Our industry has seen many shifts and rapid changes – which we experienced together. In 2022, we witnessed how data engineering teams became increasingly central to any data-driven organization. And the growth of more significant roles of Analytics Engineers, ML
As a new year is just around the corner, it is time to look ahead to the year that is coming and make some new year resolutions. It is a chance to adopt new habits that will make you more successful and more impactful, but it is also a great opportunity to stop some bad
Introduction As 2022 is about to end, many engineering leaders use this time to reflect on the year that has passed and start planning ahead on the year upcoming. Data engineering teams are usually swamped with tasks and requirements, which are many times also accompanied by failures and issues that require immediate attention. Therefore using
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
