lakeFS Acquires DVC, Uniting Data Version Control Pioneers to Accelerate AI-Ready Data
Retrieval Augmented Generation (RAG) is on its way to becoming the dominant framework for implementing enterprise applications based on Large Language Models (LLMs). However, implementing RAG on your own is tricky. The framework calls for a high degree of knowledge and skill, as well as ongoing investment in DevOps and MLOps. Not to mention staying […]
Apache Iceberg is the most popular open table format. It originated at Netflix due to the need to provide a table representation for data saved in files and to enable teams to work with those tables as if they were managed in a relational database. In broad terms, Apache Iceberg is constructed of three main
Developing a machine learning application is a complex process that involves steps such as processing massive volumes of data, testing multiple ML models, parameter optimization, feature tuning, and others. This is why data version control is critical in the ML environment. If you want your experiments and data to be reproducible, you need to use
As OpenAI unveiled ChatGPT, which swiftly explained difficult problems, carved sonnets, and discovered errors in code, the usefulness and adaptability of LLMs became clear. Soon after, companies across various sectors began exploring new use cases, testing generative AI capabilities and solutions, and incorporating these LLM processes into their engineering environments. Whether it’s a chatbot, product
Metadata enforcement is a broad term that can refer to different aspects of managing and controlling metadata. Let’s explore few key areas: Understanding Metadata Enforcement 1 – Data Privacy and Protection: 2 – Data Governance and Quality: 3 – Legal and Compliance Challenges in Metadata Enforcement Strategies for Effective Metadata Enforcement We will focus on
Databricks Delta Lake includes a number of time travel features to let you access any previous version of the extensive data that Delta automatically versions and stores in your data lake. This makes it simple to audit, roll back data in the event of unintentional poor writes or deletes, and reproduce reports and trials. How
The Write-Audit-Publish (WAP) pattern in data engineering gives teams greater control over data quality. But what does it entail, and how do you implement it? Keep reading to learn more about the Write-Audit-Publish pattern, examine its use cases, and get a practical implementation example. What is Write-Audit-Publish all about? Write-Audit-Publish (WAP) aims to boost trust
When Databricks users first hear about lakeFS, a common response is, “I already have time travel in Delta Tables.” This raises an important question: how is lakeFS better, or how can it complement Delta Tables? Let’s explore the key differences and use cases where lakeFS shines, explaining why thousands of organizations, including many large enterprises,
Snowflake has many advantages, but its security and scalability are arguably the leading magnets for data practitioners. More and more businesses are migrating their data to Snowflake from big data systems like Teradata and Hadoop. A single Snowflake account can include up to ten databases, each with thousands of Views, Tables, and Columns. To address
The lakehouse architecture has become the backbone of modern big data operations, but it comes with specific issues. The challenge of data versioning arises in various DataOps areas, including: Fortunately, open-source tools can help overcome these issues. In this article, we’ll demonstrate how by implementing Git-like semantics, Delta Lake and lakeFS can work together to
This article is the continuation of Building A Management Layer For Your Data Lake: 3 Architecture Components. In this part, we explore open table formats, metastores, and data version control across three practical examples showing how to build a management layer for data lakes using tools in the Databricks, AWS, and Snowflake ecosystems. Databricks ecosystem
The growth in amounts of data was the catalyst for replacing traditional analytics databases with data lakes. While data lakes were able to handle large amounts of data, they did not provide us with all the capabilities of an analytics database… But we did not succumb to this tradeoff, and a set of technologies emerged
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
