Effective AI Metadata Management with lakeFS

In the landscape of ML and AI, metadata is essential for building accurate, trustworthy models. By providing context around data, metadata supports efficient data discovery, tracking, and validation, which are crucial for creating reproducible and reliable models. As models grow more complex, so does the volume and complexity of metadata, making robust metadata management essential.

This post demonstrates how lakeFS, a scalable data version control system, tackles key metadata management challenges by providing a unified approach to managing both data and metadata. We’ll explore specific use cases where lakeFS stands out compared to traditional AI metadata management platforms.

The Role of Metadata in ML & AI 

Metadata is crucial in ML and AI because it adds context and structure to data, making it easier to build, manage, and optimize models effectively. Here’s how metadata supports key tasks in ML development:

Data Discovery 

Simplifies finding, understanding, and organizing data by answering key questions on available datasets, recent assets, and processing history.

Data Selection 

Enables quick, metadata-based queries to retrieve relevant training data, reducing time spent on manual filtering.

Enhancing Data Quality and Context 

Adds contextual information (e.g., location, language, source), helping models interpret data more accurately. This is especially valuable for GenAI tasks, where metadata enables a model to differentiate, for instance, between customer support conversations and product reviews, leading to more tailored responses and improved recommendations.

Feature and Prompt Engineering

Supports the creation of relevant features and effective prompts by identifying data types, relationships, and quality indicators, aligning both features and prompts with model objectives.

Lineage Tracking 

Tracks data and model lineage – origins, transformations, and dependencies – answering questions about the model-building process (e.g., who created the model, which datasets were used). This transparency enables experiment results comparison, identifying the best model, and rolling back when needed.

Regulatory Compliance 

Maintains data history and audit trails, reducing the risk of non-compliance and legal issues.

Data Management and Governance

Metadata facilitates data governance by defining attributes such as data ownership, retention policies, alignment with specific projects or budgets, and access control. It enables management of who can access or modify data with RBAC, enhancing data security. Additionally, metadata supports tagging sensitive information, like PII, which helps organizations comply with data privacy regulations and maintain strict adherence to organizational policies

Challenges of AI Metadata Management

As ML and AI systems grow more complex, managing metadata becomes increasingly challenging. Key issues include:

• Volume and Diversity: The large volume of metadata from diverse data sources makes tracking and organization difficult.
• Lack of Standardization: Inconsistent metadata structures complicate integration and querying across sources.
• Metadata Versioning: Versioning metadata is essential for reproducibility and troubleshooting, but maintaining accurate versions is challenging.
• Quality Control: Ensuring metadata accuracy is critical, as poor metadata quality can lead to poorly trained models.
• Scalable Querying and Filtering: Querying and filtering metadata at scale is difficult, especially in large data lakes where manual efforts are impractical.

How lakeFS Solves AI Metadata Management

lakeFS is a scalable data version control system letting you manage data like code. lakeFS transforms your object store into a git-like repository enabling versioning operations such as creating branches, doing commits and merging for data. lakeFS unlocks data collaboration, safe experimentation, troubleshooting, and instant error recovery and provides an easy way to implement the Write-Audit-Publish pattern for your data lake. Using lakeFS for metadata management applies the same best practices applied to data but to metadata, offering similar benefits.

Before exploring lakeFS’s AI metadata management capabilities, it’s essential to understand its approach to data and metadata. Unlike traditional metadata platforms that support various entity types (e.g., tables, views, streams, document collections, dashboards), lakeFS manages large-scale data lakes at the object level, supporting object-level metadata. This focus on objects aligns with the structure of data lakes, where datasets are stored as collections of objects, making lakeFS format-agnostic and adaptable to any dataset structure.

Object-Level Metadata in lakeFS 

lakeFS supports two types of object-level metadata:

lakeFS Metadata Management Capabilities

To demonstrate lakeFS’s metadata management capabilities, we’ll use a lakeFS repository loaded with the CIFAR-10 dataset, a popular benchmark for image classification in ML.

Attaching Metadata to Objects 

Using the lakeFS Python SDK, we uploaded the CIFAR-10 files to lakeFS and attached user-defined metadata to label each file by its class. Once the dataset was loaded, we committed these changes to lakeFS.

Here’s how it appears in the lakeFS UI:

When we drill down into the object info of all the loaded images we can see its corresponding label in the user-defined metadata section. 

Editing Object Metadata 

While exploring the dataset in our lakeFS repository, we discovered a labeling error: all images of Trucks were mistakenly labeled as Cars due to a bug in our upload code.

lakeFS allows you to correct such errors by editing object metadata directly, without needing to rewrite the object itself. Following lakeFS best practices, we’ll create a new branch to fix the metadata error in isolation, preserving the integrity of production data. We named this branch “fix-metadata-error.” Branching in lakeFS is a zero-copy operation, meaning it doesn’t duplicate data.

Here is our branch: 

Editing Metadata on the New Branch

Now, let’s edit the incorrect metadata for the Truck images labeled as Cars. The following code snippet demonstrates how to update the metadata for a specific object on our new branch, lakefs://ai-metadata-management/fix-metadata-error/datasets/cifar10/train/aerial_ladder_truck_s_000001.png:

bug_fix_branch_name = "fix-metadata-error"
object_to_edit = "datasets/cifar10/train/aerial_ladder_truck_s_000001.png"
new_metadata = lakefs.client.lakefs_sdk.UpdateObjectUserMetadata(
    set={"class": "Truck"}
)

clt.sdk_client.experimental_api.update_object_user_metadata(
    repository=repo_name,
    branch=bug_fix_branch_name,
    path=object_to_edit,
    update_object_user_metadata=new_metadata
)

After updating the metadata, we commit the change.

Reviewing and Merging the Fix

By comparing our fix branch to the main branch, we can see the diff showing the object we’ve updated its metadata.

Confident in our changes, we can click the green Merge button to apply the fix to the main branch, which typically stores production data.

Metadata Versioning

With our metadata error corrected, we can now see lakeFS’s metadata versioning in action.Let’s inspect the labels for cifar10/train/aerial_ladder_truck_s_000001.png across two different commits. The first commit shows the metadata when we initially loaded the CIFAR-10 dataset into lakeFS. The second commit reflects the corrected metadata, where all erroneous labels have been updated from “Car” to “Truck.”

example_object = "datasets/cifar10/train/aerial_ladder_truck_s_000001.png"

commit_id_before_fix = "f04d3e5127e706337b1007be34f546be6ea02a986f93ed4b26a2fb81f7a2283c"
commit_id_after_fix = "bbe0d6f83cc14c22aed022bcf1991b803530e166e4998755a2cfdd016ceb6205"

commit_before = lakefs.reference.Reference(repository_id=repo_name, reference_id=commit_id_before_fix)
obj_before = commit_before.object(path=example_object)
print("Metadata before fix:", obj_before.stat().metadata)

commit_after = lakefs.reference.Reference(repository_id=repo_name, reference_id=commit_id_after_fix)
obj_after = commit_after.object(path=example_object)
print("Metadata after fix:", obj_after.stat().metadata)

Output:

Metadata before fix: {'class': 'Car'}
Metadata after fix: {'class': 'Truck'}

This example highlights how lakeFS preserves metadata history, allowing us to track changes, troubleshoot, and verify corrections with ease.

Metadata Quality and Consistency Assurance

lakeFS supports hooks that can run before versioning operations are completed, allowing you to implement validation logic and abort operations if validations fail. Combined with branching and merging, lakeFS hooks make it easy to implement the Write-Audit-Publish pattern, catching data errors before they reach production.

In our case, we learned from the previous metadata error where Trucks were mistakenly labeled as Cars, even though the CIFAR-10 dataset doesn’t include Cars. To prevent similar issues in the future, we can introduce a pre-merge hook that validates label values. This hook will catch such errors, fail the merge, and prevent data issues from cascading into production.

lakeFS hooks are designed to let you integrate custom validations that fit your data workflows, adding an extra layer of enforcement and increasing confidence when introducing new data to production.

Scalable Metadata-Based Object Filtering

Beyond its core metadata management capabilities – versioning, editing, and quality control – lakeFS also offers powerful metadata-based object filtering and search functionality. This feature is particularly valuable for data discovery, selection, reproducibility and troubleshooting within large datasets.

Key Benefits of lakeFS Metadata Search

1. Version-Aware Search: lakeFS enables you to search and filter datasets within the context of a specific data version, ensuring that search results are accurate and aligned with the state of the data at that point in time.
2. Reproducible Queries: lakeFS not only versions your data but also tracks search queries and their inputs (metadata and data), allowing you to reproduce exact search results even as your data evolves.
3. Scalability: Designed to handle massive datasets, lakeFS metadata search scales to meet the demands of data lake environments, providing efficient filtering and search capabilities regardless of data volume.

These scalable metadata search capabilities empower data teams to easily locate, select, and validate data across vast data lakes, simplifying model development, speeding up project delivery, and enhancing confidence in ML and AI projects. Practitioners benefit from lakeFS’ ability to conduct advanced searches, not only by keywords but also using criteria such as performance metrics or feature importance, making it easy to locate relevant experiments, models, and datasets precisely when needed.

Where lakeFS Excels in AI Metadata Management

Now that we’ve explored lakeFS’s metadata management capabilities, let’s highlight where it truly shines compared to other metadata management solutions:

• Metadata Versioning for ML Reproducibility: lakeFS’s metadata versioning is essential for ensuring reproducibility and troubleshooting in ML workflows.
• Fix metadata errors in Isolation: lakeFS allows you to fix metadata errors in isolation, preventing unintended changes from affecting production data.
• Enhanced Quality Control: With lakeFS hooks, you can implement metadata validation checks that stop errors before they reach production, adding an extra layer of confidence to your workflows.
• Data Discovery and Selection: lakeFS’s search and filtering capabilities simplify data discovery and selection processes, making it easy to find relevant data in large datasets.
• Built-In Version Tracking: As a data version control system, lakeFS inherently tracks versions of both data and metadata, eliminating the need for manual version tracking. This simplifies the management of different model or input versions and addresses common challenges in using metadata for lineage tracking.

With these strengths, lakeFS provides a comprehensive solution that goes beyond typical metadata management by offering quality control, versioning and powerful search capabilities.