The now-infamous “pizza with glue” AI result is a symptom of something deeper than one bizarre edge case. When AI systems fail, the root cause is rarely mysterious. More often than not, bad outputs can be traced back to bad inputs: flawed data, unclear lineage, or uncontrolled environments.
Smarter models won’t fix this on their own. The harder challenge is building systems we can understand, debug, and trust. And that starts with one principle: reproducibility.
Adding Glue to Pizza, Seriously?
A while back, if you searched Google for “cheese not sticking to pizza,” you could get some very strange advice. One answer suggested adding non-toxic glue to the sauce to help the cheese stick better.
Technically, that might make the cheese stick. It probably wouldn’t make the pizza any more edible.
So the obvious question is: why would a system return an answer like that?
To answer that, it helps to simplify how modern AI systems work.
At the most basic level, an AI system can be thought of as a function that takes some input and produces some output:
input → function → output
Let’s break this down:
Function
The function includes everything that shapes model behavior:
- The model itself
- Training code
- Weights and parameters
- Embedding algorithms
- System prompts
- Runtime environment
Input
The input includes:
- Training data
- Fine-tuning data
- Retrieved context
- Prompts
- The user’s query
In the pizza example, the input was the user query: “cheese not sticking to pizza.”
Output
The output is whatever the model produces:
- A text response
- Generated code
- A tool call
- An automated action from an agent
That may sound simplistic, but one point matters more than anything else: the input is the part we control most. We usually don’t control the foundation model, and we cannot fully predict the output. But we do have meaningful control over the data and context we feed into the system.
What Does a Typical AI Application Look Like?
Suppose a team is building an application that works with images. Users interact with the application, and behind the scenes, the system stores and processes several different forms of data:
Raw assets
The original images may be stored in an object store.
Structured metadata
The system may store structured metadata such as:
- Image size
- Captions
- Labels
- Generated tags
- Other useful metadata
Vector representations
The system may also store vector embeddings for search and similarity. Those different forms of data are then fed into an AI application that may:
- Retrieve similar images
- Query metadata
- Generate thumbnails
- Create new content
- Trigger downstream agents
This is a simplified picture, but it captures the shape of many modern AI systems.
The Reality: AI Applications Don’t Live in Isolation
In practice, applications are surrounded by a much messier data environment.
There may be both high- and low-quality datasets, spreadsheets, stray CSV files, mysterious legacy exports. And every organization has some version of the file nobody can explain, but the system somehow cannot function without it.
And that is exactly where reproducibility becomes critical.
When the System Produces Garbage
Let’s say the AI application starts returning bad results. In Google’s case, that bad result was recommending glue on pizza.
In another context, the consequence could be much more serious:
- A user gets harmful advice
- A team makes a business decision based on incorrect information
- An automated system takes the wrong action
So, now the team needs to troubleshoot. Why did the application misbehave?
The First Layer of Troubleshooting
The first place to look is the application itself. We can inspect logs, metrics, prompts, retrieved context, and model responses.
If we have done a good job, we have records of the full prompt chain, including the user prompt and the generated response. That can help us identify what input the model saw when it produced the bad result.
But there is a harder problem underneath that.
The Hard Part: Where Did the Bad Data Come From?
In the pizza case, the issue appears to trace back to a sarcastic Reddit comment from years earlier suggesting that if cheese slides off pizza, someone should add glue to the sauce.
Somehow, that low-quality piece of content entered the broader data ecosystem, was used in a training or retrieval context, and eventually surfaced as a serious recommendation.
That is the real challenge facing troubleshooting teams: finding the needle in the haystack.
Once the model has consumed huge volumes of data, tracing a bad output back to the exact upstream source is difficult. And yet that is exactly what teams need to do if they want reliable AI systems.
Garbage In, Garbage Out Still Applies
At its core, this is the same old problem in a new form. If we feed garbage into the system, we should not be surprised when we get garbage back out.
AI hasn’t changed that truth. It only raised the stakes.
When something breaks in software, one of the first questions we ask is: What version are you running?
That question works because code is usually versioned by default. We have Git, GitHub, release workflows, automated tests, and a mature software lifecycle.
With data, that is much less common.
And that is a problem, because data and code actually share a lot in common.
Data and Code Are More Similar Than They Seem
There are several important similarities between data and code:
Both have upstream dependencies
Data comes from somewhere: APIs, scrapers, collection systems, user uploads, or external providers. Code also depends on libraries and external packages.
Both can have quality problems
Some problems are introduced internally. Others come from dependencies. In either case, we cannot assume quality by default.
Both are hard to troubleshoot
Software engineering has spent decades improving debugging, testing, and versioning. Data needs the same discipline.
Both can now be machine-generated
Increasingly, code and data are produced or transformed by machines, making traceability even more important.
That raises an obvious question: if version control works so well for code, can we apply the same thinking to data?
Why Traditional Git Isn’t Enough for Data
In theory, we would love to use the same model:
- Version everything
- Test automatically
- Reproduce releases
- Roll back when needed
The problem is scale.
Git and GitHub were designed for code, not massive data lakes. Codebases may be large, but enterprise data environments can be vastly larger, sometimes reaching petabyte or exabyte scale.
This means we need data-native tools that bring the same lifecycle ideas to data, but in infrastructure that can actually handle data volume.
A Better Approach: Bring Software Discipline to Data
The goal is to bring the same experience we have for managing code into data workflows:
- Repositories
- Branches
- Isolated changes
- Reviews
- Approvals
- Promotion to production
When applied to data, this creates provenance: a record of where data came from, how it changed, who changed it, and how it moved through the system.
What Provenance Looks Like in Practice
Imagine a dataset of scraped Reddit comments. We want to be able to answer questions like:
- Who last modified this dataset?
- When was it changed?
- What was added?
- Why was it changed?
- Where did it come from?
- What code produced it?
That means capturing metadata such as:
- Author or system identity
- Timestamp
- Commit message or change description
- Source URLs
- Ownership information
- Ingestion frequency
- Linked code commit IDs
This kind of context makes troubleshooting dramatically easier. If a model suddenly starts recommending something absurd, the team needs to know exactly which data source introduced that behavior.
Guardrails: Prevent the Problem Before It Reaches Production
Tracing bad data is important, but preventing it from entering the system is even better.
That means creating guardrails in the ingestion process. Instead of writing data directly into production, the team can ingest it into an isolated branch or staging area first. There, automated checks can evaluate whether the data is acceptable before it is merged into production.
This creates a workflow similar to a pull request in software development.
What Those Data Quality Gates Can Check
Automated checks can validate many things, including:
- Privacy and compliance – For example, ensuring there is no PII in the dataset
- File and storage format – Ensuring data lands in the expected format, such as Parquet or Iceberg
- Content quality – Checking for prohibited patterns or unsafe content
- Statistical drift – Comparing column distributions to previous versions to detect unexpected changes
- Schema compatibility – Ensuring changes remain backward-compatible
- Organization-specific definitions of quality – any rule the organization considers essential can be encoded and automatically enforced
If the data fails, it doesn’t get promoted. Just as importantly, the failing snapshot is preserved in isolation so the team can inspect it without impacting production.
Reproducibility Isn’t Only About Data
So far, the focus has been on input data, because that is often where the biggest leverage sits.
But reproducibility also applies to the function itself: the model, the logic around it, and the environment it runs in.
How to Make the Compute Layer More Reproducible
- Make randomness explicit – If sampling or randomness is involved, the seed should be explicit. The process may still be random, but it becomes repeatable
- Lower temperature where appropriate – Reducing temperature may make the model less creative, but it usually makes behavior more stable and repeatable
- Measure similarity, not just exact equality – Model outputs may not be identical run to run, but they should still be comparable. Similarity testing can help determine whether results remain within acceptable bounds
- Connect code to data – The team should link datasets to the exact code used to generate, transform, or ingest them
- Make infrastructure reproducible – Use infrastructure as code. Pin dependencies. Treat runtime environments with the same rigor used in software engineering
A system may produce different results months later simply because the environment has changed. That needs to be controlled.
Reproducibility Gets Harder With Agents
The problem becomes even more complex when the system is not just answering questions but making decisions and taking actions.
Agents often generate plans, choose tools, call other systems, use external search results, and work across multiple steps. That means there is more to capture.
What to Store for Agentic Systems
For agent-based systems, teams should store the following:
- Plans and specs – If an agent decides how it will approach a task, that plan should be recorded and versioned
- Decision records – We should capture not just what the agent did, but why it did it
- Inputs and supporting artifacts – That includes documents, retrieved context, web search results, tool calls, and tool outputs
- Model and environment details – The exact model version, execution environment, and system configuration should all be recorded
This data is usually cheap to store and extremely valuable when something goes wrong.
The Bigger Picture: Data + Compute Reproducibility
When we combine data reproducibility with compute reproducibility, we get something powerful:
- Traceable inputs
- Explainable transformations
- Inspectable execution
- Reproducible outputs
- Stronger quality control
That turns AI systems from black boxes into systems we can actually manage. Without that, we’re just flying blind.
Final Takeaways
Reproducibility is the foundation of reliable AI systems. Without it, debugging becomes guesswork, failures repeat, and trust erodes.
But when teams can trace every transformation, every parameter, every decision, to both the data and the computation, they get more than visibility. They get control: the ability to improve quality deliberately instead of guessing, and to build AI systems that behave the same way tomorrow as they did today.
Everything described here, isolated branches for ingestion, quality gates before promotion, and commit-level provenance linking data to code, is what lakeFS, the control plane for AI-ready data, provides on top of your existing object storage. See how data version control works in the lakeFS documentation, or try it on your own data.
