How Amazon FSx for NetApp ONTAP and lakeFS give file-centric organizations a practical path to governed AI and data engineering workflows
A Fortune 500 pharmaceutical company recently gave us a useful answer to a question infrastructure vendors often get wrong: Why pair NetApp with lakeFS?
The answer started with years of critical data and established workflows built around file systems. Linux users mounted NFS. Windows users expected SMB and Windows Explorer. Access controls were tied to Active Directory. When the company moved this environment from on-premises ONTAP to AWS, asking every team to adopt a new storage model wasn’t a realistic migration plan.
At the same time, its analytics and AI stack increasingly expected S3. The company selected Amazon FSx for NetApp ONTAP because it could preserve the file access its users depended on while creating a path for S3-based applications. Amazon S3 access points for FSx for ONTAP can present file data through an S3 endpoint while that data remains available through NFS and SMB.
That solved the storage and protocol problem. It didn’t solve what happened when multiple teams needed to change, validate, and publish the data feeding analytics and AI.
The missing layer above storage
Consider a data science team preparing clinical trial data for a new analysis. One engineer corrects source records. Another changes a transformation. A third adds derived features. Every change may be reasonable on its own, but nobody wants partially tested work appearing in a shared production dataset.
The usual workaround is duplication. Teams create new directories, new buckets, or files with names such as trial-data-final-v2-reviewed. Those copies consume storage, drift from one another, and make it difficult to reconstruct the exact dataset behind a result.
Storage snapshots remain important for infrastructure protection. Object-level versioning also has a role. But AI and data engineering teams need to manage a dataset as a coherent unit: isolate a change, test it, inspect the difference, and publish it only after approval.
lakeFS is the control plane for AI-ready data. It adds that workflow above the storage layer, sitting between data storage and the tools, users, and AI agents that consume the data. A team can create a zero-copy branch of a dataset, make changes without affecting production, commit an immutable version, compare it with the current production state, and merge it when the work is ready.
The data stays on the customer’s storage. The workflow changes, not the storage foundation.
A working reference environment on Amazon FSx for NetApp ONTAP
Customers can use the public lakeFS + NetApp ONTAP reference environment as a practical starting point for this architecture. It shows how lakeFS and Amazon FSx for NetApp ONTAP work together on AWS, with a deployable environment and a concrete data workflow to run against it.
The reference environment uses Amazon FSx for NetApp ONTAP, with a native ONTAP S3 object-store server and bucket configured inside the file system. lakeFS runs on an Amazon EC2 instance and uses the ONTAP S3 bucket as its backing storage. Every object written through lakeFS is physically stored on FSx for ONTAP.
Workflow Example
The included workflow follows a simple ML feature-engineering example:
- A baseline training dataset was committed to the main branch.
- A data scientist, pipeline, or AI agent created an isolated branch for a proposed feature.
- The updated dataset was written and committed on that branch without changing the shared version.
- The team inspected the exact difference between the branch and main.
- After validation, the branch was merged into main with its history intact.
No full copy of the original dataset was required when the branch was created. The branch initially referenced the existing data and stored only subsequent changes.
The repository is a working reference, not a performance benchmark. It gives customers a repeatable way to use native ONTAP S3 on Amazon FSx for NetApp ONTAP as the storage foundation for lakeFS, with branching, commits, diffs, merges, and complete version history available to the data team.
For an AI agent, the branch is an isolated data sandbox. The agent can start from an approved dataset version, write proposed outputs, and submit them to the same validation and review process used for human and pipeline changes. The input and output commits give the run exact data references, so it can be reproduced, audited, or debugged later.
One storage foundation, with clear responsibilities
The products solve different problems.
NetApp ONTAP handles how data is stored, protected, and accessed. It supports the enterprise file protocols that many organizations already depend on, while also supporting S3-based workloads. On AWS, FSx for ONTAP gives customers a managed ONTAP environment without forcing established file users to redesign their workflows.
lakeFS handles how AI and data teams change data. It provides isolation for development and testing, reproducible dataset versions, automated validation points, controlled promotion, rollback, and an audit trail of what changed. Those controls apply whether the change comes from a person, a pipeline, or an AI agent.
That division is useful inside large enterprises. Storage and infrastructure teams can keep the controls and operating model they trust. Data scientists and engineers get a workflow designed for experimentation and review. The organization doesn’t need to maintain one storage estate for traditional applications and another simply because newer applications speak S3.
Not every AI or analytics application is designed to work directly with S3. lakeFS Mount exposes a lakeFS branch through a local filesystem interface, so existing applications and AI agents can read and write data using familiar file APIs. Their changes remain isolated in the branch and can be validated, committed, and merged through lakeFS before reaching the shared version.
FSx for ONTAP can support established file workloads and new data engineering and AI projects on the same storage foundation. lakeFS brings modern data workflows to that foundation.
Native ONTAP S3 and Amazon S3 access points
There are two related deployment patterns worth distinguishing.
The reference environment uses a native ONTAP S3 bucket running on Amazon FSx for NetApp ONTAP. This is a good fit for a new lakeFS-managed namespace stored directly on ONTAP.
Amazon S3 access points for FSx for ONTAP address a different requirement. They attach an S3 endpoint to an FSx for ONTAP volume, allowing S3 applications and AWS services to access file data without moving it out of the file system. NFS and SMB access can continue alongside the S3 path.
The right pattern depends on the workload. A team building a new governed data environment may choose native ONTAP S3 as the lakeFS backing store. A team with an existing file-based data estate may use an S3 access point to make that data available to S3 applications.
In either case, ownership boundaries need to be explicit. A namespace managed by lakeFS should be changed through lakeFS so its commit history remains authoritative. Multiprotocol support should not become a way to bypass validation or write directly around the versioned workflow.
What the pharmaceutical use case looks like in practice
For the pharmaceutical company, the attraction wasn’t another storage endpoint. It was the ability to support conservative, file-based workflows and newer S3 applications on the same NetApp foundation.
That creates a practical model for clinical and research data:
- Existing applications and users can retain the NFS or SMB access patterns they require.
- New analytics and AI workloads can access the storage through an S3-based path on AWS.
- lakeFS-managed datasets can be branched for experiments or transformations without copying the complete source.
- People, pipelines, and AI agents can work in isolated branches instead of writing directly to the shared version. Workloads that expect a filesystem can access those branches through lakeFS Mount.
- Data quality and policy checks can run before changes are merged into the shared version.
- Every committed data state has an immutable identifier and full history, so experiments, analyses, and agent runs can be tied to exact input and output data versions.
A researcher can return to the exact data used for an earlier analysis. A data engineer can show which records or files changed before approving a merge. An AI agent can propose a transformed dataset without changing the approved version. A platform team can give people, pipelines, and agents isolated workspaces without creating another uncontrolled collection of copied datasets.
These are ordinary requirements in regulated data environments. The hard part has been satisfying them without disrupting the applications, access methods, and storage controls already in place.
A more practical cloud modernization path
Cloud migrations are often described as clean breaks: move the data, replace the interfaces, and retrain the users. Large data environments rarely cooperate with that plan.
The pharmaceutical company’s experience points to a more practical approach. Keep the storage model that critical users and applications depend on. Add S3 access where modern services require it. Then add lakeFS where people, pipelines, and AI agents need isolation, reproducibility, review, and governance across dataset changes.
NetApp carries the established data estate into AWS. lakeFS gives new pipelines, experiments, and agent runs a controlled way to work with it.
That may sound less dramatic than a clean-slate architecture. For a regulated enterprise, that’s usually the point. Teams can modernize the data workflow without treating every established access pattern as technical debt.
Explore the architecture
lakeFS + NetApp ONTAP reference environment
Amazon S3 access points for Amazon FSx for NetApp ONTAP
NetApp ONTAP S3 multiprotocol support
lakeFS – the Control Plane for AI Ready Data
