In highly regulated environments, improving developer experience often comes at the cost of tighter controls. For companies handling sensitive personal data, even small workflow changes can introduce compliance risks that are difficult to detect and even harder to fix at scale.
The tension between usability and governance is especially visible in machine learning pipelines. Data must remain accessible, versioned, and stable, yet fully traceable and compliant at every step.
In a recent presentation at the AI-Ready Data Summit, Mickaël Lacour showed how the digital security company Entrust approached this challenge.
Faced with a fragile storage system built on AWS S3 that created friction for engineers and uncertainty for compliance teams, the company made a deliberate shift to lakeFS. The goal went beyond modernizing infrastructure. Entrust rethought how data flows through the system so that every dataset is validated, tracked, and governed from the moment it enters storage.
The result? Entrust transformed from reactive compliance to proactive control.
By embedding validation into commit workflows, automating lifecycle management, and treating versioning as a core primitive, Entrust rebuilt its data platform around both reliability and accountability. What emerged is a system that enables applied scientists to move faster with confidence. No need to compromise the strict requirements governing PII at scale.
Addressing Compliance Requirements at Entrust
Entrust provides digital identity verification for know-your-customer and fraud-prevention workflows, using document and biometric checks at scale. To do that, the company handles a large amount of personally identifiable information.
The needs of applied scientists and engineers are familiar across the industry. They need to prepare datasets, version them, update them, protect them from changes during training, inspect their contents, and share them with colleagues. Those are standard data workflow requirements, but working with PII adds another layer of complexity.
The company must remain compliant across multiple legal and contractual frameworks, including regulations such as BIPA and GDPR, as well as internal customer agreements. That means Entrust must ensure it has proper approval and opt-in before storing or using any PII. From there, the challenge becomes managing storage, tracking, and automatic deletion in a way that remains compliant at scale.
Why Entrust Moved to lakeFS
Entrust previously used an internal storage layer built on top of AWS S3. While that system worked functionally, the user experience was poor.
The company wanted to migrate to a new storage system and chose lakeFS. The goal was to preserve strict compliance controls while significantly improving how applied scientists and engineers interact with datasets.
Tracking PII Access and Storage
A core requirement was the ability to track exactly which PII was stored. This included who had access to it, when access occurred, where it was stored internally, and how it was stored. The team needed visibility into whether data was stored as a JPEG, PNG, video, packet, or lens file. Finally, they needed to know how that data related to specific applicant identifiers.
Because the storage system has to be monitored so closely, there is always a risk that compliance controls will create friction for users. One of the main lessons from the migration was that it is possible to maintain strong oversight while still delivering a much better user experience.
How lakeFS Improved Validation at Commit Time
The first major improvement came from how lakeFS helped Entrust answer those compliance and tracking questions as data entered the system. Under the old workflow, a dataset would first be stored in S3, and the company would check compliance and full trackability only later. If it was not compliant, the dataset would be deleted.
That created a serious problem. Between the time data was saved and the time a user started training, one or two days could pass. A scientist or engineer could begin training only to have the dataset deleted mid-process, wasting both compute and storage resources and ultimately costing the company time and money.
With lakeFS, Entrust uses pre-commit hooks to validate data synchronously before it is committed to a repository.
This means the system verifies that the data is understandable, compliant, and acceptable before it is included in the official dataset. The user receives an answer immediately and cannot proceed until validation passes. Internally, Entrust built this with AWS Lambda, Python, and Polars.
For small and medium datasets, the response comes back in seconds. For very large datasets containing tens of millions of PII records, validation can take up to 10 minutes. Even so, that is far better than discovering a compliance issue days into training.
If the data is not validated, it remains in an uncommitted state. That is important because it allows the company to manage and eventually delete noncompliant or unresolved data separately.
Indexing and Metadata After Merge
After validation, Entrust also needs to know where and how the data is stored inside lakeFS. That is handled by a post-merge hook that runs asynchronously, so the user is not blocked while indexing occurs. The system builds many small indexes that later feed into broader tracking workflows.
To answer the questions of who accessed the data and when, the company relies on lakeFS audit logs and object metadata. Entrust also had requirements around user accounts and policy enforcement. For this the team used the lakeFS API to build an internal Terraform provider.
Their infrastructure is already managed by Terraform. This allows creating or deleting users, assigning them to groups, and attaching the appropriate policies in a fully coded, repeatable way.
The PII Lifecycle Inside the System
Entrust’s internal PII lifecycle works across several steps:
The first step is synchronous validation through a pre-commit hook. The user must wait until the validation completes. Once that step passes, the user is free to continue without further friction.
The second and third steps happen internally. Step two uses a lakeFS post-merge hook. Step three relies on internal jobs that aggregate the many small indexes created during indexing. Later in the lifecycle, deletion occurs, which Entrust treats as a separate, carefully managed process.
Building a Storage Manager Around lakeFS
To ensure consistency across repositories, Entrust built an internal storage manager. This was mainly needed because one feature missing from lakeFS for the company’s use case was repository templating. The storage manager solved that.
Users are not allowed to create repositories directly. Instead, they use the storage manager UI to provide the repository name. If the repository is not intended for PII, they can create it quickly. If it is a PII repository, the system asks additional questions, such as what type of PII will be stored.
This matters because different data types may have different retention requirements. For example, under BIPA, biometric embeddings may need to be deleted within one year. Even if the end user has not explicitly requested deletion.
The storage manager then creates the PII repository, configures the necessary hooks, and protects those hooks from modification. It also manages hook calls from the various PII repositories and introduces a protected branch pattern.
In this model, users cannot directly commit to protected branches; they can only merge into them. Indexing is performed only on protected branches to keep lineage cleaner and reduce indexing costs.
Handling Non-Protected Branches and Uncommitted Data
Non-protected branches need special attention. Even though the data on those branches has passed pre-commit validation, it hasn’t yet been indexed. This means the company doesn’t yet know what is inside. Entrust also has to deal with uncommitted data that failed validation or was never finalized.
These cases are especially important for compliance because regulations such as GDPR may require deletion within a strict window, such as 30 days after a valid deletion request.
To handle this, the storage manager runs a cron job that scans repositories, checks non-protected branches for uncommitted data, and deletes any items that exceed the allowed threshold. That threshold depends on the company’s contracts, but it cannot exceed the applicable legal limit. The same process also removes non-protected branches that have been left around too long.
Soft Deletion on Protected Branches
Once the indexing system has created a large aggregated index of all PII in lakeFS storage, Entrust can apply compliance rules more precisely. For example, the system can identify data, such as biometric embeddings, that are older than 1 year, or match PII UUIDs against deletion requests from production or end users directly.
Thanks to lakeFS metadata, the company can locate the relevant repository, branch, and file for each item that needs to be removed. For protected branches, the deletion workflow is handled through a dedicated branch-based process. Entrust creates a branch, performs what the team calls a “magic deletion,” commits the change, merges it, and then deletes the temporary branch.
If the data to remove is a standalone file, such as a media file, it is deleted directly. If the PII is embedded in a larger file, such as a Parquet, lens, CSV, or TensorFlow file, the system opens the file, removes the specific record, and rewrites it. The result is then committed and revalidated through the same compliance checks used during normal commits.
After the merge, the index and aggregated tracking structures are automatically updated. This was a major improvement over the old system.
Hard Deletion with Garbage Collection
Soft deletion is only part of the compliance story. Entrust also needs to ensure that deleted data is truly removed from the system. This is where lakeFS garbage collection became one of the most valuable features.
If the garbage collector is configured with a threshold lower than the compliance limit, such as 15 days, then once that period passes, deleted objects are replaced with tombstones or dead links. This means hard deletion is eventually enforced without requiring the company to build custom logic to physically purge the data.
For Entrust, that was a major advantage because it made hard deletion reliable and operationally simple.
Unexpected Benefits: Versioning and Stable Training Inputs
lakeFS also delivered benefits the team had not initially expected. In the previous system, there was no real versioning. Each time a user wanted a new dataset version, the company had to copy large amounts of data from one S3 folder to another. The versioning was effectively simulated, but the underlying data was duplicated each time.
Deletions were also applied immediately. That meant a training job could lose data between epochs without warning, creating inconsistency and confusion. With lakeFS, versioning and soft deletion together give the team immutable training inputs for a defined period. If the garbage collector is set to 15 days, for example, users can train on a dataset for those 15 days without interruption or unexpected mutation.
Using Tags and Diffs for Better Dataset Management
Another valuable capability is lakeFS tagging. Once a team is satisfied with a dataset used for training, it can tag that version as a trusted state. Later, if the team wants to train again, it can diff the current head against that tag and immediately see what changed. Because the diff is metadata-based, it is much faster than the old S3-based process, which required listing and comparing bucket contents.
This makes it easier for applied scientists to judge when too much data has been deleted and whether they should create a fresh dataset. It also helps engineers automate more of the process.
With lakeFS reference files, local tooling, and post-merge hooks, teams can build workflows that automatically compare the latest dataset state to a known-good tag, trigger alerts when deletions cross a threshold, or launch a new training and testing workflow without waiting for a human to notice a problem.
Conclusion
The main point of the talk is that lakeFS features such as versioning, hooks, and garbage collection dramatically improved the user experience for Entrust’s applied scientists and engineers while preserving a fully compliant PII-handling system.
By validating data before commit, indexing after merge, automating lifecycle controls, and using garbage collection for hard deletion, the company moved from a fragile, frustrating workflow to one that is far more reliable, transparent, and scalable.
To see the full walkthrough of Entrust’s architecture and hooks, watch Mickaël’s session on demand from the AI-Ready Data Summit.
