As you can see above, we have created two data validation checks in the form of SQL queries:
- Check for integrity between the loan_payments foreign key and loans primary keys
- Check no payment is ever higher than the total amount of the loan
Since branching and merging in lakeFS are zero-copy metadata operations, we can utilize a separate branch from the main one for ingesting new files. In this way new data gets added in isolation and can be tested by a lakeFS hook to run the validation before being merged back to main.
The first step is to create the lakeFS branch, which we will call dev-reports. We can create it using the API, CLI or the lakeFS UI: