Ready to dive into the lake?
lakeFS is currently only
available on desktop.

For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!

lakeFS Community
Einat Orr, PhD
Einat Orr, PhD Author

Einat Orr is the CEO and Co-founder of lakeFS, a...

Last updated on April 26, 2024

Despite data technology advancements, many organizations still struggle to access outdated mainframe data. Most of the time, you’re looking at siloed data architecture that just doesn’t align with their strategic goals. At the same time, organizations are under pressure from their competitors.

A good data strategy enables companies to go beyond function-specific and interdepartmental analytics and provide enterprise-wide analytics using data from both internal and external sources. It also opens the doors to using machine learning to address new issues and achieve objectives faster.

Unlike companies that were created in the cloud and use cloud-native approaches, established organizations may not have had access to all their data from the start. They must integrate data from several sources to compete successfully, yet mainframe data is typically difficult to access. 

This is where data lakes come in. What enterprises need is a data architecture that encompasses all data sources, regardless of kind, format, origin, or location. It also needs to be quick, easy, cost-effective, safe, and future-proof.

Keep reading to learn more about building a modern data lake for the GenAI and machine learning era. 

What is a data lake?

Data Lake Features

A data lake is a collection of technologies that enable the querying of data in files or blob objects. When used correctly, they allow for studying structured and unstructured data on a massive scale and with cost efficiency.

Data lakes offer a wide range of analytic operations, from simple SQL querying of data to real-time analytics and machine learning applications.

Primary components of a data lake

Data lake components

Legacy data lake operation: 4 key issues

What are the issues preventing teams from realizing the enormous potential of data-driven processes and AI?

Legacy data architectures and segregated storage systems act as substantial barriers to AI projects going forward. You won’t be able to employ AI to develop new insights with your data until your company implements current, cloud-based data architectures and frees it from silos.

1. Data Silos

Many companies save data across multiple discrete repositories, ranging from local hard drives and workstations to file sharing. These silos represent a decades-old approach to data storage that is unsuitable for sophisticated analytics and AI applications.

Moving and distributing segregated data is incredibly inefficient. When data is scattered around an organization and stored in antiquated systems, operations are sluggish, collaboration is difficult, and capitalizing on AI is impossible. 

AI and machine learning algorithms demand huge datasets with labeled outputs. Data practitioners must first grasp the present data to guarantee a successful solution. Next, they must compare various modeling strategies and train and test alternative algorithms on appropriately labeled datasets. Designing and executing algorithms when some of your data is trapped on other file shares is a challenging task.

But this is just the tip of the iceberg.

Organizations with siloed data can’t provide effective data security and backup for each individual silo, putting data at risk of loss. Furthermore, physically transferring data from one silo to another increases the chance of data input mistakes.

To make things worse, siloed data is generally stored in proprietary formats developed by suppliers and vendors. Those proprietary formats bind teams to narrow vendor ecosystems in which companies hold your data prisoner. What does that mean? They won’t be able to use data in other applications or develop specific algorithms.

You must first prepare that data for exploratory data analysis, model training, and evaluation before you can display, analyze, or use it in AI/ML applications. After the algorithm has been established, executing an AI workload in production calls for data transformation into a uniform format. That format must also align metadata taxonomies (definitions of data items and structures) with ontologies (descriptions of connections between data elements).

Data that is stored in silos and locked in proprietary formats remains stagnant. It lacks the liquidity required to facilitate collaborative scientific work or capitalize on the possibilities of AI applications. 

2. Lack of Discoverability

Legacy data systems may add information to files, but often fail to unify metadata taxonomies and ontologies, making it difficult for data practitioners to discover new or historical datasets. 

Data is searchable and digestible only if the user knows which keywords or labels to query. In many circumstances, teams repeat an assay or experiment since it is easier than locating previous data.

3. Scalability Constraints

On-premises legacy data systems are difficult and expensive to scale. Each update requires several modifications, including database, server, and file storage improvements. 

As a result, such legacy systems aren’t the optimal setting for compiling the large-scale datasets necessary for AI.

4. Costs Constraints

When organizations miss technologies that allow effective data management, they often revert to copying large amounts of data to ensure isolation. This is not only error-prone but also expensive as storage costs pile up. Such copies may become up to 80% of the storage expense in a data environment.

Building a modern data lake for the GenAI & ML era

Data Platform Architecture 101

How do you develop a modern and effective data platform? You need to have the right technology, efficient management, and smooth operations to develop. Here are the key components to which you should pay attention:

Data Governance

Data lake platforms often lack straightforward data governance enforcement mechanisms. Implementing such rules can be a challenging task that demands constant monitoring. Typically, this comes at the price of data engineering or other business-enhancing DevOps efforts.

But it’s essential that teams looking to leverage modern data lake solutions establish standards for data access and security, create contracts for data validation, and ensure compliance with regulations.

Data Sources

Another part of the equation concerns data sources. Teams looking to implement best practices for modern data lake operations and drive value from data must find a solid method for acquiring high-quality, reliable raw data from all relevant sources. This step is key to addressing the problem of data silos mentioned above.

Data Preprocessing Pipelines

Next come data preprocessing methods that play a critical role in achieving high data quality and thereby making data trustworthy to data consumers. Teams should validate, clean, and curate the ingested data. Some common pre-processing methods include data augmentation and standardization.

Data Lakehouse

The data lakehouse approach builds a repository that serves as “the source of truth” for data throughout all phases of its evolution. Teams should use the Medallion architecture model here, classifying their data from bronze (raw data) to gold (cleaned and curated data).

ML/AI and Analytics Research and Training

Gold-level data should be used as input to the architecture component that help ML/AI engineers with their research and training tasks. This produces results that can be employed further in ML applications.

Data Consumption

Finally, there’s data consumption. Common data consumption methods in modern organizations include Business Intelligence tools, AI/ML apps, and APIs.

Wrap up

One thing is clear: legacy data systems cannot provide appropriate data flexibility, searchability, accessibility, or effective scaling to accommodate the huge amounts of AI-native data required for AI algorithms.

Choosing the correct technology for the data architecture is critical, and the decision depends on the individual requirements of the organization. While there is no true “one size fits all” approach here, the components mentioned above are all worth considering when building a data strategy and a modern data lake to drive it.

At lakeFS, our objective is to become the standard data version control solution that runs smoothly across all of your data – regardless of where it is kept – and support engineering best practices throughout the data’s lifespan.

Git for Data – lakeFS

  • Get Started
    Get Started
  • Where is data engineering heading in 2024? Find out in this year’s State of Data Engineering Report -

    Read it here