Ready to dive into the lake?
lakeFS is currently only
available on desktop.

For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!

lakeFS Community
Tal Sofer
Tal Sofer Author

Tal Sofer is a product manager at Treeverse, the company...

Last updated on May 29, 2024

Building a data pipeline is a smart move for data engineers in any organization. A strong data pipeline guarantees that the information is clean, consistent, and dependable. It automates discovering and fixing issues, ensuring high data quality and integrity and preventing your company from making poor decisions based on inaccurate data.

This article dives into data pipeline development and shows how to use Databricks to create a comprehensive data pipeline. 

By the end of this article, you will be able to create an end-to-end data pipeline that automates data processing and analysis, freeing up time and resources for more vital activities. 

What is a data pipeline?

A data pipeline carries out the procedures required for transporting data from source systems, converting it according to requirements, and storing it in a target system. It contains all of the operations required to convert raw data into prepared data that customers may consume.

For example, a data pipeline prepares data so that data practitioners – from data analysts to data scientists – can extract value from it through analysis and reporting.

An extract, transform, and load (ETL) process is a popular type of data pipeline. ETL processing involves ingesting data from source systems, writing it to a staging area, transforming it according to requirements (ensuring data quality, deduplicating records, and so on), and then writing it to a destination system such as a data warehouse or data lake.

In every data pipeline, you’re likely to find three important elements: 

  • Source
  • Data processing processes
  • Destination, or “sink” 

Data can be updated throughout the transfer process, and certain pipelines can simply modify data with the same source and destination systems.

Data pipelines have become increasingly sophisticated in recent years to meet the demands of large organizations that deal with vast volumes of data.

It is essential that you follow all the steps your data pipeline outlines. This is how you can guarantee that pipelines have minimal data loss, deliver high accuracy and quality, and can grow to meet enterprises’ changing demands. They should also be adaptable enough to work with structured, unstructured, and semi-structured data.

ETL vs data pipeline

As companies rely more on data to drive decision-making, understanding the systems that handle this data becomes crucial. ETL and data pipelines are the most commonly mentioned data management solutions.

Although they handle data similarly, their operations and use cases differ significantly. Understanding your data management requirements’ unique demands and context will help you decide between an ETL process and a more comprehensive data pipeline solution.

Here’s a table-based comparison to show the fundamental distinctions between ETL and data pipelines:

ETL Data Pipeline
Purpose Designed primarily for batch processing and data integration into a single data warehouse It aims to continuously flow and process data for a variety of reasons, including loading into a data warehouse
Process Typically, a batch-driven process that runs in the following order: Extract, Transform, Load A continuous, real-time process that can handle streaming data through a series of phases that may or may not include transformation
Data Handling Frequently works with enormous amounts of data, handled in batches at regular periods Designed to handle batch and real-time data, allowing for quick processing and availability
Transformation Transformation is a critical stage that frequently involves complicated data operations Depending on the use case, transformation might be modest or substantial; it can even be completely skipped
Latency Batch processing causes higher delay; it is rarely real-time Lower latency by enabling real-time processing and instant data access
Scalability Scalable but constrained by batch processing restrictions and window durations Highly scalable, generally designed to scale automatically in response to increased data and processing needs
Flexibility It is less versatile since it is usually intended for certain, established workflows More adaptable, capable of adjusting to many sources, formats, and destinations as needed
Complexity The complicated transformation processes required frequently result in high levels of complexity Complexity ranges from low for simple data transportation to high for complicated processing pipelines
Infrastructure Frequently relies on conventional data warehousing and monolithic structures Uses current data storage options, cloud services, and microservices architecture
Use Case Ideal for use cases where data consolidation and quality are important, but real-time analysis is not required Suitable for scenarios that require quick insights, such as monitoring, alerting, and streaming analytics

Common examples of data pipelines

To help you understand the versatility of the data pipeline concept, here are a few examples of data pipelines teams are using today to accomplish different goals within their organizations:

  • Batch pipeline—A batch data pipeline is an organized and automated system used to handle huge amounts of data at regular intervals or in batches. It differs from real-time big data processing, which processes data as it comes. This method is especially useful when immediate data processing is not required. 
  • Streaming pipeline—A streaming data pipeline continually moves data from source to destination as it is generated, making it valuable along the way. Streaming data pipelines are used to load data into data lakes or warehouses and publish to a message system or data stream.
  • Delta architecture—Delta Lake is an open-source storage layer that leverages transactional databases’ ACID compliance to improve data lake dependability, performance, and flexibility. Delta Lake allows you to create Data lakehouses, which offer data warehousing and machine learning on the data lake. With capabilities like scalable metadata handling, data versioning, and schema enforcement for large-scale datasets, it provides data quality and dependability for your analytics and data science projects. It’s suited for applications requiring transactional capabilities and schema enforcement within your data lake.
  • Lambda architecture—Lambda architecture is a data deployment paradigm for processing that combines a standard batch data pipeline with a rapid streaming data pipeline to handle real-time data. In addition to the batch and speed levels, Lambda’s design contains a data-serving layer that responds to user requests.

How to build data pipelines in Databricks

This is a guide to Databricks pipelines. Start with this primer if you’re not sure how to get started with Databricks: What is Databricks?


  • You’ve logged into Databricks and are in the Data Science & Engineering workspace.
  • You have permission to build or access a cluster.
  • (Optional) To publish tables to Unity Catalog, first construct a catalog and schema.

Step 1: Create a cluster

Start by creating a cluster to provide the computing resources required to run commands.

In the sidebar, click the Compute button. On the Compute page, select Create Cluster.

On the New Cluster page, give the cluster a distinctive name.

In Access mode, choose Single User. Select your user name under Single user or service principal access.

Leave the remaining options in their default settings and click Create Cluster.

For additional information, check out this documentation page about Databricks clusters.

Step 2: Explore the source data

Understanding the pipeline’s source data is a frequent initial step in the creation process. In this stage, you will explore the source data and artifacts in a notebook using Databricks Utilities and PySpark commands.

Learn how to create a data exploration notebook and start exploring data. 

Step 3: Move raw data to Delta Lake

In this stage, you put the raw data into a table to prepare it for future processing. Databricks suggests using Unity Catalog to manage data assets on its platform, such as tables. 

However, if you don’t have permission to create the necessary catalog and schema to publish tables to Unity Catalog, you can still complete the steps below by publishing data to the Hive Metastore.

Databricks recommends Auto Loader for those looking to ingest data. Auto Loader discovers and processes new files that arrive in cloud object storage. You can set up Auto Loader to automatically determine the schema of loaded data, allowing you to begin tables without explicitly specifying the data schema and change the table schema when new columns are added. This reduces the need to manually monitor and implement schema changes over time. 

Refer here for a practical example with code.

Step 4: Transform and write data to Delta Lake

To prepare the raw data for analysis, the following steps modify data by filtering out unnecessary columns and adding a new field carrying a date to create the new record.

In the sidebar, click New Icon New and then Notebook from the menu. The Create Notebook dialog will then come up.

Enter a name for your notebook. Switch the default language to Databricks SQL and enter your code.

If you’re using Unity Catalog, replace it with a catalog, schema, and table name to store filtered and modified records. Otherwise, replace it with the table’s name containing the filtered and modified entries.

Click the Run menu and choose Run Cell.

Step 5: Analyze the transformed data

In this stage, you expand the processing pipeline by including queries for analyzing data. These queries rely on the records prepared in the preceding stage.

In the sidebar, click the New icon and then Notebook from the menu. The Create Notebook dialog will pop up.

Enter a name for your notebook and switch the default language to SQL.

Enter a query into the first cell of the notebook. Click Down Caret in the cell operations menu, pick Add Cell Below, and then type another query into the new cell.

To execute the queries and examine the results, click Run All.

Step 6: Create a Databricks job to run the pipeline

A Databricks job may be used to establish a pipeline that automates data intake, processing, and analysis.

In your Data Science & Engineering workspace, perform one of the following:

Click on the Workflows icon in the sidebar, then the Create Job button.

In the sidebar, click the New icon and then choose Job.

In the task dialog box on the Tasks tab, substitute Provide a name for your job.  Choose Notebook as the task type from Type. In Source, pick Workspace.

Use the file browser to locate the data ingestion notebook, then click the notebook name and Confirm.

In Cluster, choose Shared_job_cluster or the cluster you built during the Create a cluster stage.

Click Create.

Click the Add Task button underneath the task you’ve just created and pick Notebook.

Enter a task name. Choose Notebook as the task type from Type. In Source, pick Workspace.

Use the file browser to locate the data preparation notebook, then click the notebook name and Confirm. In Cluster, choose Shared_job_cluster or the cluster you built during the Create a cluster stage.

Click Create.

To run the workflow, click the Run Now button. To examine the run’s details, click the link in the Start time column of the task runs view. Click on each task to display the task’s run information.

Click the last data analysis job to see the findings after the procedure is complete. The output page appears, displaying the query results.

Step 7: Schedule the data pipeline job

You can divide the ingestion, preparation, and analysis phases into different notebooks – then each will be used to construct a job task. If all the processing is done in a single notebook, you may schedule it straight from the Databricks notebook interface. 

A frequent requirement is to execute a data pipeline regularly. To create a schedule for the task that runs the pipeline:

Click the Workflows icon on the sidebar. In the Name column, choose the job name. Job information is displayed on the side panel. In the Job Details window, click Add Trigger and choose Scheduled as the trigger type.

Determine the period, start time and time zone. Select the Show Cron Syntax checkbox to view and change the schedule in Quartz Cron Syntax.

And then click Save!

Databricks lakeFS Integration

Databricks’ major competitive advantage is its ability to provide a unified platform for testable data processing pipelines. Implementing Databricks eliminates the need for data teams to invest in several different technologies, decreasing complexity and streamlining the analytics process.

Databricks is an important platform in current data ecosystems because of its emphasis on performance enhancement, speed capabilities, a diverse set of advanced analytics and machine learning tools, and a collaborative environment for data experts.

Databricks has basic version control features, but if you want to expand them, you may easily add an open-source technology, such as lakeFS – using the Databricks lakeFS integration.

lakeFS allows you to manage your data as code with Git-like processes (branching, merging, committing, and so on) to create reproducible, high-quality pipelines.

Here’s a step-by-step guide to setting lakeFS on Databricks. It demonstrates how simple it is to combine the two technologies and enjoye complete data version control capabilities.


Finally, we explained how to establish a data pipeline using Databricks. By following the procedures explained in this article, you may ingest, prepare, and analyze data using Databricks. Thanks to its powerful processing capabilities, Delta Lake storage layer, and SQL and Python notebooks, Databricks architecture offers a sophisticated platform for constructing data pipelines.

However, creating a data pipeline is merely the first step in using data to gain valuable insights. To fully appreciate the value of data, it is critical to constantly monitor and update the pipeline to ensure that it supplies accurate and relevant information. With the Databricks platform, you can monitor and adjust your data pipeline to ensure it meets your business requirements.

By integrating Databricks, you can create a scalable and efficient data pipeline to deliver insights and innovation for your firm. Whether you’re a data analyst, data scientist, or business executive, Databricks has the tools to flourish in a data-driven world. So, start developing your data pipeline today and harness the potential of your data!

Git for Data – lakeFS

  • Get Started
    Get Started
  • Who’s coming to Data+AI Summit? Meet the lakeFS team at Booth #69! Learn more about -

    lakeFS for Databricks