Ready to dive into the lake?
lakeFS is currently only
available on desktop.

For an optimal experience, provide your email below and one of our lifeguards will send you a link to start swimming in the lake!

lakeFS Community
Idan Novogroder
Idan Novogroder Author

Idan has an extensive background in software and DevOps engineering....

Last updated on July 3, 2024

Quality can start from the moment you write your code in a notebook. Unit testing is a great approach to making the code in your notebooks more consistent and of higher quality. 

In general, unit testing – the practice of testing self-contained code units, such as functions, frequently and early – is a good practice to implement early on. It helps you identify issues with your code faster, detect false assumptions about it earlier, and organize your development efforts.

What are conventional unit tests and how can you implement them in notebooks? Read this article to discover the benefits of unit testing and get a step-by-step guide to creating and running a unit test in your notebook.

What is Unit Testing and Its Benefits?

Unit testing, also known as functional testing, is a testing method used to validate individual source code units. Testing is crucial because it lets you check your code for any issues. Writing unit tests equips your code with a safety net because unit tests are designed to notify you when something breaks so you can repair it.

The primary goal of unit testing is to make sure that any modifications or upgrades to the code don’t result in unforeseen errors by confirming that each individual module, such as functions and procedures, is operating as intended. 

Key benefits of unit testing:

  • It enhances the quality of the code by checking if it is correct by testing each module and feature of an application
  • It speeds up the development process
  • Early detection of defects in the development cycle reduces expenses
  • Developers are better able to grasp their code base thanks to this testing, which speeds up bug fixes.
  • Unit testing allows for the reuse of code.
  • In the long term, it helps save time and resources because it assists in detecting coding flaws relatively early.

Why Is Unit Testing Essential for Overcoming Data Science Challenges?

Unit testing is a crucial step in assuring consistency between the final product and the input specification. It also plays a key role in reducing uncertainty in a data project. 

A data scientist or data engineer will typically conduct unit tests during the development stage of an application to check that the system operates as intended and that the outputs are consistent with the input data.

Tools and Frameworks for Unit Testing

1. unittest and doctest in Jupyter Notebooks

You can use Python standard testing tools, such as doctest and unittest, directly in a notebook.

The primary source of inspiration for the unittest unit testing framework was JUnit, and it has characteristics similar to popular unit testing frameworks written in other languages. It facilitates test automation, test independence from the reporting system, sharing of test setup and shutdown code, and test collection aggregation.

The doctest module looks for passages of text that appear to be interactive Python sessions. It then runs those sessions to make sure everything operates as it should. There are numerous typical applications for doctest:

  • Checking that all interactive examples continue to function as described to ensure that a module’s docstrings remain current.
  • Performing regression testing to confirm that interactive samples from a test object or test file function as intended.
  • Create instructive documentation, heavily reliant on input-output examples, for a package. This has the flavor of either “executable documentation” or “literate testing,” depending on whether the examples or the explanatory text are highlighted more.

Are you considering using another notebook? Check out these Jupyter notebook alternatives.

2. Nutter for Databricks

You can easily test Databricks notebooks thanks to the Nutter framework. In addition to facilitating easy integration with Azure DevOps Build/Release pipelines, the framework allows for a basic inner dev loop. To test a notebook, data or machine learning engineers just need to build a test notebook called test_.

Nutter consists of two primary parts:

  • Nutter CLI is the client CLI that you can load on your machine or a build agent. 
  • Nutter Runner is the server-side component that is installed as a library on the Databricks cluster.

You can execute tests from the Nutter CLI, which helps integrate into Build/Release processes, or directly within that notebook.

3. Testbook

For testing code in Jupyter Notebooks, the testbook is an extension of the unit testing framework.

In earlier iterations of unit testing notebooks, tests were written within the notebook. testbook, on the other hand, treats.ipynb files as.py files and enables unit tests to be executed against notebooks in separate test files.

Best Practices for Notebook Testing

Modularize your code

As a project grows, modularization features become key to avoiding duplication of code and promoting reusability.

Modularization has various benefits:

What Benefit
Reusability Modules don’t require rewriting to be utilized in other projects
Maintainability Smaller, more focused modules make updates and troubleshooting easier
Scalability Enables effective scaling as a project expands
Collaboration Enables multiple developers to work at once
Testing Makes unit testing easier, which leads to code that is more dependable
Readability By concentrating on particular activities, it enhances code understanding

Creating a notebook with shared functionality and running it at the start of every notebook is one way to accomplish this. As an alternative, you can write modules that enable the use of import commands in Python, just like in standard Python programming. Long chunks of code can be made more reusable and easier to test by being divided into functions.

A folder with files containing classes or functions and a __init__.py file is a must for using modules in Databricks. 

Put functions and unit tests in order

You can use notebooks to organize your functions and their unit tests in a few typical ways. Every strategy has advantages and disadvantages.

Common methods for Python, R, and Scala notebooks are as follows:

1. Keep unit tests and store functions separate from notebooks

These functions can be used both inside and outside of notebooks, and testing outside of notebooks can be done really well using test frameworks.

However, Scala notebooks are not compatible with this approach. This method also makes it harder to keep track of and manage as many files as possible.

2. Keep functions in one notebook and their corresponding unit tests in another one

This approach makes it easier to use these features on different notebooks.

On the other hand, you’ll end up with more notebooks to keep track of and manage. These features are limited to notebook use. It may also be more challenging to test these features outside of notebooks.

3. Keep unit tests for stored functions in the same notebook

Unit tests and functions are kept together in a single notebook and are simple to track and maintain.  

However, reusing these functions across notebooks may be more challenging. These features are limited to notebook use. It may also be more challenging to test these features outside of notebooks.

Databricks advises keeping functions and their unit tests outside of notebooks for Python and R notebooks. The company furthermore suggests keeping function unit tests in a separate notebook and function notebooks together for Scala notebooks.

Databricks also advises storing functions as SQL user-defined functions (SQL UDFs) in your schemas, or databases, when using SQL notebooks. Then, from SQL notebooks, you can call these SQL UDFs and their corresponding unit tests.

How Data Version Control Works With Unit Testing for Notebooks

In Databricks notebooks, you get a feature called Revision History. It provides you with a version history that lets you see and restore earlier notebook snapshots. Not only can you post comments, but you can also delete the version history and restore and remove versions.

It’s easy to integrate Databricks with Git, and you can synchronize Notebook work with a remote Git repository.

Click the version history button located in the right sidebar to examine the history of notebook versions. Ensure that the file is saved. The version in the notebook that has the provided comment will be kept.

Check out lakeFS samples on Github to see how this plays out in lakeFS.

CI/CD Integration for Notebooks

Continuous integration and delivery, or CI/CD, is the process of creating and distributing software using automated pipelines in brief, frequent cycles. CI/CD is ubiquitous in software development and is becoming more and more important in data science and data engineering. 

Development teams can produce releases more consistently by automating the building, testing, and deploying of code as opposed to using the manual procedures that are still frequently used by data science and data engineering teams.

A Databricks notebook is a crucial component of processes for data science and data engineering. As part of a CI/CD workflow, notebooks can be validated and tested in addition to being subject to version control. Notebooks can be subjected to automated tests to verify if they are operating as intended.

Using Azure DevOps

Databricks recommends the following process for CI/CD development using Azure DevOps:

  1. With your third-party Git provider, create a repository or use an already-existing repository.
  2. Connect the same third-party repository to your local development machine. 
  3. Take any current updated artifacts from the third-party repository and transfer them to your local development machine, including code files, build scripts, and notebooks.
  4. Create, modify, and test artifacts on your local development system as needed. Next, push any updated or new artifacts from your local development system to the third-party repository. 
  5. As necessary, repeat steps 3 and 4.
  6. Periodically use Azure DevOps as an integrated method to develop, test, and execute code on your Azure Databricks workspace, automatically download artifacts from your third-party repository, and report test and run results. 

Although Azure DevOps can be executed manually, in practical applications, you would direct your external Git provider to initiate Azure DevOps each time a particular event occurs, such as a pull request for a repository.

Conclusion

You can boost the quality of your code in notebooks by running unit tests inside them. Unit testing for notebooks brings you a host of benefits, from quickly identifying issues with your code to better organization of your project.

Git for Data – lakeFS

  • Get Started
    Get Started
  • Did you know that lakeFS is an official Databricks Technology Partner? Learn more about -

    lakeFS for Databricks
    +