unitizer - Interactive R Unit Tests

Brodie Gaslam

Introduction

TL;DR

unitizer simplifies creating, reviewing, and debugging unit tests in R. To install:

install.packages('unitizer')

Please keep in mind this is an experimental framework that has been thoroughly tested by one person.

unitizer bakes in a lot of contextual help so you can get started without reading all the documentation. Try the demo to get an idea:

library(unitizer)
demo(unitizer)

Or check out the screencast to see unitizer in action.

Why Another Testing Framework?

Automated Test Formalization

Are you tired of the deparse/dput then copy-paste R objects into test file dance, or do you use testthat::expect_equal_to_reference a lot?

With unitizer you review function output at an interactive prompt as you would with informal tests. You then store the value, conditions ( e.g. warnings, etc.), and environment for use as the reference values in formal tests, all with a single keystroke.

Streamlined Debugging

Do you wish the nature of a test failure was more immediately obvious?

When tests fail, you are shown a proper diff so you can clearly identify how the test failed:

diff example

diff example

Do you wish that you could start debugging your failed tests without additional set-up work?

unitizer drops you in the test environment so you can debug why the test failed without further ado:

review example

review example

Fast Test Updates

Do you avoid improvements to your functions because that would require painstakingly updating many tests?

The diffs for the failed tests let you immediately confirm only what you intended changed. Then you can update each test with a single keystroke.

Usage

unitizer stores R expressions and the result of evaluating them so that it can detect code regressions. This is akin to saving test output to a .Rout.save file as documented in Writing R Extensions, except that we’re storing the actual R objects and it is much easier to review them.

To use unitizer:

unitizer can run in a non-interactive mode for use with R CMD check.

Documentation

How Does unitizer Differ from testthat?

Testing Style

unitizer requires you to review test outputs and confirm they are as expected. testthat requires you to assert what the test outputs should be beforehand. There are trade-offs between these strategies that we illustrate here, first with testthat:

vec <- c(10, -10, 0, .1, Inf, NA)
expect_error(
  log10(letters),
  "Error in log10\\(letters\\) : non-numeric argument to mathematical function\n"
)
expect_equal(log10(vec), c(1, NaN, -Inf, -1, Inf, NA))
expect_warning(log10(vec), "NaNs produced")

And with unitizer:

vec <- c(10, -10, 0, .1, Inf, NA)
log10(letters)                            # input error
log10(vec)                                # succeed with warnings

These two unit test implementations are functionally equivalent. There are benefits to both approaches. In favor of unitizer:

In favor of testthat:

unitizer is particularly convenient when the tests return complex objects (e.g as lm does) or produce conditions. There is no need for complicated assertions involving deparsed objects.

Converting testthat tests to unitizer

If you have a stable set of tests it is probably not worth trying to convert them to unitizer unless you expect the code those tests cover to change substantially. If you do decide to convert tests you can use the provided testthat_translate* functions (see ?testthat_translate_file).

unitizer and Packages

The simplest way to use unitizer as part of your package development process is to create a tests/unitizer folder for all your unitizer test scripts. Here is a sample test structure from the demo package:

unitizer.fastlm/         # top level package directory
    R/
    tests/
        run.R            # <- calls `unitize` or `unitize_dir`
        unitizer/
            fastlm.R
            cornerCases.R

And this is what the tests/run.R file would look like

library(unitizer)
unitize("unitizer/fastlm.R")
unitize("unitizer/cornerCases.R")

or equivalently

library(unitizer)
unitize_dir("unitizer")

The path specification for test files should be relative to the tests directory as that is what R CMD check uses. When unitize is run by R CMD check it will run in a non-interactive mode that will succeed only if all tests pass.

You can use any folder name for your tests, but if you use “tests/unitizer” unitize will look for files automatically, so the following work assuming your working directory is a folder within the package:

unitize_dir()          # same as `unitize_dir("unitizer")`
unitize("fast")        # same as `unitize("fastlm.R")`
unitize()              # Will prompt for a file to `unitize`

Remember to include unitizer as a “suggests” package in your DESCRIPTION file.

Things You Should Know About unitizer

unitizer Writes To Your Filesystem

The unitized tests need to be saved someplace, and the default action is to save to the same directory as the test file. You will always be prompted by unitizer before it writes to your file system. See storing unitized tests for implications and alternatives.

Tests Pass If They all.equal Stored Reference Values

Once you have created your first unitizer with unitize, subsequent calls to unitize will compare the old stored value to the new one using all.equal. You can change the comparison function by using unitizer_sect (see tests vignette).

Test Expressions Are Stored Deparsed

This means you need to be careful with expressions that may deparse differently on different machines. For example, in order to avoid round issues with numerics, it is better to use:

Instead of:

Increase Reproducibility with Advanced State Management

unitizer can track and manage many aspects of state to make your tests more reproducible. For example, unitizer can reset your search path to what is is found in a fresh R session prior to running tests to avoid conflicts with whatever libraries you happen to have loaded at the time. Your session state is restored when unitizer exits. The following aspects of state can be actively tracked and managed:

State management is turned off by default because it requires tracing some base functions which is against CRAN policy. If you wish to enable this feature use unitize(..., state='recommended') or options(unitizer.state='recommended'). For more details see ?unitizerState and the reproducible tests vignette.

Beware of Force Quitting from unitizer

If you interrupt evaluation with CTRL+C (or with ESC in RStudio), or if you browser/debug and quit with ‘Q’, you will exit unitizer with no opportunity to save any modifications you made during unitizer review. Make sure you quit by typing ‘Q’ at the unitizer prompt. If you are in browser, you will need to let the browsed function finish evaluation to return to the unitizer prompt, and only then quit.

Reference Objects

Tests that modify objects by reference are not perfectly suited for use with unitizer. The tests will work fine, but unitizer will only be able to show you the most recent version of the reference object when you review a test, not what it was like when the test was evaluated. This is only an issue with reference objects that are modified (e.g. environments, RC objects, data.table modified with := or set*).

unitizer Is Complex

In order to re-create the feel of the R prompt within unitizer we resorted to a fair bit of trickery. For the most part this should be transparent to the user, but you should be aware it exists in the event something unexpected happens that exposes it. Here is a non-exhaustive list of some of the tricky things we do:

Avoid Tests That Require User Input

In particular, you should avoid evaluating tests that invoke debugged functions, or introducing interactivity by using something like options(error=recover), or readline, or some such. Tests will work, but the interaction will be challenging because you will have to do it with stderr and stdout captured…

Avoid running unitize within try / tryCatch Blocks

Doing so will cause unitize to quit if any test expressions throw conditions. See discussion in error handling.

Some Base Functions are Masked at the unitizer Prompt

q and quit are masked to give the user an opportunity to cancel the quit action in case they meant to quit from unitizer instead of R. Use Q to quit from unitizer, as you would from browser.

ls is masked with a specialized version for use in unitizer.

In both cases you can still access the original functions by preceding them with base::

See miscellanous topics vignette.