Testing

DASCore’s test suite is run with pytest. While in the base dascore repo (and after installing DASCore for development) invoke pytest from the command line:

pytest tests

You can also use the cov flags to check coverage. Please make sure you don’t introduce large blocks of dead code.

pytest tests --cov dascore --cov-report term-missing

The suite is parallel safe, and CI runs it that way. Locally it is the quickest way to run the whole thing:

pytest tests -n logical --dist loadfile

CI requires 100% coverage of the combined data from every operating system rather than from any one run, so a Linux-only run can report a line or two missing which macOS and Windows cover.

If you would like to test the IO modules it can be done like so:

pytest tests/test_io

Or a particular IO module:

pytest tests/test_io/test_dasdae.py

Pytest is highly configurable and has some rather useful flags such as -s, -x, and –pdb (especially with pdbpp).

To run the docstring tests use the following:

pytest dascore --doctest-modules

To validate executable examples in the hand-written documentation without building the full site, generate the mirrored tests with:

python scripts/generate_doc_code_tests.py

Then run the generated tests:

pytest tests/test_autogenerated_doccode

The tests/test_autogenerated_doccode directory is intentionally gitignored and should be regenerated locally rather than committed.

Checking answers have not changed

Rewriting a function is only safe if it still returns what it returned before, and the test suite cannot settle that on its own: it checks what someone thought to assert, and it changes along with the code. scripts/differential_check.py settles it directly. It runs a list of calls against a checkout of any git ref and against the working tree, fingerprints every result, and reports which differ.

python scripts/differential_check.py --ref dev

Nothing is compared approximately. Data and coordinates are hashed, so a change in the last bit is a change:

abs: differs in ['data_hash']
    data_hash
      before: 17cc03bd989ebd09eb4d8ccb842dd526
      after:  cd1928741a7ac2fa4789853cf430661d

The script compares two lists. get_calls holds calls against the example patches, which carry datetime coordinates, units, and complex data from a transform. MATRIX_CALLS is run against every array make_arrays builds, so each call is checked for every dtype and for the values implementations tend to disagree about: nan, infinities, a whole slice of nulls, and numbers large enough to overflow. Add to whichever fits when a function is rewritten; one which isn’t listed isn’t checked.

The matrix is worth the runtime. A hand written list of calls missed a rewritten Patch.where returning float32 where it used to return float64, because the example patch happens to be float64; running the same call against a float32 array found it immediately.

Writing Tests

Tests should go into the tests/ folder, which mirrors the structure of the main package. For example, if you are writing tests for dascore.Patch, whose class definition is located in dascore/core/patch it should go in tests/test_core/test_patch.py.

In general, tests should be grouped together in classes. Fixtures go as close as possible to the test(s) that need them, moving from class, module, and then to conftest. Checkout the pytest documentation for a review on fixtures (and why to use them).