A Humane Python Test Runner

Erik Rose

I have a bit of a dot dependency.

As someone who spends a lot of time running tests on his Python code, my mind now self-administers a minute dopamine hit with every dot emitted by my test runner.

..........................................................................
..................
----------------------------------------------------------------------
Ran 93 tests in 30.182s

Ah, beautiful dots, each heralding a passed test. But dots have their downsides. For example, I dread early failures:

......F...................................................................
...................
======================================================================
FAIL: test_human_path (thing.tests.test_utils.UtilsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/erose/thing/tests/test_utils.py", line 14, in test_human_path
    assert False
AssertionError
----------------------------------------------------------------------
Ran 93 tests in 30.030s

After that F, I have to wait another 25 seconds before I get the skinny on the error. Ugh. Sure, I could ask nose to stop after the first failure (--stop), but then I wouldn’t get the chance to recognize patterns evinced by sets of similar failures.

Another problem: once I fix the failure, how do I re-run just that test? The obvious copy-and-paste suggested by my visual cortex doesn’t work:

% nosetests thing.tests.test_utils.UtilsTests
----------------------------------------------------------------------
Ran 0 tests in 0.000s

The line we actually need requires manual intervention, changing a period to a colon and reordering a few things:

% nosetests thing.tests.test_utils:UtilsTests.test_human_path

These and several other productivity-sappers drove me to drink (sparkling apple juice) and consider what I could do about them. After all, I had some free time while tests ran—support.mozilla.com has over 1000 of them, taking almost 4 minutes in all. My answer is nose-progressive, a more human-centric test runner that plugs into the nose testing framework. Its goal? Make the best use of your perceptual apparatus and your interaction time to speed up the debugging process.

A Few Tempting Nuggets

Here are a few of nose-progressive’s features that smooth out the testing workflow.

Quicker, Slimmer Tracebacks

nose-progressive shows its tracebacks immediately; you don’t have to wait for the test run to complete. Its output looks like this:

FAIL: kitsune.apps.notifications.tests.test_events:MailTests.test_anonymous
       vi +361 apps/notifications/tests/test_events.py
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/Users/erose/Checkouts/kitsune/../kitsune/apps/notifications/tests/test_events.py", line 361, in test_anonymous
    eq_(1, len(mail.outbox))
  File "/Users/erose/Checkouts/kitsune/vendor/packages/nose/nose/tools.py", line 31, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: 1 != 0

ERROR: kitsune.apps.questions.tests.test_templates:TemplateTestCase.test_woo
        vi +494 apps/questions/tests/test_templates.py
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/Users/erose/Checkouts/kitsune/vendor/packages/mock/mock.py", line 196, in patched
    return func(*args, **keywargs)
  File "/Users/erose/Checkouts/kitsune/../kitsune/apps/questions/tests/test_templates.py", line 494, in test_woo
    attrs_eq(mail.outbox[0], to=['some@bo.dy'],
IndexError: list index out of range

It strips 3 unnecessary lines off each traceback to make better use of precious terminal space: two lines worth of dividers, plus the “Traceback (most recent call last)” which is totally superfluous after you’ve been using Python for a few days. The bold does a fine job of visually delimiting tracebacks.

Holy Wombats, a Progress Bar

But where are the dots? My thoughts harkened back to the Zope test runner. It had a lovely percentage-done display that told me exactly how much apple juice I had time to drink. Why couldn’t we have that in nose but without madly scrolling my tracebacks-so-far off the screen? Drumroll, please:

thing.tests.test_templates:TaggingTests.test_add_new         [===========/  ]

That little bar on the right ticks and spins along merrily, indicating the completeness of the test run, while the path on the left names the currently running test. The whole thing lives down at the bottom of the terminal, while tracebacks scroll gaily upwards above it. If you have a lot of errors, tracebacks that push up above the top of the terminal slide helpfully into your scrollback buffer; I sidestepped curses and did the terminal addressing myself so you won’t lose anything.

Editor Integration

Did you notice those funny lines at the tops of the tracebacks?

vi +361 apps/notifications/tests/test_events.py

Triple-click, copy, and paste that line, and zoom!—you’re plunked down into your editor of choice right at your failing test. It even applies some heuristics to find the stack frame of your actual test rather than placing you unhelpfully in the middle of a nose helper function like eq_(). And you’re by no means limited to vi; nose-progressive obeys your $EDITOR environment variable. So far, it’s known to work with vi, emacs, and BBEdit.

Re-running Failed Tests

In case you want to re-run a specific failing test as above, nose-progressive gives you just what you need. Copy the pathname from this line…

FAIL: kitsune.apps.notifications.tests.test_events:MailTests.test_anonymous

…and paste it into your invocation of the test runner:

% nosetests kitsune.apps.notifications.tests.test_events:MailTests.test_anonymous

nose-progressive also gets along famously with nose’s --failed option in case you want to re-run all failing tests.

Getting It

Trying nose-progressive is a matter of two commands:

easy_install nose-progressive
nosetests --with-progressive --logging-clear-handlers

For more details, including how to use it with Django, check out nose-progressive’s PyPI page. Happy testing!