Unit testing Python



Dmitry Tantsur (Principal Software Engineer, Red Hat)

Slides: dtantsur.github.io/talks/berlin-python-unittest

Code: github.com/dtantsur/berlin-python-unittest

Agenda

  • Automated testing: how and why?
  • Writing and running Python tests:
    • The unittest package
    • Discovering tests
    • Assertion methods
  • Mocking:
    • Mock and MagicMock
    • Patching objects
  • Extras:
    • Spec and autospec
    • Additional runners
    • Measuring coverage

Automated testing

How and why?

Automated testing

What?

Using a program to verify correctness of a program (library, module, any piece of software).

Automated testing

Why?

  • Decisiveness
  • Repeatability
  • Parallelism
  • Coverage

Automated testing

Types

  • Black box
  • White box

Automated testing

Types

  • Unit
  • Integration
  • Functional

Automated testing

Unit testing

Testing types: unit

Usually white box

Automated testing

Integration testing

Testing types: integration

Usually black box

Automated testing

Functional testing

Testing types: functional

Black box

Automated testing

Why unit test?

  • Sanity-check during development
  • Help newcomers keep things working
  • Document design decision
  • Sign of the maturity of the project

Automated testing

Why unit test?

Unit tests do not:

  • Guarantee that your code works
  • Replace integration testing
  • Replace developer's guide
  • Have to cover literally everything

Unit test framework

Unit test framework

  • Included in the standard library
  • Heavily extended in Python 3
  • Plenty 3rd party addons

Unit test framework

Case study: quadratic equation

Package layout:

$ find my_utils
my_utils
my_utils/roots.py
my_utils/__init__.py
my_utils/tests
my_utils/tests/__init__.py
my_utils/tests/test_roots.py

Unit test framework

Case study: quadratic equation

my_utils/roots.py
import math

def roots(a, b, c):
    if not a:
        raise ValueError("a cannot be zero")

    discriminant = b ** 2 - 4 * a * c
    if discriminant < 0:
        raise ValueError("discriminant below zero")

    return ((- b - math.sqrt(discriminant)) / 2 / a,
            (- b + math.sqrt(discriminant)) / 2 / a)

Writing unit tests

  • Split the testing into independent checks
  • Group individual checks into cases
  • Don't forget negative tests

Unit test framework

Case study: quadratic equation

my_utils/tests/test_roots.py
import unittest
from my_utils import roots

class RootsTest(unittest.TestCase):
    def test_correct(self):
        self.assertEqual(roots.roots(1, -3, 2),
                         (1.0, 2.0))

    def test_negative_a(self):
        self.assertRaises(ValueError,
                          roots.roots, 0, -3, 2)

    def test_negative_discriminant(self):
        self.assertRaises(ValueError,
                          roots.roots, 1, 1, 1)

Unit test framework

Case study: quadratic equation

Running only one test file:

$ python3 -m unittest my_utils.tests.test_roots
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Unit test framework

Case study: quadratic equation

Detecting and running all tests:

$ python3 -m unittest discover my_utils
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Unit test framework

Case study: quadratic equation

And this is how it fails:

$ python3 -m unittest discover my_utils
F..
======================================================================
FAIL: test_correct (tests.test_roots.RootsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dtantsur/Projects/berlin-python-unittest/my_utils/tests/test_roots.py", line 10, in test_correct
    self.assertEqual(roots(1, -3, 2), (1.0, 2.0))
AssertionError: Tuples differ: (1.0, 1.0) != (1.0, 2.0)

First differing element 1:
1.0
2.0

- (1.0, 1.0)
?       ^

+ (1.0, 2.0)
?       ^


----------------------------------------------------------------------
Ran 3 tests in 0.002s

FAILED (failures=1)
    

Unit test framework

Quadratic equation: imports

my_utils/tests/test_roots.py
import unittest
from my_utils.roots import roots

Importing the standard unittest package and our code that we plan on testing.

Unit test framework

Quadratic equation: test case

my_utils/tests/test_roots.py
class RootsTest(unittest.TestCase):
    def test_correct(self):
        # ...

    def test_negative_a(self):
        # ...

    # ...
  • A test case is a logical group of tests
  • Each test is a method starting with test_
  • Each tests one aspect of the tested entity

Unit test framework

Quadratic equation: equality

my_utils/tests/test_roots.py
def test_correct(self):
    self.assertEqual(roots.roots(1, -3, 2),
                     (1.0, 2.0))
  1. Calculate the value using the function under test
  2. Compare it with the known correct result

Using convenience methods self.assert<***>

Unit test framework

Quadratic equation: errors

my_utils/tests/test_roots.py
def test_negative_a(self):
    self.assertRaises(ValueError,
                      roots.roots, 0, -3, 2)

def test_negative_discriminant(self):
    self.assertRaises(ValueError,
                      roots.roots, 1, 1, 1)
  • Use assertRaises to check that a callable raises an exception on invalid input
  • Note that we do not call it ourselves!

Unit test framework

Quadratic equation: errors

my_utils/tests/test_roots.py
def test_negative_a(self):
    self.assertRaisesRegex(ValueError, 'a cannot be zero',
                           roots.roots, 0, -3, 2)

def test_negative_discriminant(self):
    self.assertRaisesRegex(ValueError, 'discriminant below zero',
                           roots.roots, 1, 1, 1)
  • Use assertRaisesRegex to validate the error message
  • Useful to distinguish between different cases of the same error

Unit test framework

Available checks

assertEqual, assertNotEqual, assertIs, assertIsNot, assertTrue, assertFalse, assertIsInstance, assertNotIsInstance, assertIn, assertNotIn, assertGreater, assertGreaterEqual, assertLess, assertLessEqual, assertRegex, assertNotRegex, assertRaises, assertRaisesRegex, ...

And even more in 3rd party libraries

Unit test framework

Preparing for tests

class RootsTest(unittest.TestCase):

    def setUp(self):
        # Gets run before every test

    def tearDown(self):
        # Gets run after every test

Also use addCleanup to clean up after tests.

Mocking

Handling integration points

Mocking

Problem statement

How to test code that relies on (a lot of) other code?

How to test code that relies on something not available in a regular testing environment?

Mocking

The mock library

Python has the mock library:

  • unittest.mock in Python 3 standard library
  • just mock on PyPI for Python 2 and 3

Mocking

Case study: quadratic equation

my_utils/roots.py
import sys

def main():
    try:
        a = int(sys.argv[1])
        b = int(sys.argv[2])
        c = int(sys.argv[3])
    except IndexError:
        sys.exit('3 arguments required')
    except ValueError:
        sys.exit('all arguments must be integers')
    print(roots(a, b, c))

if __name__ == '__main__':
    main()

Mocking

Case study: quadratic equation

$ python3 -m my_utils.roots
3 arguments required

$ python3 -m my_utils.roots a b c
all arguments must be integers

$ python3 -m my_utils.roots 1 4 4
(-2.0, -2.0)

Mocking

Case study: quadratic equation

Problems:

  • provide values for sys.argv
  • test how sys.exit is called
  • test how print is called

Mocking

patch function

The unittest.mock.patch function allows to replace an object temporary while a test is running and restore it back.

Can be used in many forms, we will only consider some of them.

Mocking

Case study: quadratic equation

my_utils/tests/test_roots.py
from unittest import mock

class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_correct(self):
        roots.main()

Using patch as decorator to automate patching object and restoring it after the test.

Mocking

Case study: quadratic equation

Rough equivalent:

class MainTest(unittest.TestCase):

    def test_correct(self):
        import sys
        old_argv = sys.argv
        sys.argv = [None, '1', '-3', '2']
        try:
            roots.main()  # our actual test
        finally:
            sys.argv = old_argv

Mocking

patch function

This test does not verify that main function does anything.

To verify that printing is done correctly, we need to replace the print function with something that will track calls.

Mocking

Mock objects

Magic objects that allow any operations on them and record them for future verification.

m = mock.Mock()
r = m.abc(42, cat='meow')
assert isinstance(m.abc, mock.Mock)
assert isinstance(r, mock.Mock)

m.abc.assert_called_once_with(42, cat='meow')
assert r is m.abc.return_value

Mocking

Case study: quadratic equation

class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    @mock.patch('builtins.print')
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

Mocking

Case study: quadratic equation

If we make a mistake, the test will tell us:

$ python3 -m unittest discover my_utils
F...
======================================================================
FAIL: test_correct (tests.test_roots.MainTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.6/unittest/mock.py", line 1179, in patched
    return func(*args, **keywargs)
  File "/home/dtantsur/Projects/berlin-python-unittest/my_utils/tests/test_roots.py", line 27, in test_correct
    mock_print.assert_called_once_with((2.0, 2.0))
  File "/usr/lib64/python3.6/unittest/mock.py", line 825, in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
  File "/usr/lib64/python3.6/unittest/mock.py", line 814, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: print((2.0, 2.0))
Actual call: print((1.0, 2.0))

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1)

Mocking

Case study: quadratic equation

Handling sys.exit:

  • We need to replace it with Mock
  • But we also make sure to stop function from executing after calling it

We can make it raise an exception!

Mocking

Case study: quadratic equation

@mock.patch('builtins.print')
class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

    @mock.patch('sys.exit')
    @mock.patch('sys.argv', [None, '1', '-3'])
    def test_missing_argument(self, mock_exit, mock_print):
        mock_exit.side_effect = RuntimeError
        self.assertRaises(RuntimeError, roots.main)
        mock_exit.assert_called_once_with(
            '3 arguments required')
        mock_print.assert_not_called()

Mocking

Summary

mock.patch:

  • Can be used on methods and classes
  • Can replace an object with a given object or with a new mock
  • Passes the newly created mock to test methods

Mocking

Summary

Mock:

  • Any attribute is also a Mock
  • Can be called, result is a Mock
  • Can raise exceptions
  • Can return specified values (via setting return_value)

Questions?

Next: advanced topics

Spec and autospec

Spec and autospec

Problem statement

Mock objects can simulate anything.

How to make them simulate a specific object or function?

Spec and autospec

Mock specs

A Mock object accepts a spec - a simulated object.

mock_spec_demo.py
from unittest import mock
class A:
    def x(self, y):
        return y ** 2

m = mock.Mock(spec=A)
print(m.x)
print(m.y)

Spec and autospec

Mock specs

$ python3 mock_spec_demo.py
<Mock name='mock.x' id='140567812205928'>
Traceback (most recent call last):
  File "mock_spec_demo.py", line 11, in <module>
    print(m.y)
  File "/usr/lib64/python3.6/unittest/mock.py", line 582, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'y'

Spec and autospec

patch autospec

The autospec argument of the patch function can even check function signatures:.

mock_autospec_demo.py
class A:
    def x(self, y):
        return y ** 2

@mock.patch.object(A, 'x', autospec=True)
def test(mock_x):
    a = A()
    print(a.x(42))
    print(a.x(z=42))

test()

Spec and autospec

Mock specs

$ python3 mock_autospec_demo.py
<MagicMock name='x()' id='140700038039648'>
Traceback (most recent call last):
  File "mock_autospec_demo.py", line 16, in <module>
    test()
  File "/usr/lib64/python3.6/unittest/mock.py", line 1179, in patched
    return func(*args, **keywargs)
  File "mock_autospec_demo.py", line 13, in test
    print(a.x(z=42))
  File "<string>", line 2, in x
  File "/usr/lib64/python3.6/unittest/mock.py", line 171, in checksig
    sig.bind(*args, **kwargs)
  File "/usr/lib64/python3.6/inspect.py", line 2969, in bind
    return args[0]._bind(args[1:], kwargs)
  File "/usr/lib64/python3.6/inspect.py", line 2884, in _bind
    raise TypeError(msg) from None
TypeError: missing a required argument: 'y'

Spec and autospec

Case study: quadratic equation

@mock.patch('builtins.print', autospec=True)
class MainTest(unittest.TestCase):

    @mock.patch('sys.argv', [None, '1', '-3', '2'])
    def test_correct(self, mock_print):
        roots.main()
        mock_print.assert_called_once_with((1.0, 2.0))

    @mock.patch('sys.exit', autospec=True)
    @mock.patch('sys.argv', [None, '1', '-3'])
    def test_missing_argument(self, mock_exit, mock_print):
        mock_exit.side_effect = RuntimeError
        self.assertRaises(RuntimeError, roots.main)
        mock_exit.assert_called_once_with(
            '3 arguments required')
        mock_print.assert_not_called()

Additional runners

Additional runners

Running with python3 -m unittest is quite convenient.

But there are more feature-rich runners for Python unit tests.

Additional runners

Nose

nose.readthedocs.io

  • A lot of plugins for integration with other tools
  • Select specific tests to run
  • Additional options controlling output

Additional runners

Nose

$ nosetests-3
.....
----------------------------------------------------------------------
Ran 5 tests in 0.013s

OK

Additional runners

stestr

stestr.readthedocs.io

  • Select specific tests to run via regular expressions
  • Machine-parseable output in subunit format
  • Emphasis on parallel execution and streaming results
  • Execution time reporting

Additional runners

stestr

$ stestr-3 --test-path my_utils/tests/ run
{2} my_utils.tests.test_roots.RootsTest.test_correct [0.000341s] ... ok
{3} my_utils.tests.test_roots.RootsTest.test_negative_a [0.000647s] ... ok
{0} my_utils.tests.test_roots.MainTest.test_correct [0.011246s] ... ok
{0} my_utils.tests.test_roots.MainTest.test_missing_argument [0.004206s] ... ok
{1} my_utils.tests.test_roots.RootsTest.test_negative_discriminant [0.000660s] ... ok

======
Totals
======
Ran: 5 tests in 0.4467 sec.
 - Passed: 5
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 0.0171 sec.

==============
Worker Balance
==============
 - Worker 0 (2 tests) => 0:00:00.015929
 - Worker 1 (1 tests) => 0:00:00.000660
 - Worker 2 (1 tests) => 0:00:00.000341
 - Worker 3 (1 tests) => 0:00:00.000647

Coverage

Coverage

Problem statement

I want to know how much of my code is covered by unit tests.

The answer is the coverage utility coverage.readthedocs.io.

Coverage

Collect coverage

$ coverage3 run -m unittest discover my_utils
.....
----------------------------------------------------------------------
Ran 5 tests in 0.012s

OK

Coverage

Report coverage

$ coverage3 report
Name                           Stmts   Miss  Cover
--------------------------------------------------
my_utils/__init__.py               0      0   100%
my_utils/roots.py                 21      3    86%
my_utils/tests/__init__.py         0      0   100%
my_utils/tests/test_roots.py      21      0   100%
--------------------------------------------------
TOTAL                             42      3    93%

Coverage

Rich report

Collect also branch information:

$ coverage3 run --branch -m unittest discover my_utils
.....
----------------------------------------------------------------------
Ran 5 tests in 0.012s

OK

Coverage

Rich report

Show what is not covered:

$ coverage3 report -m
Name                           Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------
my_utils/__init__.py               0      0      0      0   100%
my_utils/roots.py                 21      3      8      2    83%   24-25, 30, 22->24, 29->30
my_utils/tests/__init__.py         0      0      0      0   100%
my_utils/tests/test_roots.py      21      0      2      0   100%
--------------------------------------------------------------------------
TOTAL                             42      3     10      2    90%

Questions?