Unit Testing

This section describes how the Kodo unit tests work, how to run them and how to extend them.

Overview

The purpose of the Kodo unit tests is to assert that the various Kodo features work across a number of different platforms and compilers. Having an extensive unit test suite makes it easier for developers to ensure that their changes work as expected and do not cause regressions in unexpected areas of the library.

The goal is that all features in Kodo should have a corresponding test case. Often a test case is added while when implementing a new feature. This makes it possible to assert that the new functionality works as expected as early as possible.

The unit tests are implemented using the Google C++ Testing Framework (gtest) which defines a bunch of helpers useful when writing tests. You can find more information on gtest on their homepage.

Running the tests

One of the first things you might want to try is to run the Kodo unit tests on your own machine. There are two ways to do this:

Run the test binary manually

The test binary is built using the waf build scripts shipped as part of the Kodo source code. You can read more about how you can get the source code and how to build it in the Getting Started section.

Once the code is built, the test binary will be located in a subfolder of the build folder that depends on your platform:

Linux
build/linux/test/kodo_tests
Mac OSX
build/darwin/test/kodo_tests
Windows
build/win32/test/kodo_tests.exe

If you are cross-compiling with an mkspec (as described in the Cross-compilation and Tool Options section), then the resulting binary will be located in:

mkspec
build/[mkspec]/test/kodo_tests

Note

Running the unit tests may take a long time on mobile and embedded platforms, since we test with an extensive set of parameters. You can lower the complexity and speed up the tests if you invoke the test binary with the embedded profile:

./kodo_tests --profile=embedded

Run the test as part of the build

In some cases it is convenient to run the test binary as part of a build. This can be done by passing the following option to waf:

python waf --run_tests

Adding a new test

When adding a new feature to Kodo it is a good idea to also add a corresponding unit test. The source code for the different unit tests are placed in the test/src folder of the Kodo project.

All files with a .cpp file extension in the test/src will automatically be included in the test executable produced when building Kodo with waf.

In general we follow these guidelines regarding the unit tests:

  1. Every class should have a corresponding unit test cpp file.
  2. Remember to place the test file as described in Namespaces and directories

The purpose of this is to make it easy to find the unit test for a specific class. In some cases it makes sense to have multiple classes tested in the same file. In those cases we still make a placeholder cpp file referring to the actual cpp file where the test can be found. An example of this can be seen for some of the codecs e.g. the class full_rlnc_encoder located in src/kodo/rlnc/full_rlnc_encoder.hpp is tested in full_rlnc_codes.cpp but the place-holder still exists.

The placeholder file in this cases (test/src/test_full_rlnc_encoder.cpp) looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
// Copyright Steinwurf ApS 2011.
// Distributed under the "STEINWURF RESEARCH LICENSE 1.0".
// See accompanying file LICENSE.rst or
// http://www.steinwurf.com/licensing

#include <kodo_rlnc/full_vector_encoder.hpp>

/// @file test_full_vector_encoder.cpp The unit tests for the
///       full_vector_encoder stack are defined in
///       test_full_vector_codes.cpp

Once the .cpp test file has been created we can start to implement the unit test code. This is done with the help of the gtest framework.

Dealing with type aliases

In some cases we have headers containing only type-aliases such as using statements. We currently do not require that these are explicitly unit tested.

Regardless of whether a unit test is implemented or not we still leave a .cpp file in the test folder.

An example is src/kodo/partial_mutable_shallow_storage_layers.hpp which only contains a single using declaration. It does not have any explicit unit tests but a place holder file is still created (test/src/test_mutable_partial_shallow_symbol_storage.cpp):

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
// Copyright Steinwurf ApS 2015.
// Distributed under the "STEINWURF RESEARCH LICENSE 1.0".
// See accompanying file LICENSE.rst or
// http://www.steinwurf.com/licensing

/// @file test_mutable_partial_shallow_symbol_storage.cpp Unit tests for the
///       mutable_partial_shallow_symbol_storage class

#include <kodo_core/mutable_partial_shallow_symbol_storage.hpp>

#include <cstdint>

#include <storage/storage.hpp>
#include <storage/mutable_storage.hpp>
#include <storage/is_same.hpp>

#include <gtest/gtest.h>
#include <stub/function.hpp>

namespace kodo_core
{
// Put dummy layers and tests classes in an anonymous namespace
// to avoid violations of ODF (one-definition-rule) in other
// translation units
namespace
{
/// Helper layer
class dummy_layer
{
public:

    template<class Factory>
    void construct(Factory& the_factory)
    {
        (void) the_factory;
        m_construct();
    }

    template<class Factory>
    void initialize(Factory& the_factory)
    {
        m_initialize();
        symbol_size.set_return(the_factory.symbol_size());
    }

    stub::function<void()> m_initialize;
    stub::function<void()> m_construct;

    stub::function<uint32_t()> block_size;
    stub::function<uint32_t()> symbol_size;
    stub::function<uint32_t()> symbols;
    stub::function<void(uint32_t, const storage::mutable_storage&)>
    set_mutable_symbol;
    stub::function<void(const storage::mutable_storage&)> set_mutable_symbols;
};

// Helper stack
class dummy_stack : public mutable_partial_shallow_symbol_storage<dummy_layer>
{ };

// Helper factory
class dummy_factory
{
public:
    stub::function<uint32_t()> max_symbol_size;
    stub::function<uint32_t()> symbol_size;
};
}
}

// Test that the stack functions properly when the partial symbol is
// not needed
TEST(test_mutable_partial_shallow_symbol_storage, no_partial)
{
    kodo_core::dummy_factory factory;

    factory.max_symbol_size.set_return(10U);

    uint32_t symbol_size = 5U;
    factory.symbol_size.set_return(symbol_size);

    kodo_core::dummy_stack stack;
    kodo_core::dummy_layer& layer = stack;

    stack.construct(factory);
    stack.initialize(factory);

    // This is initialize in the dummy layer using the factory
    EXPECT_EQ(stack.symbol_size(), 5U);

    // Check state is correct and that initialize and construct calls
    // were correctly forwarded
    EXPECT_TRUE(layer.m_construct.expect_calls().with().to_bool());
    EXPECT_TRUE(layer.m_initialize.expect_calls().with().to_bool());
    EXPECT_FALSE(stack.has_partial_symbol());

    // Set the remaining needed state in the stack
    uint32_t symbols = 5U;
    layer.symbols.set_return(symbols);
    uint32_t block_size = 25U;
    layer.block_size.set_return(block_size);

    // Make a buffer that perfectly fits 5 symbols of size 5 bytes
    std::vector<uint8_t> data(block_size);

    stack.set_mutable_symbols(storage::storage(data));
    EXPECT_FALSE(stack.has_partial_symbol());

    EXPECT_EQ(0U, layer.set_mutable_symbol.calls());

    EXPECT_TRUE(layer.set_mutable_symbols.expect_calls()
                .with(storage::storage(data))
                .to_bool());
}

// Test that the stack functions properly when the partial symbol is
// needed
TEST(test_mutable_partial_shallow_symbol_storage, partial)
{
    kodo_core::dummy_factory factory;

    factory.max_symbol_size.set_return(10U);
    uint32_t symbol_size = 5U;
    factory.symbol_size.set_return(symbol_size);

    kodo_core::dummy_stack stack;
    kodo_core::dummy_layer& layer = stack;

    stack.construct(factory);
    stack.initialize(factory);

    // This is initialize in the dummy layer using the factory
    EXPECT_EQ(stack.symbol_size(), 5U);

    // Check state is correct and that initialize and construct calls
    // were correctly forwarded
    EXPECT_TRUE(layer.m_construct.expect_calls().with().to_bool());
    EXPECT_TRUE(layer.m_initialize.expect_calls().with().to_bool());
    EXPECT_FALSE(stack.has_partial_symbol());

    // Set the remaining needed state in the stack
    uint32_t symbols = 5U;
    layer.symbols.set_return(symbols);
    uint32_t block_size = 25U;
    layer.block_size.set_return(block_size);

    // Make a buffer that is 23 bytes. That means we will have 4
    // symbols of 5 bytes and one symbol which only be 3 bytes.
    std::vector<uint8_t> data(block_size - 2U, 'a');

    stack.set_mutable_symbols(storage::storage(data));
    EXPECT_TRUE(stack.has_partial_symbol());

    // To check that the calls made to the set_mutable_symbols functions
    // are what we expect we need a custom predicate function to
    // compare the arguments. The reason is that we need to use the
    // is_same function to compare the storage::mutable_storage
    // objects. is_same compares that the pointers point to the same
    // memory as opposed to just checking whether the content is equal
    using parameter = std::tuple<uint32_t, storage::mutable_storage>;

    auto compare = [](const parameter& a, const parameter& b) -> bool
    {
        if (std::get<0>(a) != std::get<0>(b))
            return false;

        if (storage::is_same(std::get<1>(a), std::get<1>(b)))
            return true;

        return false;
    };

    EXPECT_EQ(layer.set_mutable_symbol.calls(), 5U);

    // The actual calls that were made
    auto zero = std::make_tuple(0, storage::mutable_storage(&data[0], 5U));
    auto one = std::make_tuple(1, storage::mutable_storage(&data[5], 5U));
    auto two = std::make_tuple(2, storage::mutable_storage(&data[10], 5U));
    auto three = std::make_tuple(3, storage::mutable_storage(&data[15], 5U));
    auto four = std::make_tuple(4, storage::mutable_storage(&data[20], 5U));

    EXPECT_TRUE(compare(layer.set_mutable_symbol.call_arguments(0), zero));
    EXPECT_TRUE(compare(layer.set_mutable_symbol.call_arguments(1), one));
    EXPECT_TRUE(compare(layer.set_mutable_symbol.call_arguments(2), two));
    EXPECT_TRUE(compare(layer.set_mutable_symbol.call_arguments(3), three));
    EXPECT_FALSE(compare(layer.set_mutable_symbol.call_arguments(4), four));
}

Example unit test

The Kodo library is build using the parameterized-inheritance/ mixin-layers C++ design technique. When unit testing a layer we try to isolate it as much as possible. To do this we typically introduce dummy layers with the sole purpose of satisfying the layer’s dependencies. To see this in action let’s look at one of the existing unit tests.

The storage_bytes_used layer is used when we want add functionality allowing us to keep track of how many useful bytes an encoder or decoder contains.

Note

In general the amount of data which can be encoded or decoded will be determined by the number of symbols we are coding and the size of every symbol in bytes (we call this the block size). However, in practical applications we sometimes do not have enough data to fill an entire block. In those cases we can add the storage_bytes_used layer to embed in every encoder and decoder the ability to store the number of actual data bytes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
// Copyright Steinwurf ApS 2011.
// Distributed under the "STEINWURF RESEARCH LICENSE 1.0".
// See accompanying file LICENSE.rst or
// http://www.steinwurf.com/licensing

#pragma once

#include <cassert>
#include <cstdint>

namespace kodo_core
{
/// @ingroup storage_info_layers
///
/// @brief Provides access to the number of useful bytes used out
///        of the total size of the encoders or decoders storage.
template<class SuperCoder>
class storage_bytes_used : public SuperCoder
{
public:

    /// Constructor
    storage_bytes_used() :
        m_bytes_used(0)
    {}

    /// @copydoc layer::initialize(Factory&)
    template<class Factory>
    void initialize(Factory& the_factory)
    {
        SuperCoder::initialize(the_factory);
        m_bytes_used = 0;
    }

    /// @copydoc layer::set_bytes_used(uint32_t)
    void set_bytes_used(uint32_t bytes_used)
    {
        assert(bytes_used > 0);
        assert(bytes_used <= SuperCoder::block_size());

        m_bytes_used = bytes_used;
    }

    /// @copydoc layer::bytes_used() const
    uint32_t bytes_used() const
    {
        return m_bytes_used;
    }

protected:

    /// The number of bytes used
    uint32_t m_bytes_used;
};
}

As seen, the layer depends on two functions being provided by the SuperCoder:

  1. SuperCoder::initialize(the_factory)
  2. SuperCoder::block_size()

Using our Doxygen documentation, it is possible to look up the purpose of the two undefined functions.

In this case we want to check that the state is correctly updated when calling set_bytes_used and that the state is correctly reset when calling initialize. The following unit test code was implemented in test/src/test_storage_bytes_used.cpp to test this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
// Copyright Steinwurf ApS 2011.
// Distributed under the "STEINWURF RESEARCH LICENSE 1.0".
// See accompanying file LICENSE.rst or
// http://www.steinwurf.com/licensing

/// @file test_storage_bytes_used.cpp Unit tests for the
///       storage_bytes_used class

#include <kodo_core/storage_bytes_used.hpp>

#include <cstdint>

#include <gtest/gtest.h>
#include <stub/function.hpp>

namespace kodo_core
{
// Put dummy layers and tests classes in an anonymous namespace
// to avoid violations of ODF (one-definition-rule) in other
// translation units
namespace
{
/// Helper layer
class dummy_layer
{
public:

    template<class Factory>
    void initialize(Factory& the_factory)
    {
        (void) the_factory;
        m_initialize();
    }

    uint32_t block_size() const
    {
        return m_block_size();
    }

    stub::function<void()> m_initialize;
    stub::function<uint32_t()> m_block_size;
};

// Helper stack
class dummy_stack : public storage_bytes_used<dummy_layer>
{ };

// Helper factory
class dummy_factory
{ };
}
}

TEST(test_storage_bytes_used, api)
{
    kodo_core::dummy_factory factory;
    kodo_core::dummy_stack stack;

    stack.m_block_size.set_return(10U);

    stack.initialize(factory);
    stack.set_bytes_used(9U);

    EXPECT_EQ(stack.m_initialize.calls(), 1U);
    EXPECT_TRUE(stack.m_block_size.expect_calls().with().to_bool());
    EXPECT_EQ(stack.bytes_used(), 9U);

    // Initialize again and check that the state is reset
    stack.initialize(factory);
    EXPECT_EQ(stack.m_initialize.calls(), 2U);
    EXPECT_TRUE(stack.m_block_size.expect_calls().with().to_bool());
    EXPECT_EQ(stack.bytes_used(), 0U);
}

In the above test code we use one test helper which allows use to easily add testing stubs to the unit test.

http://en.wikipedia.org/wiki/Test_stub

The library is called stub and is freely available under the BSD license:

https://github.com/steinwurf/stub

Naming the test case

When we define a test using gtest we use the TEST(test_case_name, test_name) macro to define and name a test function. In Kodo we use the following naming guideline:

  1. The test_case_name should be name according to its placement in the test/src directory. If the file is place in the root of the test/src folder e.g. test/src/test_my_fancy_code.cpp we name the test_case_name as test_my_fancy_code. Similarly if the file is placed in a subdirectory e.g. object/test_new_code.cpp we will specify the test_case_name as object_test_new_code. This should make it easy to find the source code of a failing unit test. Remember to place the test file as described in Namespaces and directories
  2. The test_name is up to the developer but should be as descriptive of the purpose of the unit test as possible.