This is a post that relates to C programming rather than hardware development, for a change.

Two things I'm a big fan of when developing software is designing with assertions (e.g. assert(index >= 0); in C) and Unit Testing (Unit Testing Wikipedia article).

Assertions is available in VHDL as well. I motivate their use in the post Assertions - extending their use and much of that will be valid also in a software context.

This post will describe the way I use to integrate the C assert(...) statements into the test routines utilizing a Unit Testing Framework. I use the CUnit framework but the method will work with any framework.

The example case

Assume our program is defined in prog.c and our test routines in tests.c. tests.c will #include prog.c to get the routines in tests.c access to the functions in prog.c.

Assume our program has a function getChar which we will test with CUnit. getChar has a simple assert statement to check for a negative index:

// in prog.c:
// function to test:
char getChar(char str[], int i) { // return str[i]
  assert(i >= 0);
  return str[i];
}

(To get access to assert(), assert.h in C standard library must be included.)

A CUnit test function could perform a simple test of getChar like this:

// in tests.c:
char c1 = getChar("test", 0);
CU_ASSERT_EQUAL(c1, 't');

If getChar is called with a negative i argument, program execution will halt.

Requirements

My requirements for a solution to integrate assert() with unit testing were:

  • It should be possible to compile and run the program stand-alone (without and independent of the testing routines) without any changes to prog.c. Any assert(...) statements should then behave as usual
  • When testing the code with CUnit:
    • the assert() should be transformed to a CUnit assert so it is reported as a passed/failed test within the framework
    • a failing assert() shouldn't halt execution of all the other tests 1
    • however, a failing assert() should terminate the function where it appears 2
    • it should be possible to write a test that tests the assertion itself, e.i., that it is activated if it should be, e.i., it should be possible run getChar("test", -1), and make CUnit report it as a passed test if the assert() statement would have been activated

    Solution

    Put the following macros in top of prog.c. When testing, they will replace the assert() from assert.h with assert statements from CUnit. This solution will fulfill all the requirements.

    For testing of an assertion itself, invert_cassertions can be set to true just before calling a function with arguments that should make the assert fail. With invert_cassertions=true, the assert will be replaced by a CUnit assertion that will pass if the original condition is not met. See the example code further below.

    // place this in prog.c (after #include <assert.h>):

    #define XOR(x, y) ( !(x) != !(y) )

    // if ISTESTING, overwrite macro definion of assert() in assert.h:
    #ifdef ISTESTING
    #undef assert
    #define assert( x ) \
      do { \
        CU_ASSERT( XOR(x, invert_cassertions) )\
        if (!(x)) return; } \
      while (0)

    #endif

    Place the following near the beginning of tests.c:

    // place before '#include "prog.c"':

    #define ISTESTING
    //declare here so it's global, init to false:
    _Bool invert_cassertions = 0;

    Our test code in tests.c will now be e.g:

    // first a simple test:
    char c1 = getChar("test", 0);
    CU_ASSERT_EQUAL(c1, 't');
     
    // make sure assertion in getChar works:
    invert_cassertions=1;
    char c2 = getChar("test", -1); // should fire assertion
    invert_cassertions=0;

    The second test sets invert_cassertions to true when running the getChar function. Here, we are testing that the assert in getChar does really fire when we provide a negative index. When invert_cassertions is true, the assert condition is inverted, and hence, the new assertion will pass if and only if the original assertion would have fired.

    How it works

    XOR

    Having an original assert condition x, and a variable y (invert_cassertions) that should invert the logical value of x if it is set to true, we can see that we get the desired behavior if we replace the original condition with x \oplus y (where \oplus is the XOR operator).

    The C language doesn't have a logical XOR operator so we build one with:

    #define XOR(x, y) ( !(x) != !(y) )

    which does define the XOR since some logical equivalences yields (not = \neg):

     x \oplus y  \quad \Leftrightarrow \quad x \neq y \quad \Leftrightarrow \quad \neg x \neq \neg y

    We need to normalize x and y to booleans. Usually this is done by !!(x) etc., but in our case the last equivalence above shows us that we can just as well use a single negation on both sides.

    Macro replacing assert(...)

    Enclosing the macro in #ifdef ISTESTING makes sure the replacement only takes place when we are testing. #undef assert is needed since the definitition of assert(...) in assert.h is actually not a function but a macro definition as well. We need to undefine that definition and the re-define it.

    Multi-statement macros

    The do { ...} while 0 construct, which will always execute the block eactly once, is there to make sure that the macro expands correctly in all possible cases 3.

    Resulting pre-parser output

    Our getChar function with the assert:

    char getChar(char str[], int i) {
      assert(i >= 0);
      return str[i];
    }

    will with the macro replacements expand to:

    char getChar(char str[], int i) {
      do
      {
        { CU_assertImplementation(( !(i >= 0) != !invert_cassertions ), 36,
            "XOR(i >= 0, invert_cassertions)", "prog.c", "", 0);
        }
        if ( !(i >= 0) ) return;
      } while (0);

      return str[i];
    }

    CU_assertImplementation is the result of CUnits macro expansion of CU_ASSERT. XOR will not be expanded in the string since macro expansion doesn't take place in strings.

    A reservation

    I use if ( !(i >= 0) ) return; to leave the function prematurely if the assertion fails. Having a return statement with no return value for a function with a return value does not comply with the C standard, but I found that gcc (at least my version, 4.6.3) doesn't complain. If your compiler is more picky about following the C standard, you might need to do without the functionality of leaving the function if the assertion condition fails.

    (Having a return statement like return; is allowed for functions with no (e.i. void) return type. Also, the standard does actually allow functions declared as having a return value to leave out the return statement, although the returned value is then undefined.)

    Complete code listings

    prog.c:

    #include <stdio.h>
    #include <assert.h>

    #define XOR(x, y) ( !(x) != !(y) )

    // if ISTESTING, overwrite macro definition of assert() in assert.h:
    #ifdef ISTESTING
      #undef assert
      #define assert( x ) \
        do { \
          CU_ASSERT( XOR(x, invert_cassertions) )\
          if (!(x)) return; } \
        while (0)

    #endif

    char getChar(char str[], int i);

    #ifndef ISTESTING
    int main() {
     
      printf("This will run fine:\n");
      getChar("test", 2); // runs fine
     
      printf("This will fire assertion:\n");
      getChar("test", -1); // fires assertion
     
    }
    #endif

    //return str[i]
    char getChar(char str[], int i) {
      assert(i >= 0);
      return str[i];
    }

    tests.c:

    #include <CUnit/CUnit.h>
    #include <CUnit/Basic.h>

    // don't include main() in prog.c:
    #define ISTESTING

    //declare here so it's global, init to false:
    _Bool invert_cassertions = 0;

    //include file with code to test:
    #include "prog.c"

    void doTests();

    int main() {

      CU_initialize_registry(); // initialize
     
      // create suite:
      CU_pSuite suitePtr=CU_add_suite("", NULL, NULL);
      // add doTest() to the suite:
      CU_add_test(suitePtr, "", doTests);
     
      CU_basic_set_mode(CU_BRM_VERBOSE);  // set run mode
      CU_basic_run_tests();               // run tests
      CU_cleanup_registry(); 
     
    }

    void doTests() {
     
      // first a simple test:
      char c1 = getChar("test", 0);
      CU_ASSERT_EQUAL(c1, 't');
     
      // make sure assertion in getChar works:
      invert_cassertions=1;
      char c2 = getChar("test", -1);  // should fire assertion
      invert_cassertions=0;
     
    }

    Example compilation commands and run output

    Compile and run prog.c:

    $ gcc prog.c -o prog
    $ ./prog
    This will run fine:
    This will fire assertion:
    prog: prog.c:36: getChar: Assertion 'i >= 0' failed.
    Aborted (core dumped)
    $

    Compile and run tests.c:

    $ gcc tests.c -o test -I ~/CUnit/include/ -L ~/CUnit/lib/ -lcunit
    $ ./test


         CUnit - A unit testing framework for C - Version 2.1-2
         http://cunit.sourceforge.net/


    Suite:
      Test:  ...passed

    Run Summary:    Type  Total    Ran Passed Failed Inactive
                  suites      1      1    n/a      0        0
                   tests      1      1      1      0        0
                 asserts      3      3      3      0      n/a

    Elapsed time =    0.000 seconds
    $

    (Replace ~/CUnit in the compilation command with the CUnit installation path.)

    Note that three CUnit tests have been performed - the first getChar function call will also result in an assertion that will now be reported as a successfull assert test.

    References

    The benefits of programming with assertions by Philip Guo

    Wrap multiple statement macros in a do-while loop

    Notes:

    1. the assert() from assert.h will halt program execution
    2. a failing assert() is likely to result in program termination later in that function - that's e.g. the case with getChar since str[-1] etc. will be evaluated causing the program to halt, and the other tests can't continue
    3. https://www.securecoding.cert.org/confluence/display/seccode/PRE10-C.+Wrap+multi-statement+macros+in+a+do-while+loop