C testing for Postgres

During the unconference at PGCon in Ottawa, I asked about writing C-based
tests for Postgres. There was interest in trying a tool and also some
hesitation to depend on a third-party library. So, I wrote a tool that I'd
like to contribute to Postgres. I’ve been calling it cexpect [1]
<https://github.com/berlin-ab/cexpect>.

cexpect is a general-use library for creating test suites in C. It includes:

- a core library for creating and running suites of tests.

- a standard set of test expectations for C

- a default formatter (dots)

- an extensible matcher framework

- an extensible formatter framework

Why add a testing framework for C to Postgres?

An xUnit-style test framework [2] is a tool for writing tests that is not
currently an option for people hacking on Postgres.

-

C-based tests could help increase code coverage in parts of the codebase
that are difficult to reach with a regress-style test (for example: gin
posting list compression [3]).
-

Writing tests for internal components help developers become more
familiar with a codebase.
-

Writing C-based tests go beyond providing regression value by providing
feedback on design decisions like modularity and dependencies.
-

Test suites already existing in `src/test/modules` could benefit from
having a consistent way to declare expectations, run suites, and display
results.
-

Forks and extensions that write tests could benefit from a testing
framework provided by the upstream project. An extensible framework will
allow forks and extensions to create their own matchers and formatters
without changing the core framework.

The extensible matcher framework has benefits for decoupled, isolated unit
tests, and also high-level system tests. The complexity of a common
assertions can be packaged into a matcher - abstracting the complexity and
making the assertion easier to reuse.

For example, there could be expectation matchers specifically crafted for
the domain of Postgres:

`expect(xlog, to_have_xlog_record(XLOG_XACT_PREPARE))`

`expect(“postgresadmin”, to_be_an_admin_user())`

The matchers that come with cexpect out of the box are for C datatypes:

`expect(1, is_int_equal_to(1))`

`expect(1 == 2, is_false())`

`expect(“some random string”, is_string_containing(“and”))’

… and more, with the goal of having a matcher for all standard data types.

The extensible formatter framework could be used to create a test-anything
protocol (TAP) output, familiar to Postgres hackers. It could also be used
to insert test results into a database table for later analysis, or create
a consistent way of reporting results from a user-defined function - the
pattern often used for creating tests under `src/test/modules`.

How does it work?

Create an executable that links to the core shared library, include the
core library headers, create a suite, add some tests, and run it.

test.c:

```

#include "cexpect.h"

#include "cexpect_cmatchers.h"

void some_passing_test(Test *test) {

expect(test, 1, is_int_equal_to(1));

}

int main(int argc, char *args[]) {

Suite *suite = create_suite("Example test");

add_test(suite, some_passing_test);

start_cexpect(suite);

}

```

Running test.c:

```bash

export DYLD_LIBRARY_PATH=$path_to_cexpect_library

export compile_flags=”-Wno-int-conversion -Wno-pointer-to-int-cast test.c
-L $path_to_cexpect_library -I $path_to_cexpect_headers”

gcc $compile_flags -l cexpect -o test.o

./test.o

Running suite: Example test

.

Summary:

Ran 1 test(s).

1 passed, 0 failed, 0 pending

```

Rather than post a patch, I'd rather start a conversation first. I'm
guessing there are some improvements that we'd want to make (for example:
the Makefile) before commiting a patch. Let's iterate on improvements
before creating a formal patch.

Thoughts?

Thanks,

Adam Berlin

Software Engineer at Pivotal Greenplum

[1] https://github.com/berlin-ab/cexpect

[2] https://en.wikipedia.org/wiki/XUnit

[3]
https://coverage.postgresql.org/src/backend/access/gin/ginpostinglist.c.gcov.html