Testing framework

Introduction

Goal of the go-testing framework is to provide unified building blocks for writing short and effective unit component, and integration tests as well as benchmarks in go using simple common patterns.

To accomplish this, the go-testing framework provides a couple of extensions for go's testing package that support setup of strongly isolated and parallel running unit tests using gomock and/or gock that work under various failure scenarios even in the presence of spawned go-routines.

The core idea of the mock/gock packages is to provide a short pragmatic domain language for defining mock requests with responses that enforce validation, while the test package provides the building blocks for efficient test setup and test isolation.

While still on version 0.1.x the code and the API has proven to be pretty stable over the last years. The only reason, why it has not been released as 1.0 is that one of the core ideas of this framework, the extended mock generator, has not progressed as intended for an initial release.

Example Usage

First you have to define a test/benchmark parameter set. While this can be done in many ways, the following setup structure is considered to be the go-testing framework idiomatic way due to its readability and wide coverage of different use cases:

type UnitParams struct {
    setup        mock.SetupFunc
    input*...    *model.*
    expect       test.Expect
    expect*...   *model.*
    expectError  error
}

var unitTestCases = map[string]UnitParams {
    "success" {
        setup: mock.Chain(
            CallMockA(input..., output...),
            ...
            test.Panic("failure message"),
       ),
        ...
        expect: test.ExpectSuccess
    }
}

Now you can set up a strongly isolated and parallel running test. While there are many ways to define such tests (see package test), the following pattern is considered to be the most go-testing framework idiomatic way due to its readability and wide coverage of different use cases:

func TestUnit(t *testing.T) {
    // Setup the test using a map fo parameterization.
    test.Map(t, unitTestCases).
        // Exclude of test cases temporary or permanent.
        Filter(test.Not(test.Pattern[T]("^test-case-prefix"))).
        // Include of test cases temporary or permanent.
        Filter(test.Pattern[T]("^test-case-name$")).
        // Run the test in parallel.
        Run(func(t test.Test, param UnitParams){

            // Given
            mocks := mock.NewMock(t).
                SetArg("common-arg", local.input*)...
                Expect(param.setup)

            unit := NewUnitService(
                mock.Get(mocks, NewServiceMock),
                ...
            )

            // When
            result, err := unit.call(param.input*...)

            mocks.Wait()

            // Then
            assert.Equal(t, param.expectError, err)
            assert.Equal(t, param.expect*, result)
        })
}

As an addon, you can also use the same pattern to define benchmarks for a system under test based on the before defined test parameter set. The following setup structure is considered to be the most go-testing framework idiomatic way (see also Test benchmark setup):

func BenchmarkUnit(b *testing.B) {
    test.Map(test.Benchmark(b), unitTestCases).
        // Exclude of test cases temporary or permanent.
        Filter(test.Not(test.Pattern[T]("^test-case-prefix"))).
        // Include of test cases temporary or permanent.
        Filter(test.Pattern[T]("^test-case-name$")).
        // Execute benchmark setup and loop phases.
        Benchmark(func(b *testing.B, param UnitParams) func(b *testing.B) {
            // Setup
            unit := NewUnitService(param.input*...)

            // Define processed bytes.
            b.SetBytes(len(param.input*))

            // Loop
            return func(b *testing.B) {
                result, err := unit.call(param.input*...)

                // Prevent optimization.
                runtime.KeepAlive(result)
                runtime.KeepAlive(err)
            }
        })
}

Note: in a benchmark you need to ensure that you reserve sufficient memory for the unit-under-test in the setup phase to avoid additional memory allocs in the loop. While you also should prevent return values from being optimized away in the loop using runtime.KeepAlive, you should not do this for multi-byte results, since these also creates additional memory allocations due the the copy nature of the runtime.KeepAlive.

For more test patterns and variations have a closer look at details in the test package or read the package docs.

Why parameterized test?

Parameterized (table-driven) test are an effective way to set up a systematic set of test cases covering a system under test in a black or white box mode. With the right tools and concepts — such as supported by this test framework —, parameterized test allow to cover all success and failure paths of a system under test.

Why parallel tests?

Running tests in parallel makes the feedback loop on failures faster and helps to detect failures from concurrent access. By using go test -race we can easily uncover race conditions, that else only appear randomly in production, and foster a design with clear responsibilities. This side-effects compensate for the small additional effort needed to write parallel tests.

Why isolation of tests?

Test isolation is a precondition to have stable running test — especially run in parallel. Isolation must happen from input perspective, i.e. the outcome of a test must not be affected by any previous running test, but also from output perspective, i.e. it must not affect any later running test. This is often complicated since many tools, patterns, and practices break the test isolation (see requirements for parallel isolated tests.

Why strong validation?

Test are only meaningful, if they ensure/validate pre-conditions as well as validate/ensure post-conditions sufficiently strict. Without validation test cannot ensure that the system under test behaves as expected — even with 100% code and branch coverage. As a consequence, a system may fail in unexpected ways in production.

Thus, it is advised to validate input parameters for mocked requests and to carefully define the order of mock requests and responses. The mock framework makes this approach as simple as possible, but it is still the responsibility of the test developer to set up the validation correctly.

Framework structure

The go-testing framework consists of the following sub-packages:

• test provides a small framework to isolate the test execution and safely check whether a test fails or succeeds as expected in combination with the mock package — even if a system under test spans detached go-routines.

• mock provides the means to set up a simple chain as well as a complex network of expected mock calls with minimal effort. This makes it easy to extend the usual narrow range of mocking to larger components using a unified test pattern.

• gock provides a drop-in extension for the Gock package consisting of a controller and a mock storage that allows running tests isolated. This allows parallelizing simple test as well as parameterized tests.

• perm provides a small framework to simplify permutation tests, i.e. a consistent test set where conditions can be checked in all known orders with different outcome. This was very handy in combination with test for validating the mock framework, but may be useful in other cases too.

Please see the documentation of the sub-packages for more details.

Requirements for parallel isolated tests

Running tests in parallel makes test not only faster, but also helps to detect race conditions that else randomly appear in production, by running tests using go test -race.

Note: there are some general requirements for running test in parallel:

1. Tests must not modify environment variables dynamically — utilize test friendly configuration concepts instead.
2. Tests must not require reserved service ports and open listeners — setup services to acquire dynamic ports instead.
3. Tests must not share any files, folders, and pipelines, e.g. stdin, stdout, or stderr — implement logic by using wrappers that can be easily redirected and mocked.
4. Tests must not share database schemas or tables, that are updated during execution of parallel tests — implement test to set up test specific database schemas.
5. Tests must not share process resources, that are update during execution of parallel tests. Many frameworks make use of common global resources that make them unsuitable for parallel tests — use frameworks that do not suffer by these flaws.

Examples for such shared resources in common frameworks are:

• Using of monkey patching to modify commonly used global functions, e.g. time.Now() — implement access to these global functions using lambdas and interfaces to allow for mocking.
• Using of gock to mock HTTP responses on transport level — make use of the gock-controller provided by this framework.
• Using the Gin HTTP web framework which uses a common json-parser setup instead of a service specific configuration. While this is not a huge deal, the repeated global setup creates race alerts. Instead, use chi that supports a service specific configuration.

With a careful system design, the general pattern provided above can be used to create parallel test for a wide range of situations.

Building

This project is using a custom build system called go-make, that provides default targets for most common tasks. Makefile rules are generated based on the project structure and files for common tasks, to initialize, build, test, and run the components in this repository.

To get started, run one of the following commands.

make help
make show-targets

Read the go-make manual for more information about targets and configuration options.

Not: go-make installs pre-commit and commit-msg hooks calling make commit to enforce successful testing and linting and make git-verify message to validate whether the commit message is following the conventional commit best practice.

Terms of Usage

This software is open source under the MIT license. You can use it without restrictions and liabilities. Please give it a star, so that I know. If the project has more than 25 Stars, I will introduce semantic versions v1.

Contributing

If you like to contribute, please create an issue and/or pull request with a proper description of your proposal or contribution. I will review it and provide feedback on it as fast as possible.

Disclaimer

This software is developed with the help of AI following the highest human standards. All actions executed by AI are carefully reviewed, counter-checked, and corrected with the highest human standards and quality goals in mind. No AI generate code is allowed to be merged or released without a careful human reviews to prevent systematic degeneration of coding standards and code quality.