Dynamic Tests with @TestFactory

Repeated and parameterized tests (chapters 7 to 9) run one test method several times. A test factory goes one step further and creates the tests themselves while the suite runs.

A @TestFactory method is not a test: it is a factory that creates tests, at runtime.

Static vs dynamic tests

A normal @Test is static: the test is fully known at compile time.

@Test
void addTest() {
    assertEquals(7, 4 + 3);
}

A dynamic test is created while the tests run:

@TestFactory
DynamicTest addDynamicTest() {
    return DynamicTest.dynamicTest(
            "addTest",                       // display name
            () -> assertEquals(7, 4 + 3)     // executable: the actual test
    );
}

DynamicTest.dynamicTest takes a display name and an Executable (a lambda with no parameters): the code of the test.

Inside the engine

From the architecture chapter: in the discover phase the engine builds TestDescriptors, and in the execute phase it runs them.

• A normal @Test becomes a TestMethodTestDescriptor. Its display name is the method name, and its "executable" is the method itself: the framework wraps the method in a lambda that calls it through reflection.
• A dynamic test becomes a DynamicTestTestDescriptor. Its display name is the one you gave, and its executable is your lambda, which the engine simply runs.

So to the engine both are "a display name plus something to execute".

One difference: the per-test lifecycle. @BeforeEach and @AfterEach run around the @TestFactory method as a whole, once, not around each dynamic test it produces. If every generated test needs fresh setup, dynamic tests may not fit; do the setup inside each test's lambda. (@BeforeAll and @AfterAll behave as usual.)

@BeforeEach
    Positive Check for number: 1
    Positive Check for number: 2
    Positive Check for number: 3
    Positive Check for number: -5
@AfterEach

When dynamic tests help

A payment gateway receives a daily settlement.csv file from a bank.

The file is stable: a fixed schema, and the same validation for every row. Two options:

• a parameterized test with @CsvFileSource (chapters 8 and 9): one test method executed N times, once per row, with the same assertions;
• dynamic tests: N independent tests, one per row.

Both work.

The file is not stable:

• columns may appear or disappear,
• the column order may change,
• different rows need different validations (row 1 needs check 1; row 2 checks 1 and 2; row 3 check 3...).

A parameterized test runs the same method for every row, so it cannot vary the checks. With dynamic tests, the code that builds the tests can decide, row by row, which tests to create. The same applies to rows read from a database. For most cases (well over 90%) a parameterized test is enough; dynamic tests are for when the tests themselves depend on the data.

The hierarchy: a composite

This is the composite design pattern: a DynamicTest is a leaf (it runs); a DynamicContainer is a named group, like a folder, holding dynamic tests and/or other containers, nested to any depth.

settlement.csv (container)
├── file exists and is readable     (test)
├── header and footer present       (test)
└── transactions (container)
    ├── at least one transaction    (test)
    └── ...

Writing dynamic tests

1. A single test

@TestFactory
DynamicTest addDynamicTest() {
    return DynamicTest.dynamicTest("addTest", () -> assertEquals(7, 4 + 3));
}

The return type may also be DynamicNode, the parent type.

2. Many tests

Return a Stream, a Collection, an Iterable, an Iterator or an array of dynamic nodes. A stream is the most common:

@TestFactory
Stream<DynamicTest> multipleDynamicTest() {
    return Stream.of(1, 2, 3, -5)
            .map(n -> DynamicTest.dynamicTest(
                    "Positive Check for number: " + n,
                    () -> assertTrue(n > 0)));
}

Four tests are created; the one for -5 fails.

@TestFactory
Collection<DynamicTest> multipleDynamicTestsCollectionReturn() {
    List<DynamicTest> tests = new ArrayList<>();
    for (int n : List.of(1, 2, 3, -5)) {
        tests.add(DynamicTest.dynamicTest("Positive Check for number: " + n, () -> assertTrue(n > 0)));
    }
    return tests;
}

@TestFactory
DynamicTest[] multipleDynamicTestsArrayReturn() {
    return new DynamicTest[] {
            DynamicTest.dynamicTest("Positive Check for number: 1", () -> assertTrue(1 > 0)),
            DynamicTest.dynamicTest("Positive Check for number: 2", () -> assertTrue(2 > 0)),
            DynamicTest.dynamicTest("Positive Check for number: 3", () -> assertTrue(3 > 0))
    };
}

3. Containers: different tests for different rows

Txn_Id   Amount   Currency
Txn1     100      INR
Txn2     ABC      USD

Every row must have an integer amount; INR rows must also have an amount greater than zero.

@TestFactory
Stream<DynamicContainer> fileRowDynamicTests() {
    List<String[]> rows = List.of(
            new String[]{"Txn1", "100", "INR"},
            new String[]{"Txn2", "ABC", "USD"}
    );

    return rows.stream().map(row -> {
        String txnId = row[0];
        String amount = row[1];
        String currency = row[2];

        List<DynamicTest> tests = new ArrayList<>();

        // every row
        tests.add(DynamicTest.dynamicTest("amount is integer",
                () -> assertTrue(amount.matches("\\d+"))));

        // INR rows only
        if ("INR".equals(currency)) {
            tests.add(DynamicTest.dynamicTest("amount is greater than zero",
                    () -> assertTrue(Integer.parseInt(amount) > 0)));
        }

        // group this row's tests
        return DynamicContainer.dynamicContainer("Row " + txnId, tests);
    });
}

fileRowDynamicTests()
├── Row Txn1
│   ├── amount is integer             ✔
│   └── amount is greater than zero   ✔
└── Row Txn2
    └── amount is integer             ✘  ("ABC")

Row Txn1 got two tests, row Txn2 one. This is what a parameterized test cannot do.

4. A container with tests and containers

@TestFactory
DynamicContainer containerWithTestAndContainer() {
    DynamicTest simpleTest = DynamicTest.dynamicTest("addTest", () -> assertEquals(2, 1 + 1));

    DynamicContainer nestedContainer = DynamicContainer.dynamicContainer(
            "Nested container",
            Stream.of(DynamicTest.dynamicTest("MultiplyTest", () -> assertEquals(4, 2 * 2))));

    return DynamicContainer.dynamicContainer(
            "Parent container",
            Stream.of(simpleTest, nestedContainer));
}

containerWithTestAndContainer()
└── Parent container
    ├── addTest
    └── Nested container
        └── MultiplyTest

Next: That completes JUnit 5. The rest of the series is about Mockito: replacing a unit's dependencies with fake objects so the unit can be tested in isolation, as chapter 2 required. Chapter 14 starts with its three core ideas: mocks, stubs and spies.

Check yourself

1. Is a @TestFactory method a test? What does it do?
2. To the Jupiter engine, what do a normal test and a dynamic test each consist of?
3. How do @BeforeEach and @AfterEach behave with a factory that creates five dynamic tests?
4. When is a parameterized test enough, and when do you need dynamic tests?
5. Explain DynamicNode, DynamicTest and DynamicContainer as a composite.
6. Which return types can a @TestFactory method have?
7. How would you give INR rows an extra check that other rows do not get?

Answers

1. No; it creates tests at runtime and returns them.
2. A display name and something to execute: for @Test, the method (called via reflection); for a dynamic test, your lambda.
3. They run once, around the factory method, not around each dynamic test.
4. Parameterized: the same validation for every input. Dynamic: when the tests to run depend on the data, e.g. different rows need different checks.
5. DynamicNode is the abstract parent; DynamicTest is a leaf that runs; DynamicContainer groups nodes (tests or containers) like a folder.
6. A single DynamicNode (DynamicTest or DynamicContainer), or a Stream, Collection, Iterable, Iterator or array of them.
7. In the code that builds each row's tests, add the extra DynamicTest only if ("INR".equals(currency)), and group the row's tests in a DynamicContainer.