Test Suites, and Suites vs Tags

The previous chapter grouped tests with tags, chosen at run time. Test suites are the older, more explicit way, still widely used.

A test suite is a collection of related tests grouped together so they run as one unit, with one report. You can create as many suites as you need: say 100 bank tests in one, 200 card tests in another.

Real uses

1. Pull request validation. Instead of running thousands of tests on every PR, run only the P0 (critical) suite: the tests most important to the business.
2. Hotfixes and targeted changes. For a hotfix in the bank flow, run only the bank suite, which saves time.
3. Organization. Group by feature: instrument-wise (bank, card, balance suites) or country-wise (India, US suites).

How a suite works

Suites come from the junit-platform-suite module, which adds a test engine of its own, "JUnit Platform Suite", next to Jupiter and Vintage. A class annotated @Suite is discovered by that engine, which then uses the Launcher to discover and run the tests the suite describes.

A suite works in two logical steps, in order:

1. Discovery: find all candidate tests, by package, class or method (selectors).
2. Filtering (optional): from the discovered tests, keep or drop some by tag, package, class name pattern or engine (filters). The remaining set runs.

Dependency

spring-boot-starter-test (or junit-jupiter) brings the Jupiter API and engine and the platform launcher, but not the suite annotations or the suite engine. Add:

<dependency>
    <groupId>org.junit.platform</groupId>
    <artifactId>junit-platform-suite</artifactId>
    <scope>test</scope>
</dependency>

With the Spring Boot parent (or JUnit's BOM) the version is managed, so leave it out; a hard-coded version can mismatch the rest of JUnit.

Discovery: selectors

@Suite
public class MySuite {
}

@Suite marks the class as a suite: JUnit looks for no @Test methods in it. With no selector, though, there is nothing to find, and running it fails with "Suite [MySuite] did not discover any tests".

@SelectPackages

All test classes and methods in these packages and their sub-packages (recursively):

@Suite
@SelectPackages({
        "com.example.taggingandfilter",
        "com.example.repeatedtest",
        "com.example.parameterisedtest"
})
public class MySuite {
}

@SelectClasses

All test methods of specific classes:

@Suite
@SelectClasses({
        PaymentTaggingTest.class,
        RepeatedTestDemo.class
})
public class MySuiteClassSelectorExample {
}

@SelectMethod and @SelectMethods

Specific test methods:

@Suite
@SelectMethod(type = PaymentTaggingTest.class, name = "validateCardScenario1")
@SelectMethods({
        @SelectMethod(type = AssumptionTest.class, name = "testForProduction"),
        @SelectMethod(type = RepeatedTestDemo.class, name = "testMethod")
})
public class MySuiteMethodSelectorExample {
}

Methods with parameters, such as a parameterized test:

@ParameterizedTest
@ValueSource(strings = {"civic", "level", "madam"})
public void testPalindrome(String word) { ... }

@SelectMethod(type = MyStringUtilTest.class, name = "testPalindrome") looks for testPalindrome() without parameters, finds nothing, and the suite fails with "did not discover any tests". Give the parameter types:

@SelectMethod(type = MyStringUtilTest.class, name = "testPalindrome",
              parameterTypeNames = "java.lang.String")

A suite may combine several selectors. If two selectors find the same test (a package and a class inside it), it runs only once.

Filtering

Filter	Effect
@IncludePackages / @ExcludePackages	keep only / drop these packages
@IncludeTags / @ExcludeTags	keep only / drop tests with one of these tags
@IncludeClassNamePatterns / @ExcludeClassNamePatterns	keep only / drop classes whose name matches a regex
@IncludeEngines / @ExcludeEngines	keep only / drop tests of these engines

Packages

@Suite
@SelectPackages("com.example")
@IncludePackages({"com.example.taggingandfilter", "com.example.repeatedtest"})
public class MySuiteFilterExample {
}

Discovery finds every sub-package of com.example (assumptions, parameterisedtest, repeatedtest, taggingandfilter, ...); the filter keeps only the two listed. @ExcludePackages does the opposite: drop the listed packages, keep the rest.

With both, the result is (discovered ∩ included) − excluded:

@Suite
@SelectPackages("com.example")
@IncludePackages({"com.example.taggingandfilter", "com.example.repeatedtest"})
@ExcludePackages({"com.example.parameterisedtest", "com.example.testsuite", "com.example.repeatedtest"})
public class MySuiteFilterExample {
}
// only com.example.taggingandfilter runs

Tags

@Suite
@SelectPackages("com.example")
@IncludeTags({"card", "bank"})        // tests tagged card OR bank
public class MySuiteFilterExample {
}

@ExcludeTags({"bank"}) keeps everything except tests tagged bank.

Class name patterns

@IncludeClassNamePatterns(".*Test")      // classes whose name ends in Test
@ExcludeClassNamePatterns(".*Example")   // drop classes ending in Example

The patterns are regular expressions matched against the fully qualified class name (com.example.card.PaymentTest), so a pattern usually starts with .*. ^Payment.* matches nothing, because the name starts with the package.

The default pattern. Without @IncludeClassNamePatterns, a suite includes only classes whose names start or end with Test or end with Tests (^(Test.*|.+[.$]Test.*|.*Tests?)$). A test class named PaymentChecks in a selected package is silently skipped. Use @IncludeClassNamePatterns(".*") to include every class.

Regex reminders:

Pattern	Meaning	Example
.	any single character	s.j matches s_j, saj
*	the previous item zero or more times	sj* matches s, sj, sjj
.*	anything	.*payment.*: "payment" anywhere
^ / $	start / end of the text	(payment|order)$: ends with payment or order
|	or	.*(payment|order).*
(?i)	case-insensitive from here on	(?i).*payment.* also matches CardPaymentTest

Regexes are case-sensitive by default: .*payment.* does not match CardPaymentTest.

Engines

The platform knows which engine owns which test (from the discovery phase), so a suite can filter by engine:

@IncludeEngines({"junit-jupiter"})    // JUnit 5 tests only
@ExcludeEngines({"junit-vintage"})    // no JUnit 3/4 tests

Running suites

In IntelliJ, right-click the suite class and run it. In CI, with Maven:

mvn test -Dtest=MySuite1,MySuite2

A "bank CI job" can then run mvn test -Dtest=BankSuite.

Avoiding duplicate runs

Suites are tests too. If a suite selects a package that contains other suite classes, those suites are discovered and run as well, and their tests run again. In the test above, with all class names included, a package suite ran a nested suite's test twice. Keep suite classes in their own package, outside the packages they select.

Tags vs suites

Both are industry standard; neither replaces the other.

	Tags with filtering	Test suites
Selection	dynamic: chosen at run time (mvn test -Dgroups="bank | card")	static: hard-coded in a Java class
CI	preferred for flexibility: change the job, not the code	the job just names the suite
Readability	the groups are spread over annotations	a suite class shows clearly what belongs to the group
Duplicate runs	little risk	possible (nested suites, overlapping suites)
Maintenance	low: only annotations	extra classes to keep up to date; suites per feature pile up over the years

Next: Repeated and parameterized tests run one test method several times, but the number of tests is still fixed when the code is written. Chapter 13, the last JUnit chapter, creates tests at run time with @TestFactory.

Check yourself

1. What is a test suite, and give two real uses.
2. Which component runs a @Suite class?
3. What are the two steps of a suite, and which is optional?
4. What happens when a suite has no selector?
5. How do you select a parameterized test method in a suite?
6. With @IncludePackages and @ExcludePackages together, which tests run?
7. A test class PaymentChecks sits in a selected package but never runs. Why, and how do you fix it?
8. Why does @IncludeClassNamePatterns("^Payment.*") match nothing?
9. How can a suite run the same test twice, and how do you avoid it?
10. When would you choose tags over a suite, and vice versa?

Answers

1. A group of related tests run as one unit. PR validation with a P0 suite; running only the bank suite for a bank hotfix; grouping by feature or country.
2. The JUnit Platform Suite engine (from junit-platform-suite), which uses the Launcher to discover and run the selected tests.
3. Discovery (selectors) and filtering (filters); filtering is optional.
4. It discovers nothing and fails: "did not discover any tests".
5. @SelectMethod(type = ..., name = "testPalindrome", parameterTypeNames = "java.lang.String").
6. (Discovered ∩ included) − excluded.
7. By default suites include only classes named like Test*, *Test or *Tests. Add @IncludeClassNamePatterns(".*") (or rename the class).
8. Patterns are matched against the fully qualified name, which starts with the package.
9. By selecting a package that contains other suites (or overlapping suites). Keep suite classes in a separate package.
10. Tags: flexible, chosen at run time in CI, low maintenance. Suites: when an explicit, readable list of what belongs to a group matters.