Build Status

We know which tests you want to run.

We all know that as the project evolves we keep adding more and more tests to ship our products confidently. This however has an impact on the build time and thus we waste time waiting for the most important question to be answered - "Did I break anything with my changes?".

Let it be your local development or a CI server - what if you could know this as soon as possible?

We created Smart Testing to give you the fastest possible feedback loop when it comes to executing your tests.

1. What is this?

Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health.

Suppose that you need to implement a new feature in your current project. If you are following TDD / BDD approach, you’ll write some test(s) to validate that the new feature behaves as expected. Then you’ll implement the new feature and when you get that all tests passes. Last step is to push this code to your source control, which probably will trigger a new build in your CI server running all the tests.

And at this point two things can happen:

  1. The build takes several minutes and it does not fail.

  2. The build takes several minutes but it fails because your new tests are failing (yes I know it worked in your machine).

Usually you don’t have that much influence on when your new tests are going to be executed. They could be the first ones, in that case your build will fail fast, or they might be the last ones, so you’ll need to wait some time until you get the failure.

Smart testing project aims to solve this uncertainty by ordering tests so that the first tests that could fail (because for example they are new or because failed previously) are going to be executed first.

1.1. Integrations

Currently Smart Testing supports Maven through Maven Extension mechanism and works together with maven-surefire-plugin and maven-failsafe-plugin.

1.2. Minimum requirements

Java

Requires Java 8.

Currently we don’t support Java 9 for Affected strategy.
Maven

Tested with Maven 3.3.9 and 3.5.0 but any version above 3.1.X should work. Our recommendation is that you use at least 3.3.X version to avoid any extension API problems.

Surefire/Failsafe

Minimum version required is 2.19.1 or above. It is important to make sure that you have it defined in your pom.xml, as Maven comes with the default version which might not be compatible with our extension.

2. Installation

2.1. Don’t make me think

Ok, just execute following snippet and you are all set:

curl -sSL https://git.io/v5jy6 | bash

You can also add these flags:

  • -l|--latest latest published artifact in Maven Central (that’s the default)

  • -v|--version= - specific version to install

For example:

curl -sSL https://git.io/v5jy6 | bash -s -- --latest

This script will automatically do what we described below. If you are curious keep on reading, but you may skip it too.

This script requires xmllint, and xsltproc, so make sure you have it installed.

2.2. Maven Extension

Smart Testing is a Maven extension, so depending on the version of Maven you have to follow slightly different approach of installing it.

2.2.1. Maven above 3.1.X

Get Smart Testing Extension shaded jar from Maven Central and copy it to M2_HOME/lib/ext.

2.2.2. Maven above 3.3.X

You can still use the process described at [Maven >= 3.1.X] or use the new core extension configuration mechanism by creating folder called .mvn in the root of your project and create inside it an extensions.xml file which registers the smart testing extension:

${maven.projectBasedir}/.mvn/extensions.xml
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
  <extension>
    <groupId>org.arquillian.smart.testing</groupId>
    <artifactId>maven-lifecycle-extension</artifactId>
    <version>${version}</version>
  </extension>
</extensions>

3. Configuration

This section explains configuration parameters of Smart Testing. Notice that they are all summarized at Reference Card section.

Also Smart Testing can be configured with a configuration file instead of System properties. You can read about configuration file at Configuration File.

3.1. Modes

So far in What is this? section we explained that Smart testing changes the order of test execution plan to run first the important tests and then the rest. This is know as ordering mode, but that’s not the only one.

ordering

ordering mode as its name suggests orders the test execution plan so important tests are executed first and then the rest.

selecting

selecting mode just selects the important tests and execute them, skipping the rest of the tests.

The selecting is the default mode.

In order to define the mode use smart.testing.mode Java system property with either one.

To get fast feedback loop, you can use surefire’s skip after N failures/errors feature by setting system property surefire.skipAfterFailureCount to N or by following configuration:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${version.surefire.plugin}</version>
    <configuration>
      <skipAfterFailureCount>N</skipAfterFailureCount>
    </configuration>
</plugin>

However this functionality cannot be fully guaranteed (real first failure) in concurrent mode due to race conditions. Read skipAfterFailureCount for more information.

3.2. Strategies

Until now, you’ve read that smart testing is changing test execution plan running or only including important tests. But how do we know which tests are important and which ones not?

There are several strategies that you can choose from which determine what are the important tests. Currently we have following strategies in place: new, changed, affected, failed and categorized.

To set them you need to set Java system property smart.testing to one or more strategies in comma-separated value form.

Smart Testing is able to auto correct misspelled strategies in case of setting Java System property smart.testing.autocorrect to true.

Subsequent sections dive deeper into each and every one of them.

3.2.1. New

New strategy uses SCM information (currently only Git is supported) to detect new tests and treat them as important tests to run them first (ordering) or filtered (selecting).

So internally this strategy inspects your SCM history (and local changes) and all tests that were added (so they are effectively new) are marked as important ones.

This strategy can be configured in several ways through Java system properties:

  • scm.range.head and scm.range.tail are used to set the range of commits you want to inspect for new tests. The values can be commit ids or using HEAD notation. For example: -Dscm.range.head=HEAD -Dscm.range.tail=HEAD~ By default if not specified, default value is HEAD.

  • scm.last.changes can be used to set the number of commits from HEAD that you want to inspect. For example -Dscm.last.changes=3 will be equivalent as -Dscm.range.head=HEAD -Dscm.range.tail=HEAD~.

Those properties might be in particular handy when used in the CI builds. For example in Jenkins when using Git Plugin you can configure your build as follows: $ mvn clean verify -Dsmart.testing=new, affected -Dscm.range.head=GIT_COMMIT -Dscm.range.tail=GIT_PREVIOUS_COMMIT

Currently not committed changes (those that are added and untracked) are considered as important tests as well. This effectively means that if you don’t specify any commit range, only these information is considered.

3.2.2. Changed

Changed strategy is like New strategy, but it uses only tests that are modified (they were already committed in the past) instead of new ones.

In this strategy not committed changes (those that are modified or changed) are considered as important tests as well.

Usually new and changed strategies are used together -Dsmart.testing=new, changed.

3.2.3. Affected

Affected strategy uses a different approach to choose what are the important tests to run first (ordering) or filtered (selecting). This strategy also relies on SCM information but in this case it retrieves any new or modified business class between commits range and local changes as well.

scm.range.head, scm.range.tail and scm.last.changes Java system properties are valid as well.

When this strategy gets all changes then inspect all tests of current project checking which ones imports these classes. If the test exercises a business class that has been modified, we treat it as important so it will be executed earlier in the test plan.

About transitivity

Our approach does not find only direct test classes related to the business code which was changed, but takes into an account transitivity. This means that any import of a business class is considered as a dependency of the test too.

Suppose we have ATest.java which imports A.java. At the same time A.java imports B.java (ATest → A → B). If B.java is modified, then ATest.java is considered an important test too.

By default this import transitivity is applied to all imports except the ones from java.

Sometimes you might want to stop this transitivity before reaching these imports, for example in case of developing an application with any third-party library, you’d probably want to exclude its imports. Or maybe just include imports from your business code, for example all imports from org.superbiz.

Affected provides a way to provide inclusions and exclusions in smart-testing.yml:

smart-testing.yml
strategies: affected
strategiesConfiguration:
    affected:
      transitivity: true
      exclusions:
         - org.springframework.*
         - org.apache.commons.*
      inclusions:
         - org.mysuperbiz.*

You can overwrite inclusions/exclusions by using smart.testing.affected.inclusions/smart.testing.affected.exclusions which accepts comma-separated values. For example: -Dsmart.testing.affected.inclusions=org.mysuperbiz.*.

Exclusions has precedence over inclusions.

You can also disable transitivity by setting -Dsmart.testing.affected.transitivity to false.

This strategy is currently only applicable for white box testing approach. At this point our approach is to analyze direct code dependencies, but we are working on broader use cases.
At this moment, this strategy does not work with Java 9.
Explicitly Set

By default affected strategy uses imports of tests to build the graph of collaborators. This approach is fine for unit tests (white box tests) but might not work in all cases of high level tests (black box test).

There are some test technologies that allows you to deploy an application locally and then run tests against it. For example Wildfly Swarm has @DefaultDeployment annotation or Spring (Boot) deploys all application automatically.

This means that production classes are not directly imported into the test, so there is no way to get a relationship between test and production classes. For this reason affected provides an annotation to set package(s) of production classes that are deployed by the test.

The first thing you need to do is register following artifact into your build script: org.arquillian.smart.testing:api:<version>.

Then you can annotate your test with org.arquillian.smart.testing.strategies.affected.ComponentUnderTest annotation.

For example:

@ComponentUnderTest(packages = "org.arquillian.smart.testing.strategies.affected.fakeproject.main.superbiz.*")
public class ZTest {

In previous example all classes belonging to packages and subpackages specified at packages attribute are considered as classes used by the test.

You can also use packageOf attribute to set a reference class. With this attribute all classes that are placed in the same package as the reference class are considered as classes used by the test. With this approach your tests are resilient to package name changes.

If none of the attributes are set, then all production classes with same package as test and its subpackages are added automatically as classes used by the test.

Regular Files

By default affected strategy uses imports of tests to build the graph of collaborators or if you use the Explicitly Set, you can explicitly set class dependencies from an annotation. This approach is fine for Java classes, but a project usually contains another kind of files such as configuration files like persistence.xml, deployment files like kubernetes.json or build tool like pom.xml.

You can react to these file changes as well and execute some specific tests in these cases by using a special annotation provided by affected strategy.

The first thing you need to do is register following artifact into your build script: org.arquillian.smart.testing:api:<version>.

Then you can annotate your test with org.arquillian.smart.testing.strategies.affected.WatchFile annotation.

For example:

@WatchFile("src/main/resources/META-INF/persistence.xml")
public class XTest {

In previous example when src/main/resources/META-INF/persistence.xml file is added or modified, then the test is eligible to be run.

Location is relative to current project/module. This means that if you want to refer to a file that it is in another module, you need to navigate to parent folder by using .. symbol. For example: ../myothermodule/src/main/resources/applicationcontext.xml.

3.2.4. Failed

Failed strategy just gets all tests that failed from previous executions and mark them as important tests to run first (ordering) or not filtered (selecting). When the selecting mode is used then, by default, the resolution is on the test methods level (selects the particular failed test methods) - you can disable it by setting the parameter -Dsmart.testing.failed.methods=false

This strategy uses the JUnit XML report for reading past executions. All reports from previous local build are automatically copied by the maven extension to a temp directory ${project.directory}/.smart-testing/temporary/reports and when the build is finished the directory is removed.

3.2.5. Categorized

Categorized strategy selects tests based in the JUnit 4 annotation @Category or JUnit 5 annotations @Tag and @Tags (including meta annotations). With this strategy, you can easily set which classes or methods you want to run (or don’t want to run) as you would in the configuration of the maven-surefire-plugin without any need of changing your pom.xml file with complicated profiles.

In case of JUnit 4, you can say which categories should be executed by saying either the fully qualified names of the classes or just simple class names (case sensitivity can be set via configuration). For more information about the parameters see Configuration Options.

4. Configuration File

You can use smart-testing YAML file at ${project.dir}/smart-testing.yml or at ${project.dir}/smart-testing.yaml.

You can also point to a custom configuration file from a different project by setting the smart.testing.config property.

Configuration file looks as follows:

inherit: ../smart-testing.yml (1)
mode: ordering (2)
strategies: new, changed, affected (3)
applyTo: surefire (4)
debug: true (5)
disable: false (6)
report:
    enable: true (7)
scm:
    range:
      head: HEAD (8)
      tail: HEAD~2 (9)
autocorrect: true (10)
customStrategies:  (11)
    - smart.testing.strategy.cool=org.arquillian.smart.testing:strategy-cool:1.0.0
    - smart.testing.strategy.experimental=org.arquillian.smart.testing:strategy-experimental:1.0.0
strategiesConfiguration: (12)
    affected:
      transitivity: true (13)
      exclusions: (14)
         - org.package.*
         - org.arquillian.package.*
      inclusions: (15)
         - org.package.exclude.*
         - org.arquillian.package.exclude.*
customProviders: (16)
    - org.foo:my-custom-provider=fully.qualified.name.to.SurefireProviderImpl

1 Config file’s absolute or relative path from where Smart Testing overwrites parameters not defined in current config file.
2 This defines mode to be used by Smart Testing.
3 This defines strategies to be used while finding important tests.
4 This defines plugin to be used for Smart Testing.
5 This enables debug logs for Smart Testing.
6 This disables Smart Testing if set to true.
7 This enables Smart Testing report if set to true.
8 This sets first commit sha or HEAD notation for inspecting changes.
9 This sets last commit sha or HEAD notation for inspecting changes.
10 This defines if smart testing should auto correct misspelled strategies.
11 This defines the pair key/value of custom strategies as list.
12 This defines list of strategy related configurations that you want to apply while applying strategy.
13 This enables transitivity.
14 This defines list of packages to be excluded while applying transitivity.
15 This defines list of packages to be included while applying transitivity.
16 GroupId and artifactId with fully qualified name of the implementation of SurefireProvider the execution should be delegated to.

All parameters in configuration file are optional. If you haven’t used any parameter in configuration file, Smart testing will use default value for that parameter. You can look at references for default value of parameter.

However you can overwrite all configuration options using system properties supported by Smart Testing. You can look references for all supported system properties.

4.1. How configuration is applying to each module?

Smart Testing is looking in each module’s root dir for configuration file. If it didn’t find config file there, it’ll look recursively for the first config file in it’s parent dir.

4.2. How Configuration parameters are inherited?

If config file has inherit defined with absolute or relative path of config file, then Smart Testing will lookup for all undefined properties in config file defined using inherit. However it won’t lookup for disable parameter.

If config file doesn’t contain inherit parameter. Then Smart Testing won’t inherit any configuration properties.

4.3. Configuration File per Maven Module:

Let’s take a look into following example:

e.g.

parent
 - config
   - smart-testing.yml      (1)
   - api
   - impl-base
   - spi
     - smart-testing.yml     (2)
 - container
    - api
    - impl-base
 - smart-testing.yml         (3)

Configuration files used in above example:

parent/config/smart-testing.yml
inherit: ../smart-testing.yml
strategies: affected
parent/config/spi/smart-testing.yml
strategies: affected,new
parent/smart-testing.yml
strategies: new
scm:
  lastChanges: 1

Configuration file selection for the above example will be as follows:

  • config/api - parent/config/smart-testing.yml

  • config/impl-base - parent/config/smart-testing.yml

  • config/spi - parent/config/spi/smart-testing.yml

  • container/api - parent/smart-testing.yml

  • container/impl-base - parent/smart-testing.yml

1 inside this file we have inherit: ../smart-testing.yml, so it will take all undefined properties in this file from the one in parent/smart-testing.yml.
2 inside this file we have only strategies: affected,new, so it won’t inherit from any other configuration file.
3 inside this file we have strategies: new & scm: lastChanges: 1 defined.(You can set it using respective system property also).
In case of mvn extension, whenever you are using configuration file per maven module, make sure to define strategies in project’s parent configuration file or using system property. If you haven’t defined strategy in parent config, Smart testing will load parent configuration having no strategy defined which will disable ST by saying Smart Testing is disabled. Reason: strategy not defined. If you are running tests against range of commits locally, you need to also define scm properties in project’s parent config file or using system properties. If you haven’t defined it in parent config, Smart Testing will use default scm configuration with head as HEAD & tail as HEAD~0 due to which Smart Testing won’t find required changes in range of commits.

4.4. Configuration File Reference

The smart-testing.yml file is a YAML file defining required configuration to configure Smart Testing.

You can use either a .yml or .yaml extension for this file.
Field Description

inherit

This is used to define absolute or relative path to parent configuration file

strategies

This is used to define required strategies to find important tests. Look at strategies for all supported options.

mode

This is used to select mode for Smart Testing. Look at modes for all supported options & default value.

applyTo

This applies smart testing to use with surefire or failsafe plugin definition.

debug

This option runs smart testing in debug mode.

disable

This disables smart testing extension without removing it.

report

This configures report options for smart testing. Look at Report Options for all available options.

scm

To run Smart Testing with SCM configuration. You can either define range or lastChanges. Look at Scm Options for all available options.

autocorrect

This configures Smart Testing to auto correct misspelled strategies to the closest one. For example in case of user set strategies to nwe, if autocorrect is enabled then it is automatically changed to new.

customStrategies

A list of custom strategies in the form of key/value. It is important to notice that the key part must be prefixed with smart.testing.strategy..

strategiesConfiguration

A list of strategies with it’s configuration.

customProviders

A pair of groupId and artifactId with fully qualified name of SurefireProvider implementation the test execution should be delegated to. For more information see Custom Surefire Providers

4.4.1. Report Options

Field Description

enable

This generates smart testing report with selected tests.

4.4.2. Scm Options

Field Description

range

This configures range for Scm configuration. Look at Range Options for all available options.

lastChanges

This is used to set the number of commits from HEAD that you want to inspect.

4.4.3. Range Options

Field Description

head

Sets first commit sha or HEAD notation.

tail

Sets last sha or HEAD notation.

4.4.4. Affected Configuration Options

Field Description

transitivity

Sets transitivity to find all affected tests.

exclusions

Sets list of packages to be excluded while applying transitivity.

inclusions

Sets list of packages to be included while applying transitivity.

4.4.5. Categorized Configuration Options

Field Description

categories

List of categories the strategy should be looking for (can be either whole fully qualified names or only simple class names). If nothing is set, but the strategy is used, then all classes are selected.

excludedCategories

List of categories the strategy should exclude from looking for (can be either whole fully qualified names or only simple class names)

caseSensitive

Sets case sensitivity of the category names. (default is false)

methods

Sets if the resolution of categories/tags should be performed only on test classes or also on test methods. (default is true)

4.4.6. Failed Configuration Options

Field Description

methods

Sets if the strategy should select the particular failed test methods or take the whole classes. (default is true)

5. Usage

5.1. Examples

After installing Smart Testing Maven Extension you can use any goal you use with maven-surefire-plugin or maven-failsafe-plugin.

To configure it you only need to pass from CLI the Java system properties. Let’s see some examples:

Remember that the default mode is selecting.

You want to run only tests that you’ve just added or modified locally

mvn clean test -Dsmart.testing="new, changed"

You want to run all tests but given priority to the latest tests added or modified

mvn clean test -Dsmart.testing.mode=ordering -Dscm.last.changes=1 -Dsmart.testing="new, changed"

You want to run only tests that validates new or modified business classes locally

mvn clean test -Dsmart.testing="affected"

When you are running Smart Testing, you’ll see following logs in the console, showing your current configuration:

[INFO] [Smart Testing Extension] Applied strategies: [new, affected]
[INFO] [Smart Testing Extension] Applied usage: [selecting]

This can be used as feedback to check that the extension has been installed correctly.

5.2. Disabling Smart Testing

Sometimes you might want to disable smart testing extension without removing it.

To do it you just need to set smart.testing.disable Java system property to true and then tests will run with standard surefire / failsafe plugins.

5.3. Plugin Selection

Smart testing extension hooks into surefire and failsafe lifecycle to provide the order/selection of tests to execute.

If you only want to use smart testing on surefire or failsafe plugin definition.

To do it you just need to set smart.testing.apply.to with either surefire or failsafe to just enable in one of them.

5.4. Debug Mode

In order to get diagnostic information related to Smart Testing extension, you can enable the debug mode by either setting the system property smart.testing.debug or by setting the mvn debug output (-X) flag to have the entire build execution in the debug mode (available out of the box in Maven).

Debug logs provide information related to smart testing execution, system properties set by the user, test selection information. In addition for each and every module we store modified pom.xml at target/smart-testing/reporting as smart-testing-effective-pom.xml

All Smart Testing related logs can be found with the prefix Smart Testing Extension -

5.5. Specifying Concrete Set of Test Classes

During development, you may want to run a single test class or a set of concrete test classes. You can do this by setting the maven test property to specific test class(es).

5.5.1. To Run a Single Test

mvn test -Dtest=SampleTest

This configuration disables smart testing and executes only the specified sample test.

5.5.2. To Run Multiple Test Classes

You can also choose to run only a concrete set of multiple test classes.

mvn test -Dtest=SampleOneTest, SampleTwoTest

When multiple specific tests are set without any pattern, smart testing is disabled just like the case for single test.

You can further choose to execute multiple test classes in combination with smart testing’s selecting and ordering mode by using patterns supported by Maven.

mvn test -Dtest=Sample*Test

When used with ordering mode, the specified tests are executed in the order defined by the smart testing strategy used. If none is applicable to the strategy, they are executed in order defined by Surefire.

When used with selecting mode, the specified tests that fit the defined strategy are executed. However, if none fulfil the strategy criteria, no tests would be executed.

5.6. Skipping Tests

To skip executing the tests for a particular project, Smart Testing aligns with Maven Surefire/Failsafe plugins' skip test functionality and respects system properties skipTests, skipITs and maven.test.skip.

5.7. Running Examples

5.7.1. Execute Smart Testing with Changed strategy and Selecting Mode

5.7.2. Execute Smart Testing with Affected strategy and Selecting Mode

6. Registering Custom Strategies

By default Smart Testing comes with a set of strategies such as new, changed, affected, failed or categorized, but you can implement your own strategies and registering them.

To do it you just need to follow next four steps:

  • Implement org.arquillian.smart.testing.spi.TestExecutionPlannerFactory Java SPI and registering as such.

  • Implement org.arquillian.smart.testing.spi.TestExecutionPlanner interface.

  • Register strategy on Smart Testing configuration file (optional).

6.1. Implementing TestExecutionPlannerFactory

TestExecutionPlannerFactory is a Java SPI interface that manages the creation of the strategy. You need to implement three simple methods, two related with the name of the strategy and one that generates the strategy class.

As an example from failed strategy:

FailedTestsDetectorFactory.java
import java.io.File;
import org.arquillian.smart.testing.api.TestVerifier;
import org.arquillian.smart.testing.configuration.Configuration;
import org.arquillian.smart.testing.spi.StrategyConfiguration;
import org.arquillian.smart.testing.spi.TestExecutionPlanner;
import org.arquillian.smart.testing.spi.TestExecutionPlannerFactory;

import static org.arquillian.smart.testing.strategies.failed.FailedTestsDetector.FAILED;

public class FailedTestsDetectorFactory implements TestExecutionPlannerFactory {

    @Override
    public String alias() {
        return FAILED;
    } (1)

    @Override
    public boolean isFor(String name) {
        return alias().equalsIgnoreCase(name);
    }

    @Override
    public TestExecutionPlanner create(File projectDir, TestVerifier verifier, Configuration configuration) {
        return new FailedTestsDetector(projectDir, configuration);
    } (2)

    @Override
    public StrategyConfiguration strategyConfiguration() {
        return new FailedConfiguration();
    }
}
1 Returns the alias of the strategy
2 Returns an instance of TestExecutionPlanner

Of course as any other Java SPI, you need to register it in META-INF/services.

META-INF/services/org.arquillian.smart.testing.spi.TestExecutionPlannerFactory
org.arquillian.smart.testing.strategies.failed.FailedTestsDetectorFactory

6.2. Implementing TestExecutionPlanner

TestExecutionPlanner is the implementation of the strategy. Apart from the method getName() that returns a name of the strategy there are two important methods:

    Collection<TestSelection> selectTestsFromNames(Iterable<String> testsToRun);

    Collection<TestSelection> selectTestsFromClasses(Iterable<Class<?>> testsToRun);

These methods return a collection of the tests that must be executed. There is only one call of one of these two methods - it won’t happen that both methods are called at once. The reason for the existence of these two methods is that in some environments there aren’t available Classes for the tests, but just the names.

For example, failed strategy is implemented as:

FailedTestsDetector.java
import java.io.File;
import java.util.Collection;
import org.arquillian.smart.testing.TestSelection;
import org.arquillian.smart.testing.configuration.Configuration;
import org.arquillian.smart.testing.spi.JavaSPILoader;
import org.arquillian.smart.testing.spi.TestExecutionPlanner;

public class FailedTestsDetector implements TestExecutionPlanner {

    static final String FAILED = "failed";

    private final File projectDir;
    private final FailedConfiguration strategyConfig;

    public FailedTestsDetector(File projectDir, Configuration configuration) {
        this.projectDir = projectDir;
        strategyConfig = (FailedConfiguration) configuration.getStrategyConfiguration(FAILED);
    }

    @Override
    public Collection<TestSelection> selectTestsFromNames(Iterable<String> testsToRun) { (1)
        return getTests();
    }

    @Override
    public Collection<TestSelection> selectTestsFromClasses(Iterable<Class<?>> testsToRun) { (2)
        return getTests();
    }

    public Collection<TestSelection> getTests() {
        InProjectTestReportLoader reportLoader = new InProjectTestReportLoader(new JavaSPILoader(), projectDir);
        return TestResultsFilter.getFailedTests(strategyConfig, reportLoader.loadTestResults());
    }

    @Override
    public String getName() {
        return FAILED;
    }
}
1 Method that returns list of tests to execute based on the provided set of fully qualified class names of tests that were previously selected to be executed.
2 Method that returns list of tests to execute based on the provided set of tests classes that were previously selected to be executed.

6.2.1. ChangeStorage, ChangeResolver and TestResultParser

Smart Testing provides some services to get some common operations. These services can be used to get information from current project.

ChangeStorage

Change Storage service is used to get the changes (from SCM perspective) that has occurred in current project. This service does not read changes from SCM directly but from a file that has precalculated all changes at the beginning of the Smart Testing process.

Notice that in this way your strategy is more performant since, code does not need to calculate changes for each strategy.

To get it you just need to call new JavaSPILoader().onlyOne(ChangeStorage.class).get().

ChangeResolver

Change Resolver service is like ChangeStorage but in this case the changes are calculated directly from SCM.

To get it you just need to call new JavaSPILoader().onlyOne(ChangeResolver.class).get().

This service is usually not used directly in strategies since we recommend using ChangeStorage service for this purposes and just fall back to this service in case of error.

For example:

final Collection<Change> files = changeStorage.read(projectDir) (1)
    .orElseGet(() -> {
        logger.warn("No cached changes detected... using direct resolution");
        return changeResolver.diff(projectDir, configuration, getName()); (2)
    });
1 Changes are read from precalculated file
2 Fallback to calculate directly from SCM
TestResultParser

Test Result Parser is a service that reads previous test results. Currently it behaves like a parser for surefire report file.

To use it you just need to call new JavaSPILoader().onlyOne(TestResultParser.class).get().

6.3. Registering strategy

Finally you need to register strategy in Smart Testing configuration file where you set the strategy name as key and the Maven coordinates as value. This file is required so Smart Testing can automatically register the artifacts in build tool script.

As an example of registering previous strategy:

smart-testing.yml
customStrategies:
    - smart.testing.strategy.failed=org.arquillian.smart.testing:strategy-failed:1.0.0

It is important to notice that the key part must be prefixed with smart.testing.strategy..

If you need to specify any additional coordinate (such as type or classifier), you have to use the whole coordinates format: groupId:artifactId:packaging:classifier:version:scope

6.4. Alternative Approach

You can still use another approach to register the pair strategy.name=Maven:coordinates without configuring them in Smart Testing configuration file and it is using a System property.

For example previous example would be equivalent as running Smart Testing with next System property:

-Dsmart.testing.strategy.failed=org.arquillian.smart.testing:strategy-failed:1.0.0

Notice again that it must be prefixed with smart.testing.strategy..

7. Surefire Providers

Internally, Smart Testing uses its own implementation of SurefireProvider where calls Smart Testing API. Then, based on the information retrieved from the classpath, it decides which provider should be used for a delegation of the test execution - eg. JUnit4 or TestNG (in the same way as it is done inside of the Surefire plugin itself). But some projects may use their own surefire provider implementations or can specify one specific known provider that should scan the classpath and execute the test suite.

7.1. Known Surefire Providers

Known Surefire Providers are those implementations that are part of the Surefire project itself plus one implementation distributed as part of JUnit 5. The complete list of known providers:

  • JUnit 4

    • groupId: org.apache.maven.surefire

    • artifactId: surefire-junit4

    • provider implementation: org.apache.maven.surefire.junit4.JUnit4Provider

  • JUnit 47 and above

    • groupId: org.apache.maven.surefire

    • artifactId: surefire-junit47

    • provider implementation: org.apache.maven.surefire.junitcore.JUnitCoreProvider

  • JUnit 5

    • groupId: org.junit.platform

    • artifactId: junit-platform-surefire-provider

    • provider implementation: org.junit.platform.surefire.provider.JUnitPlatformProvider

  • TestNG

    • groupId: org.apache.maven.surefire

    • artifactId: surefire-testng

    • provider implementation: org.apache.maven.surefire.testng.TestNGProvider

When one of these providers is specified inside of the Surefire’s dependencies, then Smart Testing is able to automatically detect it and delegate the execution to it.

7.2. Custom Surefire Providers

If some unknown provider (or some other SurefireProvider implementation) is used then it is necessary to set this provider either in the configuration file or using system properties. The format is a pair of groupId and artifactId with a fully qualified name of the class that implements SurefireProvider interface and that should be loaded. For JUnit 5 provider it would look like this (in case of configuration file):

customProviders:
    - org.junit.platform:junit-platform-surefire-provider=org.junit.platform.surefire.provider.JUnitPlatformProvider

If this is specified then Smart Testing will detect the dependency and delegate the test execution to it. System property dedicated for this usage is smart.testing.custom.providers

8. Reports

8.1. Build Artifacts

To identify which tests were selected to run looking in the logs (or even using some CLI tricks with grep) is not the most efficient way to get a quick answer to:

  • How tests are ordered?

  • To which strategy they belong?

  • What was the configuration used?

Smart Testing provides concise report in the XML format with all this information at hand.

Here’s how the sample report looks like:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<smart-testing-execution>
    <module>core</module>
    <executionConfiguration>
        <usageMode>selecting</usageMode>
        <strategies>
            <strategy>new</strategy>
            <strategy>changed</strategy>
        </strategies>
        <properties>
            <property name="smart.testing" value="new,changed"/>
            <property name="smart.testing.mode" value="selecting"/>
        </properties>
    </executionConfiguration>
    <selection>
        <tests>
            <test name="org.arquillian.smart.testing.report.ExecutionReporterTest" strategies="new,changed"/>
            <test name="org.arquillian.smart.testing.report.ExecutionReporterUsingPropertyTest" strategies="new"/>
        </tests>
    </selection>
</smart-testing-execution>

8.2. Configuration

By default this feature is disabled. You can enable it by setting property smart.testing.report.enable to true. If you use the property, then you get the generated reports in target/smart-testing/reporting for every module separately.

If you are running Smart Testing in debug mode using property smart.testing.debug or Maven build output in debug mode using -X or --debug then Smart Testing will generate report.

9. Jenkins

So far you’ve seen how to use Smart Testing from developer perspective (running on local machine). But ultimately your software is going to be built on CI/CD server and saving time there means more resources for other projects.

One of important things to take into consideration is that meanwhile on the developer’s machine selecting mode might be the one used most frequently, in CI/CD environment you should consider executing the build in the ordering mode at some point (let it be regular build or a step in the pipeline). You can read more about modes at Modes section.

In case of development machine, you’ll probably want to build simply against the local changes, but in case of CI/CD environment, probably the changes you want to take into consideration are those between the commits you are going to run the build.

Let’s see how to configure Smart Testing in Jenkins.

9.1. Jenkins Freestyle project

Freestyle project is the most basic way of creating builds with Jenkins.

To use Smart Testing in this kind of project, you only need to create a build step of kind Execute Shell/Execute Windows batch command running Maven with required Smart Testing configuration parameters.

mvn -Dsmart.testing="new, affected"
    -Dsmart.testing.mode=ordering
    -Dscm.range.head=${GIT_COMMIT}
    -Dscm.range.tail=${GIT_PREVIOUS_COMMIT}
    test

On the next figure you can see the step configuration.

st jenkins freestyle job
Figure 1. Freestyle Execute Shell Configuration

After setting this up you are ready to run the build.

What you will notice in this case is that since the configured mode is ordering, all tests are going to be executed, but the ones marked as important by new and affected strategies are executed first.

9.2. Jenkins Pipeline

Jenkins Pipeline is a group of plugins which support implementing and integrating continuous delivery pipelines into Jenkins.

The definition of a Jenkins Pipeline is typically written into a text file (called a Jenkinsfile) which in turn is checked into a project’s source control repository

To run Smart Testing in Jenkins Pipeline you need to manually call checkout scm process to get access to GIT_COMMIT and PREVIOUS_GIT_COMMIT variables.

This might not be a problem if you are using scripted pipeline but in case of using declarative pipeline, since the checkout process is done automatically. you have no access to GIT_COMMIT and PREVIOUS_GIT_COMMIT variables.

So in next snippet you can see an example of how to use declarative pipeline with Smart Testing:

Jenkinsfile
pipeline {
    options {
        skipDefaultCheckout()
    }
    agent any
    stages {
        stage('Compile and Test') {
            steps {
                script {
                    def scmVars = checkout scm
                    sh "mvn -Dsmart.testing='new, affected' -Dsmart.testing.mode=ordering -Dscm.range.head=${scmVars.GIT_COMMIT} -Dscm.range.tail=${scmVars.GIT_PREVIOUS_COMMIT} test"
                }
            }
            post {
                success {
                    junit 'target/surefire-reports/**/*.xml'
                }
            }
        }
    }
}

There are few important things you have to keep in mind when adjusting your Jenkinsfile if you are using declarative script approach.

First of all you need to disable automatic checkout. Then you need to manually call checkout and store the result into a variable. Finally you can call Maven, getting Git parameters from scmVars.

9.3. Smart Testing Jenkins Pipeline Shared Library

Instead of running shell script using mvn call, you can use Smart Testing Jenkins Pipeline Shared Library which simplifies the configuration of Smart Testing in Jenkins.

9.3.1. Installing

To install this shared library you basically have two ways:

  • Installing as global shared libraries in Manage Jenkins >> Configure System >> Global Pipeline Libraries

  • Setting repository directly inside Jenkinsfile. For example @Library('github.com/organization/project@maven') _

You can read more about how to install shared libraries at https://jenkins.io/doc/book/pipeline/shared-libraries/

Using it

To use it, you just need to create the Jenkinsfile and add an stage similar to:

Jenkinsfile
@Library('smart-testing') _ (1)
pipeline {
  options { skipDefaultCheckout() } (2)
  agent any
  stages {
    stage('Compile and Test') {
      steps {
        script {
            def scmVars = checkout scm (3)
            smartTesting(strategies: 'new, changed', scmRange: scmVars) (4)
        }
      }
    }
  }
}
1 Refers a global shared library registered inside Jenkins with smart-testing name.
2 In declarative Jenkins pipeline you need to skip default checkout to get scm info.
3 Executed the checkout process and store info into scmVars variable.
4 Calls Maven with Smart Testing parameters.

Running previous script results as next shell call mvn '-Dsmart.testing=new, changed' -Dsmart.testing.mode=ordering -Dscm.range.head=653317b065ee8c54f9e806bc801b00d4c6371a20 -Dscm.range.tail=653317b065ee8c54f9e806bc801b00d4c6371a20 clean test

9.3.2. Configuration

Next configuration parameters are valid to be set in the smartTesting method.

Parameter Default Value Description

mvnHome

mvn

Sets Maven Home. If not set, then it runs mvn directly.

goals

clean, test

Sets goal of Maven execution.

profiles

Sets Maven profile.

strategies

Sets Smart Testing strategies.

scmRange

last commit

Map resulting of calling checkout scm. If not set, last commit is used.

extraParams

String that is concatenated before goals. Used to set any custom parameters such as Java system properties.

10. Reference Card

10.1. Installation

${maven.projectBasedir}/.mvn/extensions.xml
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
  <extension>
    <groupId>org.arquillian.smart.testing</groupId>
    <artifactId>maven-lifecycle-extension</artifactId>
    <version>${version}</version>
  </extension>
</extensions>

10.2. Configuration

Property Description Default Possible Values

smart.testing.mode

Set running mode

selecting

ordering, selecting

smart.testing

Set strategies in CSV

-

new, changed, affected, failed, categorized

smart.testing.disable

Disable Smart Testing

false

true, false

smart.testing.apply.to

Set plugin to apply Smart Testing in CSV

surefire, failsafe

surefire, failsafe

smart.testing.autocorrect

Enable auto correct of misspelled strategies

false

true, false

smart.testing.config

Set custom configuration file

-

custom configuration yml file

10.3. Strategies

Property Description Default Applicable Strategies

scm.range.tail

Set first commit id for inspecting changes

HEAD~0

new, changed, affected

scm.range.head

Set last commit id for inspecting changes

HEAD

new, changed, affected

scm.last.changes

Set the number of commits from HEAD that you want to inspect

0

new, changed, affected

smart.testing.affected.inclusions

Set classes to be included for scanning

affected

smart.testing.affected.exclusions

Set classes to be excluded for scanning

affected

smart.testing.affected.transitivity

Set transitivity enabled

true

affected

smart.testing.strategy.<strategyName>

Register custom strategies implementations

Value is the Maven coordinates of the custom strategy

smart.testing.categorized.categories

List of categories the strategy should be looking for (can be either whole fully qualified names or only simple class names)

-

categorized

smart.testing.categorized.excluded.categories

List of categories the strategy should exclude from looking for (can be either whole fully qualified names or only simple class names)

-

categorized

smart.testing.categorized.case.sensitive

Sets case sensitivity of the category names.

false

categorized

smart.testing.categorized.methods

Sets if the resolution of categories/tags should be performed only on test classes or also on test methods.

true

categorized

smart.testing.failed.methods

Sets if the strategy should select the particular failed test methods or take the whole classes

true

failed

10.4. Getting insights of the execution

Property Description Default

smart.testing.debug

enables the debug mode (alternatively you can use Maven debug output (-X) flag)

false

10.5. Configuration File

mode: ordering
strategies:
  - new
  - changed
  - affected
applyTo: surefire
debug: true
disable: false
report:
    enable: true
scm:
    lastChanges: 1
autocorrect: true

11. Developing

This section will be improved as we continue with the SPI design to let you hook your own logic if needed.

11.1. Test Bed

Test Bed brings a convenient way of writing functional tests. Here’s how the sample test looks like:

public class LocalChangesMixedStrategySelectionExecutionFunctionalTest {

    @ClassRule
    public static final GitClone GIT_CLONE = new GitClone(testRepository());

    @Rule
    public final TestBed testBed = new TestBed(GIT_CLONE);

    @Rule
    public final SmartTestingSoftAssertions softly = new SmartTestingSoftAssertions();

    @Test
    public void should_execute_all_new_tests_and_related_to_production_code_changes() throws Exception {
        // given
        final Project project = testBed.getProject();

        project.configureSmartTesting()
                .executionOrder(NEW, AFFECTED)
                .inMode(SELECTING)
            .enable();

        final Collection<TestResult> expectedTestResults = project
            .applyAsLocalChanges("Single method body modification - sysout",
                "Inlined variable in a method", "Adds new unit test");

        // when
        final TestResults actualTestResults = project.build("config/impl-base").run();

        // then
        softly.assertThat(actualTestResults.accumulatedPerTestClass())
            .containsAll(expectedTestResults)
            .hasSameSizeAs(expectedTestResults);

        softly.assertThat(project)
            .doesNotContainDirectory(TemporaryInternalFiles.getScmChangesFileName())
            .doesNotContainDirectory(SMART_TESTING_WORKING_DIRECTORY_NAME);
    }
}

We are using arquillian-core project (in a form of snapshot and squashed repo) for running those tests. You can find it here.

You can also run particular tests or all tests against defined repo. For the first option simply GitClone rule constructor

public GitClone(String repositoryUrl)

or define system property test.bed.repo which will replace the default repository for all the tests which are relying on it.

11.1.1. Design Idea

The main goal of Test Bed module is to automate this process and simplify functional testing of our test optimization tool by providing a programmatic way for these activities.

Key Principles
Project configuration

Easy way of configuring any Maven-based project in the test itself is through simple and self-explanatory fluent API.

project.configureSmartTesting()
            .executionOrder(NEW)
            .inMode(SELECTING)
       .enable();

This snippet will enable smart-testing extension in the background. It will also apply test selection criteria and execution mode (selecting meaning only tests which belong to categories defined in executionOrder).

Config file creation

We have self-explainatory fluent api to create configuration with required options.

final Configuration configuration = new ConfigurationBuilder()
        .mode(SELECTING)
        .strategies(AFFECTED)
        .scm()
            .lastChanges("2")
            .build()
        .build();

Also we have fluent api to create configuration file at the root of your maven project using given configuration

project.configureSmartTesting()
            .withConfiguration(configuration)
        .createConfigFile()
    .enable();

OR using required options

project.configureSmartTesting()
            .executionOrder(FAILED)
            .inMode(SELECTING)
        .createConfigFile()
    .enable();
Applying changes

We decided to apply changes to the selected code base by modifying sample repository directly. This is done by creating atomic commits directly rather than applying code changes in the tests themselves (e.g. by using Forge Roaster). This comes with several advantages:

  • Ability to test those changes directly on the project

  • Easier refactoring

  • Ability to analyze execution when tests are failing by simply opening modified project rather than debugging test itself

Creating atomic change, which will be later used for tests, consists of following steps:

  • Making commit with the change

  • Tagging it with additional message

This message is then used as a changeDescription in the project API, for example:

project.applyAsLocalChanges("Single method body modification - sysout",
    "Inlined variable in a method");

This will look in the repository for a tag with message Single method body modification - sysout and apply as local change (no commits created). And the same for Inlined variable in a method.

Creating such a tag on the commit is as easy as:

$ git tag affected_01 -m "Single method body modification - sysout"

Defining test result expectations

In order to verify if our tool works, we have to specify which tests are expected to be executed as well as their statuses. This is done as part of the commit, as at this point we know exactly what we want our tool to execute and what should be the status of our tests based on modifications we made. For example in arquillian-core we would like to test affected strategy, where we modify ConfigurationRegistrar#loadConfiguration by adding System.out.println to the method. This should result in finding two related tests ConfigurationRegistrarTestCase and SyspropReplacementInArqXmlTestCase. Both of them should be executed successfully.

In order to define these expectations we create commit with the following message:

affected: adds system.out to ConfigurationRegistrar#loadConfiguration (1)

+ org.jboss.arquillian.config.impl.extension.ConfigurationRegistrarTestCase (2)
+ org.jboss.arquillian.config.impl.extension.SyspropReplacementInArqXmlTestCase
1 The first line describes the change, so it’s up to you to provide meaningful information.
2 Next two lines define actual expectations.

For expectations, it’s the list of the executed tests (fully qualified names) prefixed with an expected status. In this case +, which indicates all test methods in the particular class should be executed successfully. See org.arquillian.smart.testing.ftest.testbed.testresults.Status for all prefixes.

Executing test optimization

Once we configured the project and applied changes we can build it. This is what we want to really test after all.

Under the hood it’s using Embedded Maven. In addition, after the build is finished, it returns actual test results so we can verify expected and actual test execution. It’s done this way:

final TestResults actualTestResults = project.build().run();
Usage of Git

Before the test class starts "project under test" is cloned and for each test method separated copy is created to execute all the changes and builds in isolation. Folders are named using the following convention:

GIT_REPO_FOLDER + "" + getClass().getSimpleName() + "" + name.getMethodName()

for example:

smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_only_execute_tests_related_to_multiple_commits_in_business_logic_when_affected_is_enabled/

11.1.2. Troubleshooting tests

You can execute embedded build in Test Bed in the debug mode. This gives better analysis of the build execution, as it lets us connect to the build execution to analyze behaviour of our extension in the runtime by debugging both Maven execution as well as Surefire Provider.

Built-in DSL debugging features

You can use project configuration DSL for enabling debug mode directly in the test code:

final TestResults actualTestResults = project
    .build()
    .options()
        .withSystemProperties("scm.range.head", "HEAD", "scm.range.tail", "HEAD~")
        .withDebugOutput()
        .withRemoteDebugging()
        .withRemoteSurefireDebugging()
    .configure()
    .run();

Setting up any of the debugging modes would set an agent in suspend mode for both Maven and Surefire execution, thus giving you time to attach debuggers to both.

Alternatively, this can be also achieved through system properties which you can set as part of your "Run Configuration" in the IDE or build command. Following system properties are available:

test.bed.mvn.remote.debug

enables remote debugging of the embedded maven build with suspend set to y (so the build will wait until we attach remote debugger to the port) and port 8000 to listen).

test.bed.mvn.remote.debug.suspend

true or false (both for maven and surefire)

test.bed.mvn.remote.debug.port

valid port used for remote debugging of maven execution

test.bed.mvn.surefire.remote.debug

enables remote debugging for surefire

test.bed.mvn.surefire.remote.debug.port

valid port used for remoting debugging of surefire execution

test.bed.mvn.debug.output

true or false - sets -X flag for embedded maven build for more details build log.

Configuration precedence

If both system property and the programmatic option is used system property takes precedence.

Storing project under test

Project Persistence feature is included in Test Bed to store repository used test’s embedded build. This lets you execute the same build outside of test execution for further analysis in case of failure.

By default this feature is enabled to copy repository only in case of test failure.

In order to copy repository for all tests irrespective of test result, explicitly set system property test.bed.project.persist to true.

Persisted project are located in target/test-bed-executions/[current timestamp]/ with name followed by naming convention:

getClass().getSimpleName() + "_" + name.getMethodName()