Smart Testing

We know which tests you want to run.

We all know that as the project evolves we keep adding more and more tests to ship our products confidently. This however has an impact on the build time and thus we waste time waiting for the most important question to be answered - "Did I break anything with my changes?".

Let it be your local development or a CI server - what if you could know this as soon as possible?

We created Smart Testing to give you the fastest possible feedback loop when it comes to executing your tests.

1. What is this?

Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health.

Suppose that you need to implement a new feature in your current project. If you are following TDD / BDD approach, you’ll write some test(s) to validate that the new feature behaves as expected. Then you’ll implement the new feature and when you get that all tests passes. Last step is to push this code to your source control, which probably will trigger a new build in your CI server running all the tests.

And at this point two things can happen:

The build takes several minutes and it does not fail.
The build takes several minutes but it fails because your new tests are failing (yes I know it worked in your machine).

Usually you don’t have that much influence on when your new tests are going to be executed. They could be the first ones, in that case your build will fail fast, or they might be the last ones, so you’ll need to wait some time until you get the failure.

Smart testing project aims to solve this uncertainty by ordering tests so that the first tests that could fail (because for example they are new or because failed previously) are going to be executed first.

1.1. Integrations

Currently Smart Testing supports Maven through Maven Extension mechanism and works together with maven-surefire-plugin and maven-failsafe-plugin.

1.2. Minimum requirements

Java: Requires Java 8. In case of using Affected strategy, Java 9 is not supported yet.
Maven: Tested with Maven 3.3.9 and 3.5.0 but any version above 3.1.X should work. Our recommendation is that you use at least 3.3.X version to avoid any extension API problems.
Surefire/Failsafe: Minimum version required is 2.19.1 or above. It is important to make sure that you have it defined in your pom.xml, as Maven comes with the default version which might not be compatible with our extension.

2. Installation

2.1. Maven Extension

Smart Testing is a Maven extension, so needs to be installed as is.

2.1.1. Maven >= 3.1.X

Get Smart Testing shaded copy from Bintray and copy it to M2_HOME/lib/ext

2.1.2. Maven >= 3.3.X

You can still use the process described at Maven >= 3.1.X or use the new core extension configuration mechanism by creating folder called .mvn in the root of your project and create inside it an extensions.xml file which registers the smart testing extension:

${maven.projectBasedir}/.mvn/extensions.xml

<?xml version="1.0" encoding="UTF-8"?>
<extensions>
  <extension>
    <groupId>org.arquillian.smart.testing</groupId>
    <artifactId>maven-lifecycle-extension</artifactId>
    <version>${version}</version>
  </extension>
</extensions>

3. Configuration

3.1. Modes

So far in What is this? section we explained that Smart testing changes the order of test execution plan to run first the important tests and then the rest. This is know as ordering mode, but that’s not the only one.

ordering: ordering mode as its name suggests orders the test execution plan so important tests are executed first and then the rest.
selecting: selecting mode just selects the important tests and execute them, skipping the rest of the tests.

The selecting is the default mode.

In order to define the mode use smart.testing.mode Java system property with either one.

3.2. Strategies

Until now, you’ve read that smart testing is changing test execution plan running or only including important tests. But how do we know which tests are important and which ones not?

There are several strategies that you can choose from which determine what are the important tests. Currently we have following strategies in place: new, changed, affected and failed.

To set them you need to set Java system property smart.testing to one or more strategies in comma-separated value form.

Subsequent sections dive deeper into each and every one of them.

3.2.1. New

New strategy uses SCM information (currently only Git is supported) to detect new tests and treat them as important tests to run them first (ordering) or filtered (selecting).

So internally this strategy inspects your SCM history (and local changes) and all tests that were added (so they are effectively new) are marked as important ones.

This strategy can be configured in several ways through Java system properties:

scm.range.head and scm.range.tail are used to set the range of commits you want to inspect for new tests. The values can be commit ids or using HEAD notation. For example: -Dscm.range.head=HEAD -Dscm.range.tail=HEAD~ By default if not specified, default value is HEAD.
scm.last.changes can be used to set the number of commits from HEAD that you want to inspect. For example -Dscm.last.changes=3 will be equivalent as -Dscm.range.head=HEAD -Dscm.range.tail=HEAD~~.

Those properties might be in particular handy when used in the CI builds. For example in Jenkins when using Git Plugin you can configure your build as follows: $ mvn clean verify -Dsmart.testing=new, affected -Dscm.range.head=GIT_COMMIT -Dscm.range.tail=GIT_PREVIOUS_COMMIT

Currently not committed changes (those that are added and untracked) are considered as important tests as well. This effectively means that if you don’t specify any commit range, only these information is considered.

3.2.2. Changed

Changed strategy is like New strategy, but it uses only tests that are modified (they were already committed in the past) instead of new ones.

In this strategy not committed changes (those that are modified or changed) are considered as important tests as well.

Usually new and changed strategies are used together -Dsmart.testing=new, changed.

3.2.3. Affected

Affected strategy uses a different approach to choose what are the important tests to run first (ordering) or filtered (selecting). This strategy also relies on SCM information but in this case it retrieves any new or modified business class between commits range and local changes as well.

scm.range.head, scm.range.tail and scm.last.changes Java system properties are valid as well.

When this strategy gets all changes then inspect all tests of current project checking which ones imports these classes. If the test exercises a business class that has been modified, we treat it as important so it will be executed earlier in the test plan.

About transitivity

Our approach does not find only direct test classes related to the business code which was changed, but takes into an account transitivity. This means that any import of a business class is considered as a dependency of the test too.

Suppose we have ATest.java which imports A.java. At the same time A.java imports B.java (ATest → A → B). If B.java is modified, then ATest.java is considered an important test too.

By default this import transitivity is applied to all imports except the ones from java.

Sometimes you might want to stop this transitivity before reaching these imports, for example in case of developing an application with any third-party library, you’d probably want to exclude its imports. Or maybe just include imports from your business code, for example all imports from org.superbiz.

Affected provides two ways to provide inclusions and exclusions:

inclusions/exclusions system proeprty: you can set inclusions and/or exclusions by using smart.testing.affected.inclusions/smart.testing.affected.exclusions which accepts comma-separated values. For example: -Dsmart.testing.affected.inclusions=org.mysuperbiz.*.
config properties file: you can create a properties file containing inclusions and/or exclusions and set its location using smart.testing.affected.config system property. For example -Dsmart.testing.affected.config=affected-configuration.properties

affected-configuration.properties

inclusions=org.mysuperbiz.*
exclusions=org.springframework.*, org.apache.commons.*

Exclusions has precedence over inclusions.

This strategy is currently only applicable for white box testing approach. At this point our approach is to analyze direct code dependencies, but we are working on broader use cases.

At this moment, this strategy does not work with Java 9.

3.2.4. Failed

Failed strategy just gets all tests that failed from previous executions and mark them as important tests to run first (ordering) or not filtered (selecting).

This strategy uses the JUnit XML report for reading past executions. All reports from previous local build are automatically copied by the maven extension to a temp directory ${project.directory}/.smart-testing/execution/reports and when the build is finished the directory is removed.

4. Usage

4.1. Examples

After installing Smart Testing Maven Extension you can use any goal you use with maven-surefire-plugin or maven-failsafe-plugin.

To configure it you only need to pass from CLI the Java system properties. Let’s see some examples:

Remember that the default mode is selecting.

You want to run only tests that you’ve just added or modified locally

$ mvn clean test -Dsmart.testing="new, changed"

You want to run all tests but given priority to the latest tests added or modified

$ mvn clean test -Dsmart.testing.mode=ordering -Dscm.last.changes=1 -Dsmart.testing="new, changed"

You want to run only tests that validates new or modified business classes locally

$ mvn clean test -Dsmart.testing="affected"

When you are running Smart Testing, you’ll see following logs in the console, showing your current configuration:

[INFO] [Smart Testing Extension] Applied strategies: [new, affected]
[INFO] [Smart Testing Extension] Applied usage: [selecting]

This can be used as feedback to check that the extension has been installed correctly.

4.2. Disabling Smart Testing

Sometimes you might want to disable smart testing extension without removing it.

To do it you just need to set smart.testing.disable Java system property to true and then tests will run with standard surefire / failsafe plugins.

4.3. Plugin Selection

Smart testing extension hooks into surefire and failsafe lifecycle to provide the order/selection of tests to execute.

If you only want to use smart testing on surefire or failsafe plugin definition.

To do it you just need to set smart.testing.apply.to with either surefire or failsafe to just enable in one of them.

4.4. Debug Mode

In order to get diagnostic information related to Smart Testing extension, you can enable the debug mode by either setting the system property smart.testing.debug or by setting the mvn debug output (-X) flag to have the entire build execution in the debug mode (available out of the box in Maven).

Debug logs provide information related to smart testing execution, system properties set by the user, test selection information. In addition for each and every module we store modified pom.xml at target/smart-testing as smart-testing-effective-pom.xml

4.5. Specifying Concrete Set of Test Classes

During development, you may want to run a single test class or a set of concrete test classes. You can do this by setting the maven test property to specific test class(es).

4.5.1. To Run a Single Test

$ mvn test -Dtest=SampleTest

This configuration disables smart testing and executes only the specified sample test.

4.5.2. To Run Multiple Test Classes

You can also choose to run only a concrete set of multiple test classes.

$ mvn test -Dtest=SampleOneTest, SampleTwoTest

When multiple specific tests are set without any pattern, smart testing is disabled just like the case for single test.

You can further choose to execute multiple test classes in combination with smart testing’s selecting and ordering mode by using patterns as supported by Maven.

$ mvn test -Dtest=Sample*Test

When used with ordering mode, the specified tests are executed in the order defined by the smart testing strategy used. If none is applicable to the strategy, they are executed in order defined by Surefire.

When used with selecting mode, the specified tests that fit the defined strategy are executed. However, if none fulfil the strategy criteria, no tests would be executed.

4.6. Skipping Tests

To skip executing the tests for a particular project, Smart Testing aligns with Maven Surefire/Failsafe plugins' skip test functionality and respects system properties skipTests, skipITs and maven.test.skip.

4.7. Running Examples

4.7.1. Execute Smart Testing with Changed strategy and Selecting Mode

4.7.2. Execute Smart Testing with Affected strategy and Selecting Mode

5. Reports

5.1. Build Artifacts

To identify which tests were selected to run looking in the logs (or even using some CLI tricks with grep) is not the most efficient way to get a quick answer to:

How tests are ordered?
To which strategy they belong?
What was the configuration used?

Smart Testing provides concise report in the XML format with all this information at hand.

Here’s how the sample report looks like:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<smart-testing-execution>
    <module>core</module>
    <executionConfiguration>
        <usageMode>selecting</usageMode>
        <strategies>
            <strategy>new</strategy>
            <strategy>changed</strategy>
        </strategies>
        <properties>
            <property name="smart.testing" value="new,changed"/>
            <property name="smart.testing.mode" value="selecting"/>
            <property name="smart.testing.report.dir" value="target/smart-testing-report"/>
            <property name="smart.testing.report.name" value="report"/>
        </properties>
    </executionConfiguration>
    <selection>
        <tests>
            <test name="org.arquillian.smart.testing.report.ExecutionReporterTest" strategies="new,changed"/>
            <test name="org.arquillian.smart.testing.report.ExecutionReporterUsingPropertyTest" strategies="new"/>
        </tests>
    </selection>
</smart-testing-execution>

5.2. Configuration

By default this feature is disabled. You can enable it by setting property smart.testing.report.enable to true.

You will get generated reports in target/smart-testing-report.xml by default.

To change the location of the generated output report, following properties should be set to the desired alternative location.

smart.testing.report.dir: To configure directory where generated reports should be stored.
smart.testing.report.name: To configure the file name of the generated report.

6. Reference Card

6.1. Installation

${maven.projectBasedir}/.mvn/extensions.xml

<?xml version="1.0" encoding="UTF-8"?>
<extensions>
  <extension>
    <groupId>org.arquillian.smart.testing</groupId>
    <artifactId>maven-lifecycle-extension</artifactId>
    <version>${version}</version>
  </extension>
</extensions>

6.2. Configuration

Property Description Default Possible Values

Property	Description	Default	Possible Values
`smart.testing.mode`	Set running mode	`selecting`	`ordering`, `selecting`
`smart.testing`	Set strategies in CSV	-	`new`, `changed`, `affected`, `failed`
`smart.testing.disable`	Disable Smart Testing	`false`	`true`, `false`
`smart.testing.apply.to`	Set plugin to apply Smart Testing in CSV	`surefire, failsafe`	`surefire`, `failsafe`

smart.testing.mode

Set running mode

selecting

ordering, selecting

smart.testing

Set strategies in CSV

new, changed, affected, failed

smart.testing.disable

Disable Smart Testing

false

true, false

smart.testing.apply.to

Set plugin to apply Smart Testing in CSV

surefire, failsafe

surefire, failsafe

6.3. Strategies

Property Description Default Applicable Strategies

Property	Description	Default	Applicable Strategies
`scm.range.tail`	Set first commit id for inspecting changes	`HEAD~0`	`new`, `changed`, `affected`
`scm.range.head`	Set last commit id for inspecting changes	`HEAD`	`new`, `changed`, `affected`
`scm.last.changes`	Set the number of commits from `HEAD` that you want to inspect	`0`	`new`, `changed`, `affected`
`smart.testing.affected.inclusions`	Set classes to be included for scanning		`affected`
`smart.testing.affected.exclusions`	Set classes to be excluded for scanning		`affected`
`smart.testing.affected.config`	Set location of affected configuration file		`affected`

scm.range.tail

Set first commit id for inspecting changes

HEAD~0

new, changed, affected

scm.range.head

Set last commit id for inspecting changes

HEAD

new, changed, affected

scm.last.changes

Set the number of commits from HEAD that you want to inspect

0

new, changed, affected

smart.testing.affected.inclusions

Set classes to be included for scanning

affected

smart.testing.affected.exclusions

Set classes to be excluded for scanning

affected

smart.testing.affected.config

Set location of affected configuration file

affected

6.4. Getting insights of the execution

Property Description Default

Property	Description	Default
`smart.testing.debug`	enables the debug mode (alternatively you can use Maven debug output `(-X)` flag)	`false`
`smart.testing.report.dir`	directory where generated reports should be stored	`target` of the current module being built
`smart.testing.report.name`	the file name of the generated report	`smart-testing-report.xml`

smart.testing.debug

enables the debug mode (alternatively you can use Maven debug output (-X) flag)

false

smart.testing.report.dir

directory where generated reports should be stored

target of the current module being built

smart.testing.report.name

the file name of the generated report

smart-testing-report.xml

7. Developing

This section will be improved as we continue with the SPI design to let you hook your own logic if needed.

7.1. Test Bed

Test Bed brings a convenient way of writing functional tests. Here’s how the sample test looks like:

public class LocalChangesMixedStrategySelectionExecutionFunctionalTest {

    @ClassRule
    public static final GitClone GIT_CLONE = new GitClone(testRepository());

    @Rule
    public final TestBed testBed = new TestBed(GIT_CLONE);

    @Test
    public void should_execute_all_new_tests_and_related_to_production_code_changes() throws Exception {
        // given
        final Project project = testBed.getProject();

        project.configureSmartTesting()
                .executionOrder(NEW, AFFECTED)
                .inMode(SELECTING)
            .enable();

        final Collection<TestResult> expectedTestResults = project
            .applyAsLocalChanges("Single method body modification - sysout",
                "Inlined variable in a method", "Adds new unit test");

        // when
        final TestResults actualTestResults = project.build().run();

        // then
        assertThat(actualTestResults.accumulatedPerTestClass()).containsAll(expectedTestResults).hasSameSizeAs(expectedTestResults);
        FileVerifier.assertThatFileIsNotPresent(project, SMART_TESTING_SCM_CHANGES);
        FileVerifier.assertThatFileIsNotPresent(project, SMART_TESTING_WORKING_DIRECTORY_NAME);
    }
}

We are using arquillian-core project (in a form of snapshot and squashed repo) for running those tests. You can find it here.

You can also run particular tests or all tests against defined repo. For the first option simply GitClone rule constructor

public GitClone(String repositoryUrl)

or define system property test.bed.repo which will replace the default repository for all the tests which are relying on it.

7.1.1. Design Idea

The main goal of Test Bed module is to automate this process and simplify functional testing of our test optimization tool by providing a programmatic way for these activities.

Key Principles

Project configuration

Easy way of configuring any Maven-based project in the test itself is through simple and self-explanatory fluent API.

project.configureSmartTesting()
            .executionOrder(NEW)
            .inMode(SELECTING)
       .enable();

This snippet will enable smart-testing extension in the background. It will also apply test selection criteria and execution mode (selecting meaning only tests which belong to categories defined in executionOrder).

Applying changes

We decided to apply changes to the selected code base by modifying sample repository directly. This is done by creating atomic commits directly rather than applying code changes in the tests themselves (e.g. by using Forge Roaster). This comes with several advantages:

Ability to test those changes directly on the project
Easier refactoring
Ability to analyze execution when tests are failing by simply opening modified project rather than debugging test itself

Creating atomic change, which will be later used for tests, consists of following steps:

Making commit with the change
Tagging it with additional message

This message is then used as a changeDescription in the project API, for example:

project.applyAsLocalChanges("Single method body modification - sysout",
    "Inlined variable in a method");

This will look in the repository for a tag with message Single method body modification - sysout and apply as local change (no commits created). And the same for Inlined variable in a method.

Creating such a tag on the commit is as easy as:

$ git tag affected_01 -m "Single method body modification - sysout"

Defining test result expectations

In order to verify if our tool works, we have to specify which tests are expected to be executed as well as their statuses. This is done as part of the commit, as at this point we know exactly what we want our tool to execute and what should be the status of our tests based on modifications we made. For example in arquillian-core we would like to test affected strategy, where we modify ConfigurationRegistrar#loadConfiguration by adding System.out.println to the method. This should result in finding two related tests ConfigurationRegistrarTestCase and SyspropReplacementInArqXmlTestCase. Both of them should be executed successfully.

In order to define these expectations we create commit with the following message:

affected: adds system.out to ConfigurationRegistrar#loadConfiguration (1)

+ org.jboss.arquillian.config.impl.extension.ConfigurationRegistrarTestCase (2)
+ org.jboss.arquillian.config.impl.extension.SyspropReplacementInArqXmlTestCase

1	The first line describes the change, so it’s up to you to provide meaningful information.
2	Next two lines define actual expectations.

For expectations, it’s the list of the executed tests (fully qualified names) prefixed with an expected status. In this case +, which indicates all test methods in the particular class should be executed successfully. See org.arquillian.smart.testing.ftest.testbed.testresults.Status for all prefixes.

Executing test optimization

Once we configured the project and applied changes we can build it. This is what we want to really test after all.

Under the hood it’s using Embedded Maven. In addition, after the build is finished, it returns actual test results so we can verify expected and actual test execution. It’s done this way:

final TestResults actualTestResults = project.build().run();

Usage of Git

Before the test class starts "project under test" is cloned and for each test method separated copy is created to execute all the changes and builds in isolation. Folders are named using the following convention:

GIT_REPO_FOLDER + "" + getClass().getSimpleName() + "" + name.getMethodName()

for example:

smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_only_execute_tests_related_to_multiple_commits_in_business_logic_when_affected_is_enabled/

7.1.2. Troubleshooting tests

You can execute embedded build in Test Bed in the debug mode. This gives better analysis of the build execution, as it lets us connect to the build execution to analyze behaviour of our extension in the runtime by debugging both Maven execution as well as Surefire Provider.

Built-in DSL debugging features

You can use project configuration DSL for enabling debug mode directly in the test code:

final TestResults actualTestResults = project
    .build()
    .options()
        .withSystemProperties("scm.range.head", "HEAD", "scm.range.tail", "HEAD~")
        .withDebugOutput()
        .withRemoteDebugging()
        .withRemoteSurefireDebugging()
    .configure()
    .run();

Setting up any of the debugging modes would set an agent in suspend mode for both Maven and Surefire execution, thus giving you time to attach debuggers to both.

Alternatively, this can be also achieved through system properties which you can set as part of your "Run Configuration" in the IDE or build command. Following system properties are available:

test.bed.mvn.remote.debug: enables remote debugging of the embedded maven build with suspend set to y (so the build will wait until we attach remote debugger to the port) and port 8000 to listen).
test.bed.mvn.remote.debug.suspend: true or false (both for maven and surefire)
test.bed.mvn.remote.debug.port: valid port used for remote debugging of maven execution
test.bed.mvn.surefire.remote.debug: enables remote debugging for surefire
test.bed.mvn.surefire.remote.debug.port: valid port used for remoting debugging of surefire execution
test.bed.mvn.debug.output: true or false - sets -X flag for embedded maven build for more details build log.

Configuration precedence

If both system property and the programmatic option is used system property takes precedence.

Storing project under test

Project Persistence feature is included in Test Bed to store repository used test’s embedded build. This lets you execute the same build outside of test execution for further analysis in case of failure.

By default this feature is enabled to copy repository only in case of test failure.

In order to copy repository for all tests irrespective of test result, explicitly set system property test.bed.project.persist to true.

Persisted project are located in target/test-bed-executions/[current timestamp]/ with name followed by naming convention:

getClass().getSimpleName() + "_" + name.getMethodName()