Build Status

We know which tests you want to run.

We all know that as the project evolves we keep adding more and more tests to ship our products confidently. This however has an impact on the build time and thus we waste time waiting for the most important question to be answered - "Did I break anything with my changes?".

Let it be your local development or a CI server - what if you could know this as soon as possible?

We created Smart Testing to give you the fastest possible feedback loop when it comes to executing your tests.

1. What is this?

Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health.

Suppose that you need to implement a new feature in your current project. If you are following TDD / BDD approach, you’ll write some test(s) to validate that the new feature behaves as expected. Then you’ll implement the new feature and when you get that all tests passes. Last step is to push this code to your source control, which probably will trigger a new build in your CI server running all the tests.

And at this point two things can happen:

  1. The build takes several minutes and it does not fail.

  2. The build takes several minutes but it fails because your new tests are failing (yes I know it worked in your machine).

Usually you don’t have that much influence on when your new tests are going to be executed. They could be the first ones, in that case your build will fail fast, or they might be the last ones, so you’ll need to wait some time until you get the failure.

Smart testing project aims to solve this uncertainty by ordering tests so that the first tests that could fail (because for example they are new or because failed previously) are going to be executed first.

1.1. Integrations

Currently Smart Testing supports Maven through Maven Extension mechanism and works together with maven-surefire-plugin and maven-failsafe-plugin.

1.2. Minimum requirements

Java

Requires Java 8.

Currently we don’t support Java 9 for Affected strategy.
Maven

Tested with Maven 3.3.9 and 3.5.0 but any version above 3.1.X should work. Our recommendation is that you use at least 3.3.X version to avoid any extension API problems.

Surefire/Failsafe

Minimum version required is 2.19.1 or above. It is important to make sure that you have it defined in your pom.xml, as Maven comes with the default version which might not be compatible with our extension.

2. Installation

2.1. Don’t make me think

Ok, just execute following snippet and you are all set:

curl -sSL https://git.io/v5jy6 | bash

You can also add these flags:

  • -l|--latest latest published artifact in Maven Central (that’s the default)

  • -v|--version= - specific version to install

For example:

curl -sSL https://git.io/v5jy6 | bash -s -- --latest

This script will automatically do what we described below. If you are curious keep on reading, but you may skip it too.

This script requires xmllint, and xsltproc, so make sure you have it installed.

2.2. Maven Extension

Smart Testing is a Maven extension, so depending on the version of Maven you have to follow slightly different approach of installing it.

2.2.1. Maven above 3.1.X

Get Smart Testing Extension shaded jar from Maven Central and copy it to M2_HOME/lib/ext.

2.2.2. Maven above 3.3.X

You can still use the process described at [Maven >= 3.1.X] or use the new core extension configuration mechanism by creating folder called .mvn in the root of your project and create inside it an extensions.xml file which registers the smart testing extension:

${maven.projectBasedir}/.mvn/extensions.xml
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
  <extension>
    <groupId>org.arquillian.smart.testing</groupId>
    <artifactId>maven-lifecycle-extension</artifactId>
    <version>${version}</version>
  </extension>
</extensions>

3. Configuration

This section explains configuration parameters of Smart Testing. Notice that they are all summarized at Reference Card section.

Also Smart Testing can be configured with a configuration file instead of System properties. You can read about configuration file at Configuration File.

3.1. Modes

So far in What is this? section we explained that Smart testing changes the order of test execution plan to run first the important tests and then the rest. This is know as ordering mode, but that’s not the only one.

ordering

ordering mode as its name suggests orders the test execution plan so important tests are executed first and then the rest.

selecting

selecting mode just selects the important tests and execute them, skipping the rest of the tests.

The selecting is the default mode.

In order to define the mode use smart.testing.mode Java system property with either one.

To get fast feedback loop, you can use surefire’s skip after N failures/errors feature by setting system property surefire.skipAfterFailureCount to N or by following configuration:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${version.surefire.plugin}</version>
    <configuration>
      <skipAfterFailureCount>N</skipAfterFailureCount>
    </configuration>
</plugin>

However this functionality cannot be fully guaranteed (real first failure) in concurrent mode due to race conditions. Read skipAfterFailureCount for more information.

3.2. Strategies

Until now, you’ve read that smart testing is changing test execution plan running or only including important tests. But how do we know which tests are important and which ones not?

There are several strategies that you can choose from which determine what are the important tests. Currently we have following strategies in place: new, changed, affected and failed.

To set them you need to set Java system property smart.testing to one or more strategies in comma-separated value form.

Smart Testing is able to auto correct misspelled strategies in case of setting Java System property smart.testing.autocorrect to true.

Subsequent sections dive deeper into each and every one of them.

3.2.1. New

New strategy uses SCM information (currently only Git is supported) to detect new tests and treat them as important tests to run them first (ordering) or filtered (selecting).

So internally this strategy inspects your SCM history (and local changes) and all tests that were added (so they are effectively new) are marked as important ones.

This strategy can be configured in several ways through Java system properties:

  • scm.range.head and scm.range.tail are used to set the range of commits you want to inspect for new tests. The values can be commit ids or using HEAD notation. For example: -Dscm.range.head=HEAD -Dscm.range.tail=HEAD~ By default if not specified, default value is HEAD.

  • scm.last.changes can be used to set the number of commits from HEAD that you want to inspect. For example -Dscm.last.changes=3 will be equivalent as -Dscm.range.head=HEAD -Dscm.range.tail=HEAD~.

Those properties might be in particular handy when used in the CI builds. For example in Jenkins when using Git Plugin you can configure your build as follows: $ mvn clean verify -Dsmart.testing=new, affected -Dscm.range.head=GIT_COMMIT -Dscm.range.tail=GIT_PREVIOUS_COMMIT

Currently not committed changes (those that are added and untracked) are considered as important tests as well. This effectively means that if you don’t specify any commit range, only these information is considered.

3.2.2. Changed

Changed strategy is like New strategy, but it uses only tests that are modified (they were already committed in the past) instead of new ones.

In this strategy not committed changes (those that are modified or changed) are considered as important tests as well.

Usually new and changed strategies are used together -Dsmart.testing=new, changed.

3.2.3. Affected

Affected strategy uses a different approach to choose what are the important tests to run first (ordering) or filtered (selecting). This strategy also relies on SCM information but in this case it retrieves any new or modified business class between commits range and local changes as well.

scm.range.head, scm.range.tail and scm.last.changes Java system properties are valid as well.

When this strategy gets all changes then inspect all tests of current project checking which ones imports these classes. If the test exercises a business class that has been modified, we treat it as important so it will be executed earlier in the test plan.

About transitivity

Our approach does not find only direct test classes related to the business code which was changed, but takes into an account transitivity. This means that any import of a business class is considered as a dependency of the test too.

Suppose we have ATest.java which imports A.java. At the same time A.java imports B.java (ATest → A → B). If B.java is modified, then ATest.java is considered an important test too.

By default this import transitivity is applied to all imports except the ones from java.

Sometimes you might want to stop this transitivity before reaching these imports, for example in case of developing an application with any third-party library, you’d probably want to exclude its imports. Or maybe just include imports from your business code, for example all imports from org.superbiz.

Affected provides two ways to provide inclusions and exclusions:

inclusions/exclusions system proeprty

you can set inclusions and/or exclusions by using smart.testing.affected.inclusions/smart.testing.affected.exclusions which accepts comma-separated values. For example: -Dsmart.testing.affected.inclusions=org.mysuperbiz.*.

config properties file

you can create a properties file containing inclusions and/or exclusions and set its location using smart.testing.affected.config system property. For example -Dsmart.testing.affected.config=affected-configuration.properties

affected-configuration.properties
inclusions=org.mysuperbiz.*
exclusions=org.springframework.*, org.apache.commons.*
Exclusions has precedence over inclusions.

You can also disable transitivity by setting -Dsmart.testing.affected.transitivity to false.

This strategy is currently only applicable for white box testing approach. At this point our approach is to analyze direct code dependencies, but we are working on broader use cases.
At this moment, this strategy does not work with Java 9.

3.2.4. Failed

Failed strategy just gets all tests that failed from previous executions and mark them as important tests to run first (ordering) or not filtered (selecting).

This strategy uses the JUnit XML report for reading past executions. All reports from previous local build are automatically copied by the maven extension to a temp directory ${project.directory}/.smart-testing/temporary/reports and when the build is finished the directory is removed.

4. Configuration File

You can use smart-testing YAML file at ${project.dir}/smart-testing.yml or at ${project.dir}/smart-testing.yaml.

Configuration file looks as follows:

mode: ordering (1)
strategies: new, changed, affected (2)
applyTo: surefire (3)
debug: true (4)
disable: false (5)
report:
    enable: true (6)
scm:
    range:
      head: HEAD (7)
      tail: HEAD~2 (8)
autocorrect: true  (9)

1 This defines mode to be used by Smart Testing.
2 This defines strategies to be used while finding important tests.
3 This defines plugin to be used for Smart Testing.
4 This enables debug logs for Smart Testing.
5 This disables Smart Testing if set to true.
6 This enables Smart Testing report if set to true.
7 This sets first commit sha or HEAD notation for inspecting changes.
8 This sets last commit sha or HEAD notation for inspecting changes.
9 This defines if smart testing should auto correct misspelled strategies.

All parameters in configuration file are optional. If you haven’t used any parameter in configuration file, Smart testing will use default value for that parameter. You can look at references for default value of parameter.

However you can overwrite all configuration options using system properties supported by Smart Testing. You can look references for all supported system properties.

4.1. Configuration File Reference

The smart-testing.yml file is a YAML file defining required configuration to configure Smart Testing.

You can use either a .yml or .yaml extension for this file.
Field Description

strategies

This is used to define required strategies to find important tests. Look at strategies for all supported options.

mode

This is used to select mode for Smart Testing. Look at modes for all supported options & default value.

applyTo

This applies smart testing to use with surefire or failsafe plugin definition.

debug

This option runs smart testing in debug mode.

disable

This disables smart testing extension without removing it.

report

This configures report options for smart testing. Look at Report Options for all available options.

scm

To run Smart Testing with SCM configuration. You can either define range or lastChanges. Look at Scm Options for all available options.

autocorrect

This configures Smart Testing to auto correct misspelled strategies to the closest one. For example in case of user set strategies to nwe, if autocorrect is enabled then it is automatically changed to new.

4.1.1. Report Options

Field Description

enable

This generates smart testing report with selected tests.

4.1.2. Scm Options

Field Description

range

This configures range for Scm configuration. Look at Range Options for all available options.

lastChanges

This is used to set the number of commits from HEAD that you want to inspect.

4.1.3. Range Options

Field Description

head

Sets first commit sha or HEAD notation.

tail

Sets last sha or HEAD notation.

5. Usage

5.1. Examples

After installing Smart Testing Maven Extension you can use any goal you use with maven-surefire-plugin or maven-failsafe-plugin.

To configure it you only need to pass from CLI the Java system properties. Let’s see some examples:

Remember that the default mode is selecting.

You want to run only tests that you’ve just added or modified locally

mvn clean test -Dsmart.testing="new, changed"

You want to run all tests but given priority to the latest tests added or modified

mvn clean test -Dsmart.testing.mode=ordering -Dscm.last.changes=1 -Dsmart.testing="new, changed"

You want to run only tests that validates new or modified business classes locally

mvn clean test -Dsmart.testing="affected"

When you are running Smart Testing, you’ll see following logs in the console, showing your current configuration:

[INFO] [Smart Testing Extension] Applied strategies: [new, affected]
[INFO] [Smart Testing Extension] Applied usage: [selecting]

This can be used as feedback to check that the extension has been installed correctly.

5.2. Disabling Smart Testing

Sometimes you might want to disable smart testing extension without removing it.

To do it you just need to set smart.testing.disable Java system property to true and then tests will run with standard surefire / failsafe plugins.

5.3. Plugin Selection

Smart testing extension hooks into surefire and failsafe lifecycle to provide the order/selection of tests to execute.

If you only want to use smart testing on surefire or failsafe plugin definition.

To do it you just need to set smart.testing.apply.to with either surefire or failsafe to just enable in one of them.

5.4. Debug Mode

In order to get diagnostic information related to Smart Testing extension, you can enable the debug mode by either setting the system property smart.testing.debug or by setting the mvn debug output (-X) flag to have the entire build execution in the debug mode (available out of the box in Maven).

Debug logs provide information related to smart testing execution, system properties set by the user, test selection information. In addition for each and every module we store modified pom.xml at target/smart-testing/reporting as smart-testing-effective-pom.xml

All Smart Testing related logs can be found with the prefix Smart Testing Extension -

5.5. Specifying Concrete Set of Test Classes

During development, you may want to run a single test class or a set of concrete test classes. You can do this by setting the maven test property to specific test class(es).

5.5.1. To Run a Single Test

mvn test -Dtest=SampleTest

This configuration disables smart testing and executes only the specified sample test.

5.5.2. To Run Multiple Test Classes

You can also choose to run only a concrete set of multiple test classes.

mvn test -Dtest=SampleOneTest, SampleTwoTest

When multiple specific tests are set without any pattern, smart testing is disabled just like the case for single test.

You can further choose to execute multiple test classes in combination with smart testing’s selecting and ordering mode by using patterns supported by Maven.

mvn test -Dtest=Sample*Test

When used with ordering mode, the specified tests are executed in the order defined by the smart testing strategy used. If none is applicable to the strategy, they are executed in order defined by Surefire.

When used with selecting mode, the specified tests that fit the defined strategy are executed. However, if none fulfil the strategy criteria, no tests would be executed.

5.6. Skipping Tests

To skip executing the tests for a particular project, Smart Testing aligns with Maven Surefire/Failsafe plugins' skip test functionality and respects system properties skipTests, skipITs and maven.test.skip.

5.7. Running Examples

5.7.1. Execute Smart Testing with Changed strategy and Selecting Mode

5.7.2. Execute Smart Testing with Affected strategy and Selecting Mode

6. Reports

6.1. Build Artifacts

To identify which tests were selected to run looking in the logs (or even using some CLI tricks with grep) is not the most efficient way to get a quick answer to:

  • How tests are ordered?

  • To which strategy they belong?

  • What was the configuration used?

Smart Testing provides concise report in the XML format with all this information at hand.

Here’s how the sample report looks like:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<smart-testing-execution>
    <module>core</module>
    <executionConfiguration>
        <usageMode>selecting</usageMode>
        <strategies>
            <strategy>new</strategy>
            <strategy>changed</strategy>
        </strategies>
        <properties>
            <property name="smart.testing" value="new,changed"/>
            <property name="smart.testing.mode" value="selecting"/>
        </properties>
    </executionConfiguration>
    <selection>
        <tests>
            <test name="org.arquillian.smart.testing.report.ExecutionReporterTest" strategies="new,changed"/>
            <test name="org.arquillian.smart.testing.report.ExecutionReporterUsingPropertyTest" strategies="new"/>
        </tests>
    </selection>
</smart-testing-execution>

6.2. Configuration

By default this feature is disabled. You can enable it by setting property smart.testing.report.enable to true. If you use the property, then you get the generated reports in target/smart-testing/reporting for every module separately.

If you are running Smart Testing in debug mode using property smart.testing.debug or Maven build output in debug mode using -X or --debug then Smart Testing will generate report.

7. Jenkins

So far you’ve seen how to use Smart Testing from developer perspective (running on local machine). But ultimately your software is going to be built on CI/CD server and saving time there means more resources for other projects.

One of important things to take into consideration is that meanwhile on the developer’s machine selecting mode might be the one used most frequently, in CI/CD environment you should consider executing the build in the ordering mode at some point (let it be regular build or a step in the pipeline). You can read more about modes at Modes section.

In case of development machine, you’ll probably want to build simply against the local changes, but in case of CI/CD environment, probably the changes you want to take into consideration are those between the commits you are going to run the build.

Let’s see how to configure Smart Testing in Jenkins.

7.1. Jenkins Freestyle project

Freestyle project is the most basic way of creating builds with Jenkins.

To use Smart Testing in this kind of project, you only need to create a build step of kind Execute Shell/Execute Windows batch command running Maven with required Smart Testing configuration parameters.

mvn -Dsmart.testing="new, affected"
    -Dsmart.testing.mode=ordering
    -Dscm.range.head=${GIT_COMMIT}
    -Dscm.range.tail=${GIT_PREVIOUS_COMMIT}
    test

On the next figure you can see the step configuration.

st jenkins freestyle job
Figure 1. Freestyle Execute Shell Configuration

After setting this up you are ready to run the build.

What you will notice in this case is that since the configured mode is ordering, all tests are going to be executed, but the ones marked as important by new and affected strategies are executed first.

7.2. Jenkins Pipeline

Jenkins Pipeline is a group of plugins which support implementing and integrating continuous delivery pipelines into Jenkins.

The definition of a Jenkins Pipeline is typically written into a text file (called a Jenkinsfile) which in turn is checked into a project’s source control repository

To run Smart Testing in Jenkins Pipeline you need to manually call checkout scm process to get access to GIT_COMMIT and PREVIOUS_GIT_COMMIT variables.

This might not be a problem if you are using scripted pipeline but in case of using declarative pipeline, since the checkout process is done automatically. you have no access to GIT_COMMIT and PREVIOUS_GIT_COMMIT variables.

So in next snippet you can see an example of how to use declarative pipeline with Smart Testing:

Jenkinsfile
pipeline {
    options {
        skipDefaultCheckout()
    }
    agent any
    stages {
        stage('Compile and Test') {
            steps {
                script {
                    def scmVars = checkout scm
                    sh "mvn -Dsmart.testing='new, affected' -Dsmart.testing.mode=ordering -Dscm.range.head=${scmVars.GIT_COMMIT} -Dscm.range.tail=${scmVars.GIT_PREVIOUS_COMMIT} test"
                }
            }
            post {
                success {
                    junit 'target/surefire-reports/**/*.xml'
                }
            }
        }
    }
}

There are few important things you have to keep in mind when adjusting your Jenkinsfile if you are using declarative script approach.

First of all you need to disable automatic checkout. Then you need to manually call checkout and store the result into a variable. Finally you can call Maven, getting Git parameters from scmVars.

8. Reference Card

8.1. Installation

${maven.projectBasedir}/.mvn/extensions.xml
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
  <extension>
    <groupId>org.arquillian.smart.testing</groupId>
    <artifactId>maven-lifecycle-extension</artifactId>
    <version>${version}</version>
  </extension>
</extensions>

8.2. Configuration

Property Description Default Possible Values

smart.testing.mode

Set running mode

selecting

ordering, selecting

smart.testing

Set strategies in CSV

-

new, changed, affected, failed

smart.testing.disable

Disable Smart Testing

false

true, false

smart.testing.apply.to

Set plugin to apply Smart Testing in CSV

surefire, failsafe

surefire, failsafe

smart.testing.autocorrect

Enable auto correct of misspelled strategies

false

true, false

8.3. Strategies

Property Description Default Applicable Strategies

scm.range.tail

Set first commit id for inspecting changes

HEAD~0

new, changed, affected

scm.range.head

Set last commit id for inspecting changes

HEAD

new, changed, affected

scm.last.changes

Set the number of commits from HEAD that you want to inspect

0

new, changed, affected

smart.testing.affected.inclusions

Set classes to be included for scanning

affected

smart.testing.affected.exclusions

Set classes to be excluded for scanning

affected

smart.testing.affected.config

Set location of affected configuration file

affected

smart.testing.affected.transitivity

Set transitivity enabled

true

affected

8.4. Getting insights of the execution

Property Description Default

smart.testing.debug

enables the debug mode (alternatively you can use Maven debug output (-X) flag)

false

8.5. Configuration File

mode: ordering
strategies:
  - new
  - changed
  - affected
applyTo: surefire
debug: true
disable: false
report:
    enable: true
scm:
    lastChanges: 1
autocorrect: true

== Developing

This section will be improved as we continue with the SPI design to let you hook your own logic if needed.

8.6. Test Bed

Test Bed brings a convenient way of writing functional tests. Here’s how the sample test looks like:

public class LocalChangesMixedStrategySelectionExecutionFunctionalTest {

    @ClassRule
    public static final GitClone GIT_CLONE = new GitClone(testRepository());

    @Rule
    public final TestBed testBed = new TestBed(GIT_CLONE);

    @Rule
    public final SmartTestingSoftAssertions softly = new SmartTestingSoftAssertions();

    @Test
    public void should_execute_all_new_tests_and_related_to_production_code_changes() throws Exception {
        // given
        final Project project = testBed.getProject();

        project.configureSmartTesting()
                .executionOrder(NEW, AFFECTED)
                .inMode(SELECTING)
            .enable();

        final Collection<TestResult> expectedTestResults = project
            .applyAsLocalChanges("Single method body modification - sysout",
                "Inlined variable in a method", "Adds new unit test");

        // when
        final TestResults actualTestResults = project.build("config/impl-base").run();

        // then
        softly.assertThat(actualTestResults.accumulatedPerTestClass())
            .containsAll(expectedTestResults)
            .hasSameSizeAs(expectedTestResults);

        softly.assertThat(project)
            .doesNotContainDirectory(SMART_TESTING_SCM_CHANGES)
            .doesNotContainDirectory(SMART_TESTING_WORKING_DIRECTORY_NAME);
    }
}

We are using arquillian-core project (in a form of snapshot and squashed repo) for running those tests. You can find it here.

You can also run particular tests or all tests against defined repo. For the first option simply GitClone rule constructor

public GitClone(String repositoryUrl)

or define system property test.bed.repo which will replace the default repository for all the tests which are relying on it.

8.6.1. Design Idea

The main goal of Test Bed module is to automate this process and simplify functional testing of our test optimization tool by providing a programmatic way for these activities.

Key Principles
Project configuration

Easy way of configuring any Maven-based project in the test itself is through simple and self-explanatory fluent API.

project.configureSmartTesting()
            .executionOrder(NEW)
            .inMode(SELECTING)
       .enable();

This snippet will enable smart-testing extension in the background. It will also apply test selection criteria and execution mode (selecting meaning only tests which belong to categories defined in executionOrder).

Config file creation

We have self-explainatory fluent api to create configuration with required options.

final Configuration configuration = new ConfigurationBuilder()
        .mode(SELECTING)
        .strategies(AFFECTED)
        .scm()
            .lastChanges("2")
            .build()
        .build();

Also we have fluent api to create configuration file at the root of your maven project using given configuration

project.configureSmartTesting()
            .withConfiguration(configuration)
        .createConfigFile()
    .enable();

OR using required options

project.configureSmartTesting()
            .executionOrder(FAILED)
            .inMode(SELECTING)
        .createConfigFile()
    .enable();
Applying changes

We decided to apply changes to the selected code base by modifying sample repository directly. This is done by creating atomic commits directly rather than applying code changes in the tests themselves (e.g. by using Forge Roaster). This comes with several advantages:

  • Ability to test those changes directly on the project

  • Easier refactoring

  • Ability to analyze execution when tests are failing by simply opening modified project rather than debugging test itself

Creating atomic change, which will be later used for tests, consists of following steps:

  • Making commit with the change

  • Tagging it with additional message

This message is then used as a changeDescription in the project API, for example:

project.applyAsLocalChanges("Single method body modification - sysout",
    "Inlined variable in a method");

This will look in the repository for a tag with message Single method body modification - sysout and apply as local change (no commits created). And the same for Inlined variable in a method.

Creating such a tag on the commit is as easy as:

$ git tag affected_01 -m "Single method body modification - sysout"

Defining test result expectations

In order to verify if our tool works, we have to specify which tests are expected to be executed as well as their statuses. This is done as part of the commit, as at this point we know exactly what we want our tool to execute and what should be the status of our tests based on modifications we made. For example in arquillian-core we would like to test affected strategy, where we modify ConfigurationRegistrar#loadConfiguration by adding System.out.println to the method. This should result in finding two related tests ConfigurationRegistrarTestCase and SyspropReplacementInArqXmlTestCase. Both of them should be executed successfully.

In order to define these expectations we create commit with the following message:

affected: adds system.out to ConfigurationRegistrar#loadConfiguration (1)

+ org.jboss.arquillian.config.impl.extension.ConfigurationRegistrarTestCase (2)
+ org.jboss.arquillian.config.impl.extension.SyspropReplacementInArqXmlTestCase
1 The first line describes the change, so it’s up to you to provide meaningful information.
2 Next two lines define actual expectations.

For expectations, it’s the list of the executed tests (fully qualified names) prefixed with an expected status. In this case +, which indicates all test methods in the particular class should be executed successfully. See org.arquillian.smart.testing.ftest.testbed.testresults.Status for all prefixes.

Executing test optimization

Once we configured the project and applied changes we can build it. This is what we want to really test after all.

Under the hood it’s using Embedded Maven. In addition, after the build is finished, it returns actual test results so we can verify expected and actual test execution. It’s done this way:

final TestResults actualTestResults = project.build().run();
Usage of Git

Before the test class starts "project under test" is cloned and for each test method separated copy is created to execute all the changes and builds in isolation. Folders are named using the following convention:

GIT_REPO_FOLDER + "" + getClass().getSimpleName() + "" + name.getMethodName()

for example:

smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_only_execute_tests_related_to_multiple_commits_in_business_logic_when_affected_is_enabled/

8.6.2. Troubleshooting tests

You can execute embedded build in Test Bed in the debug mode. This gives better analysis of the build execution, as it lets us connect to the build execution to analyze behaviour of our extension in the runtime by debugging both Maven execution as well as Surefire Provider.

Built-in DSL debugging features

You can use project configuration DSL for enabling debug mode directly in the test code:

final TestResults actualTestResults = project
    .build()
    .options()
        .withSystemProperties("scm.range.head", "HEAD", "scm.range.tail", "HEAD~")
        .withDebugOutput()
        .withRemoteDebugging()
        .withRemoteSurefireDebugging()
    .configure()
    .run();

Setting up any of the debugging modes would set an agent in suspend mode for both Maven and Surefire execution, thus giving you time to attach debuggers to both.

Alternatively, this can be also achieved through system properties which you can set as part of your "Run Configuration" in the IDE or build command. Following system properties are available:

test.bed.mvn.remote.debug

enables remote debugging of the embedded maven build with suspend set to y (so the build will wait until we attach remote debugger to the port) and port 8000 to listen).

test.bed.mvn.remote.debug.suspend

true or false (both for maven and surefire)

test.bed.mvn.remote.debug.port

valid port used for remote debugging of maven execution

test.bed.mvn.surefire.remote.debug

enables remote debugging for surefire

test.bed.mvn.surefire.remote.debug.port

valid port used for remoting debugging of surefire execution

test.bed.mvn.debug.output

true or false - sets -X flag for embedded maven build for more details build log.

Configuration precedence

If both system property and the programmatic option is used system property takes precedence.

Storing project under test

Project Persistence feature is included in Test Bed to store repository used test’s embedded build. This lets you execute the same build outside of test execution for further analysis in case of failure.

By default this feature is enabled to copy repository only in case of test failure.

In order to copy repository for all tests irrespective of test result, explicitly set system property test.bed.project.persist to true.

Persisted project are located in target/test-bed-executions/[current timestamp]/ with name followed by naming convention:

getClass().getSimpleName() + "_" + name.getMethodName()