We know which tests you want to run.
We all know that as the project evolves we keep adding more and more tests to ship our products confidently. This however has an impact on the build time and thus we waste time waiting for the most important question to be answered - "Did I break anything with my changes?".
Let it be your local development or a CI server - what if you could know this as soon as possible?
We created Smart Testing to give you the fastest possible feedback loop when it comes to executing your tests.
1. What is this?
Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health.
Suppose that you need to implement a new feature in your current project. If you are following TDD / BDD approach, you’ll write some test(s) to validate that the new feature behaves as expected. Then you’ll implement the new feature and when you get that all tests passes. Last step is to push this code to your source control, which probably will trigger a new build in your CI server running all the tests.
And at this point two things can happen:
-
The build takes several minutes and it does not fail.
-
The build takes several minutes but it fails because your new tests are failing (yes I know it worked in your machine).
Usually you don’t have that much influence on when your new tests are going to be executed. They could be the first ones, in that case your build will fail fast, or they might be the last ones, so you’ll need to wait some time until you get the failure.
Smart testing project aims to solve this uncertainty by ordering tests so that the first tests that could fail (because for example they are new or because failed previously) are going to be executed first.
1.1. Integrations
Currently Smart Testing supports Maven through Maven Extension mechanism and works together with maven-surefire-plugin
and maven-failsafe-plugin.
1.2. Minimum requirements
- Java
-
Requires Java 8.
| Currently we don’t support Java 9 for Affected strategy. |
- Maven
-
Tested with Maven
3.3.9and3.5.0but any version above3.1.Xshould work. Our recommendation is that you use at least3.3.Xversion to avoid any extension API problems. - Surefire/Failsafe
-
Minimum version required is
2.19.1or above. It is important to make sure that you have it defined in yourpom.xml, as Maven comes with the default version which might not be compatible with our extension.
2. Installation
2.1. Don’t make me think
Ok, just execute following snippet and you are all set:
curl -sSL https://git.io/v5jy6 | bash
You can also add these flags:
-
-l|--latestlatest published artifact in Maven Central (that’s the default) -
-v|--version=- specific version to install
For example:
curl -sSL https://git.io/v5jy6 | bash -s -- --latest
This script will automatically do what we described below. If you are curious keep on reading, but you may skip it too.
This script requires xmllint, and xsltproc, so make sure you have it installed.
|
2.2. Maven Extension
Smart Testing is a Maven extension, so depending on the version of Maven you have to follow slightly different approach of installing it.
2.2.1. Maven above 3.1.X
Get Smart Testing Extension shaded jar from Maven Central and copy it to M2_HOME/lib/ext.
2.2.2. Maven above 3.3.X
You can still use the process described at [Maven >= 3.1.X] or use the new core extension configuration mechanism by
creating folder called .mvn in the root of your project and create inside it an extensions.xml file
which registers the smart testing extension:
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
<extension>
<groupId>org.arquillian.smart.testing</groupId>
<artifactId>maven-lifecycle-extension</artifactId>
<version>${version}</version>
</extension>
</extensions>
3. Configuration
This section explains configuration parameters of Smart Testing. Notice that they are all summarized at Reference Card section.
Also Smart Testing can be configured with a configuration file instead of System properties. You can read about configuration file at Configuration File.
3.1. Modes
So far in What is this? section we explained that Smart testing changes the order of test execution plan
to run first the important tests and then the rest. This is know as ordering mode, but that’s not the only one.
ordering-
orderingmode as its name suggests orders the test execution plan so important tests are executed first and then the rest. selecting-
selectingmode just selects the important tests and execute them, skipping the rest of the tests.
The selecting is the default mode.
In order to define the mode use smart.testing.mode Java system property with either one.
|
To get fast feedback loop, you can use surefire’s skip after
However this functionality cannot be fully guaranteed (real first failure) in concurrent mode due to race conditions.
Read |
3.2. Strategies
Until now, you’ve read that smart testing is changing test execution plan running or only including important tests. But how do we know which tests are important and which ones not?
There are several strategies that you can choose from which determine what are the important tests.
Currently we have following strategies in place: new, changed, affected, failed and categorized.
To set them you need to set Java system property smart.testing to one or more strategies in comma-separated value form.
Smart Testing is able to auto correct misspelled strategies in case of setting Java System property smart.testing.autocorrect to true.
Subsequent sections dive deeper into each and every one of them.
3.2.1. New
New strategy uses SCM information (currently only Git is supported) to detect new tests and treat them as important
tests to run them first (ordering) or filtered (selecting).
So internally this strategy inspects your SCM history (and local changes) and all tests that were added (so they are effectively new) are marked as important ones.
This strategy can be configured in several ways through Java system properties:
-
scm.range.headandscm.range.tailare used to set the range of commits you want to inspect for new tests. The values can be commit ids or usingHEADnotation. For example:-Dscm.range.head=HEAD -Dscm.range.tail=HEAD~By default if not specified, default value isHEAD. -
scm.last.changescan be used to set the number of commits fromHEADthat you want to inspect. For example-Dscm.last.changes=3will be equivalent as-Dscm.range.head=HEAD -Dscm.range.tail=HEAD~.
Those properties might be in particular handy when used in the CI builds.
For example in Jenkins when using Git Plugin you can configure your build as follows:
$ mvn clean verify -Dsmart.testing=new, affected -Dscm.range.head=GIT_COMMIT -Dscm.range.tail=GIT_PREVIOUS_COMMIT
|
Currently not committed changes (those that are added and untracked) are considered as important tests as well. This effectively means that if you don’t specify any commit range, only these information is considered.
3.2.2. Changed
Changed strategy is like New strategy, but it uses only tests that are modified (they were already committed in the past)
instead of new ones.
In this strategy not committed changes (those that are modified or changed) are considered as important tests as well.
Usually new and changed strategies are used together -Dsmart.testing=new, changed.
|
3.2.3. Affected
Affected strategy uses a different approach to choose what are the important tests to run first (ordering) or filtered (selecting).
This strategy also relies on SCM information but in this case it retrieves any new or modified business class
between commits range and local changes as well.
scm.range.head, scm.range.tail and scm.last.changes Java system properties are valid as well.
|
When this strategy gets all changes then inspect all tests of current project checking which ones imports these classes. If the test exercises a business class that has been modified, we treat it as important so it will be executed earlier in the test plan.
| This strategy is currently only applicable for white box testing approach. At this point our approach is to analyze direct code dependencies, but we are working on broader use cases. |
| At this moment, this strategy does not work with Java 9. |
Explicitly Set
By default affected strategy uses imports of tests to build the graph of collaborators. This approach is fine for unit tests (white box tests) but might not work in all cases of high level tests (black box test).
There are some test technologies that allows you to deploy an application locally and then run tests against it.
For example Wildfly Swarm has @DefaultDeployment annotation or Spring (Boot) deploys all application automatically.
This means that production classes are not directly imported into the test, so there is no way to get a relationship between test and production classes.
For this reason affected provides an annotation to set package(s) of production classes that are deployed by the test.
The first thing you need to do is register following artifact into your build script: org.arquillian.smart.testing:api:<version>.
Then you can annotate your test with org.arquillian.smart.testing.strategies.affected.ComponentUnderTest annotation.
For example:
@ComponentUnderTest(packages = "org.arquillian.smart.testing.strategies.affected.fakeproject.main.superbiz.*")
public class ZTest {In previous example all classes belonging to packages and subpackages specified at packages attribute are considered as classes used by the test.
You can also use packageOf attribute to set a reference class.
With this attribute all classes that are placed in the same package as the reference class are considered as classes used by the test.
With this approach your tests are resilient to package name changes.
If none of the attributes are set, then all production classes with same package as test and its subpackages are added automatically as classes used by the test.
Regular Files
By default affected strategy uses imports of tests to build the graph of collaborators or if you use the Explicitly Set, you can explicitly set class dependencies from an annotation.
This approach is fine for Java classes, but a project usually contains another kind of files such as configuration files like persistence.xml, deployment files like kubernetes.json or build tool like pom.xml.
You can react to these file changes as well and execute some specific tests in these cases by using a special annotation provided by affected strategy.
The first thing you need to do is register following artifact into your build script: org.arquillian.smart.testing:api:<version>.
Then you can annotate your test with org.arquillian.smart.testing.strategies.affected.WatchFile annotation.
For example:
@WatchFile("src/main/resources/META-INF/persistence.xml")
public class XTest {In previous example when src/main/resources/META-INF/persistence.xml file is added or modified, then the test is eligible to be run.
Location is relative to current project/module. This means that if you want to refer to a file that it is in another module, you need to navigate to parent folder by using .. symbol. For example: ../myothermodule/src/main/resources/applicationcontext.xml.
|
3.2.4. Failed
Failed strategy just gets all tests that failed from previous executions and mark them as important tests to run first (ordering) or not filtered (selecting). When the selecting mode is used then, by default, the resolution is on the test methods level (selects the particular failed test methods) - you can disable it by setting the parameter -Dsmart.testing.failed.methods=false
This strategy uses the JUnit XML report for reading past executions.
All reports from previous local build are automatically copied by the maven extension to a temp directory ${project.directory}/.smart-testing/temporary/reports and when the build is finished the directory is removed.
3.2.5. Categorized
Categorized strategy selects tests based in the JUnit 4 annotation @Category or JUnit 5 annotations @Tag and @Tags (including meta annotations).
With this strategy, you can easily set which classes or methods you want to run (or don’t want to run)
as you would in the configuration of the maven-surefire-plugin without any need of changing your pom.xml file with complicated profiles.
In case of JUnit 4, you can say which categories should be executed by saying either the fully qualified names of the classes or just simple class names (case sensitivity can be set via configuration). For more information about the parameters see Configuration Options.
4. Configuration File
You can use smart-testing YAML file at ${project.dir}/smart-testing.yml or at ${project.dir}/smart-testing.yaml.
You can also point to a custom configuration file from a different project by setting the smart.testing.config property.
Configuration file looks as follows:
inherit: ../smart-testing.yml (1)
mode: ordering (2)
strategies: new, changed, affected (3)
applyTo: surefire (4)
debug: true (5)
disable: false (6)
report:
enable: true (7)
scm:
range:
head: HEAD (8)
tail: HEAD~2 (9)
autocorrect: true (10)
customStrategies: (11)
- smart.testing.strategy.cool=org.arquillian.smart.testing:strategy-cool:1.0.0
- smart.testing.strategy.experimental=org.arquillian.smart.testing:strategy-experimental:1.0.0
strategiesConfiguration: (12)
affected:
transitivity: true (13)
exclusions: (14)
- org.package.*
- org.arquillian.package.*
inclusions: (15)
- org.package.exclude.*
- org.arquillian.package.exclude.*
customProviders: (16)
- org.foo:my-custom-provider=fully.qualified.name.to.SurefireProviderImpl
| 1 | Config file’s absolute or relative path from where Smart Testing overwrites parameters not defined in current config file. |
| 2 | This defines mode to be used by Smart Testing. |
| 3 | This defines strategies to be used while finding important tests. |
| 4 | This defines plugin to be used for Smart Testing. |
| 5 | This enables debug logs for Smart Testing. |
| 6 | This disables Smart Testing if set to true. |
| 7 | This enables Smart Testing report if set to true. |
| 8 | This sets first commit sha or HEAD notation for inspecting changes. |
| 9 | This sets last commit sha or HEAD notation for inspecting changes. |
| 10 | This defines if smart testing should auto correct misspelled strategies. |
| 11 | This defines the pair key/value of custom strategies as list. |
| 12 | This defines list of strategy related configurations that you want to apply while applying strategy. |
| 13 | This enables transitivity. |
| 14 | This defines list of packages to be excluded while applying transitivity. |
| 15 | This defines list of packages to be included while applying transitivity. |
| 16 | GroupId and artifactId with fully qualified name of the implementation of SurefireProvider the execution should be delegated to. |
All parameters in configuration file are optional. If you haven’t used any parameter in configuration file, Smart testing will use default value for that parameter. You can look at references for default value of parameter.
However you can overwrite all configuration options using system properties supported by Smart Testing. You can look references for all supported system properties.
4.1. How configuration is applying to each module?
Smart Testing is looking in each module’s root dir for configuration file. If it didn’t find config file there, it’ll look recursively for the first config file in it’s parent dir.
4.2. How Configuration parameters are inherited?
If config file has inherit defined with absolute or relative path of config file, then Smart Testing will
lookup for all undefined properties in config file defined using inherit. However it won’t lookup for disable parameter.
If config file doesn’t contain inherit parameter. Then Smart Testing won’t inherit any configuration properties.
4.3. Configuration File per Maven Module:
Let’s take a look into following example:
e.g.
parent
- config
- smart-testing.yml (1)
- api
- impl-base
- spi
- smart-testing.yml (2)
- container
- api
- impl-base
- smart-testing.yml (3)Configuration files used in above example:
inherit: ../smart-testing.yml
strategies: affectedstrategies: affected,newstrategies: new
scm:
lastChanges: 1Configuration file selection for the above example will be as follows:
-
config/api - parent/config/smart-testing.yml
-
config/impl-base - parent/config/smart-testing.yml
-
config/spi - parent/config/spi/smart-testing.yml
-
container/api - parent/smart-testing.yml
-
container/impl-base - parent/smart-testing.yml
| 1 | inside this file we have inherit: ../smart-testing.yml, so it will take all undefined properties in this file from the one in parent/smart-testing.yml. |
| 2 | inside this file we have only strategies: affected,new, so it won’t inherit from any other configuration file. |
| 3 | inside this file we have strategies: new & scm: lastChanges: 1 defined.(You can set it using respective system property also). |
In case of mvn extension, whenever you are using configuration file per maven module, make sure to define strategies in project’s parent configuration file
or using system property. If you haven’t defined strategy in parent config, Smart testing will load parent configuration having no strategy defined which will
disable ST by saying Smart Testing is disabled. Reason: strategy not defined.
If you are running tests against range of commits locally, you need to also define scm properties in project’s parent config file or using system
properties. If you haven’t defined it in parent config, Smart Testing will use default scm configuration with head as HEAD & tail as HEAD~0
due to which Smart Testing won’t find required changes in range of commits.
|
4.4. Configuration File Reference
The smart-testing.yml file is a YAML file defining required configuration to configure Smart Testing.
You can use either a .yml or .yaml extension for this file.
|
| Field | Description |
|---|---|
inherit |
This is used to define absolute or relative path to parent configuration file |
strategies |
This is used to define required strategies to find important tests. Look at strategies for all supported options. |
mode |
This is used to select mode for Smart Testing. Look at modes for all supported options & default value. |
applyTo |
This applies smart testing to use with |
debug |
This option runs smart testing in debug mode. |
disable |
This disables smart testing extension without removing it. |
report |
This configures report options for smart testing. Look at Report Options for all available options. |
scm |
To run Smart Testing with |
autocorrect |
This configures Smart Testing to auto correct misspelled strategies to the closest one. For example in case of user set strategies to |
customStrategies |
A list of custom strategies in the form of key/value. It is important to notice that the key part must be prefixed with |
strategiesConfiguration |
A list of strategies with it’s configuration. |
customProviders |
A pair of groupId and artifactId with fully qualified name of |
4.4.1. Report Options
| Field | Description |
|---|---|
enable |
This generates smart testing report with selected tests. |
4.4.2. Scm Options
| Field | Description |
|---|---|
range |
This configures range for Scm configuration. Look at Range Options for all available options. |
lastChanges |
This is used to set the number of commits from |
4.4.3. Range Options
| Field | Description |
|---|---|
head |
Sets first commit sha or |
tail |
Sets last sha or |
4.4.4. Affected Configuration Options
| Field | Description |
|---|---|
transitivity |
Sets transitivity to find all affected tests. |
exclusions |
Sets list of packages to be excluded while applying transitivity. |
inclusions |
Sets list of packages to be included while applying transitivity. |
4.4.5. Categorized Configuration Options
| Field | Description |
|---|---|
categories |
List of categories the strategy should be looking for (can be either whole fully qualified names or only simple class names). If nothing is set, but the strategy is used, then all classes are selected. |
excludedCategories |
List of categories the strategy should exclude from looking for (can be either whole fully qualified names or only simple class names) |
caseSensitive |
Sets case sensitivity of the category names. (default is false) |
methods |
Sets if the resolution of categories/tags should be performed only on test classes or also on test methods. (default is true) |
5. Usage
5.1. Examples
After installing Smart Testing Maven Extension you can use any goal you use with maven-surefire-plugin
or maven-failsafe-plugin.
To configure it you only need to pass from CLI the Java system properties. Let’s see some examples:
Remember that the default mode is selecting.
You want to run only tests that you’ve just added or modified locally
mvn clean test -Dsmart.testing="new, changed"
You want to run all tests but given priority to the latest tests added or modified
mvn clean test -Dsmart.testing.mode=ordering -Dscm.last.changes=1 -Dsmart.testing="new, changed"
You want to run only tests that validates new or modified business classes locally
mvn clean test -Dsmart.testing="affected"
When you are running Smart Testing, you’ll see following logs in the console, showing your current configuration:
[INFO] [Smart Testing Extension] Applied strategies: [new, affected] [INFO] [Smart Testing Extension] Applied usage: [selecting]
This can be used as feedback to check that the extension has been installed correctly.
5.2. Disabling Smart Testing
Sometimes you might want to disable smart testing extension without removing it.
To do it you just need to set smart.testing.disable Java system property to true and then tests will
run with standard surefire / failsafe plugins.
5.3. Plugin Selection
Smart testing extension hooks into surefire and failsafe lifecycle to provide the order/selection of tests to execute.
If you only want to use smart testing on surefire or failsafe plugin definition.
To do it you just need to set smart.testing.apply.to with either surefire or failsafe to just enable in one of them.
5.4. Debug Mode
In order to get diagnostic information related to Smart Testing extension, you can enable the debug mode by either
setting the system property smart.testing.debug or by setting the mvn debug output (-X) flag to have the entire build execution
in the debug mode (available out of the box in Maven).
Debug logs provide information related to smart testing execution, system properties set by the user,
test selection information. In addition for each and every module we store modified pom.xml at target/smart-testing/reporting
as smart-testing-effective-pom.xml
All Smart Testing related logs can be found with the prefix Smart Testing Extension -
|
5.5. Specifying Concrete Set of Test Classes
During development, you may want to run a single test class or a set of concrete test classes.
You can do this by setting the maven test property to specific test class(es).
5.5.1. To Run a Single Test
mvn test -Dtest=SampleTest
| This configuration disables smart testing and executes only the specified sample test. |
5.5.2. To Run Multiple Test Classes
You can also choose to run only a concrete set of multiple test classes.
mvn test -Dtest=SampleOneTest, SampleTwoTest
| When multiple specific tests are set without any pattern, smart testing is disabled just like the case for single test. |
You can further choose to execute multiple test classes in combination with smart testing’s selecting and ordering mode by using patterns
supported by Maven.
mvn test -Dtest=Sample*Test
When used with ordering mode, the specified tests are executed in the order defined by the smart testing
strategy used. If none is applicable to the strategy, they are executed in order defined by Surefire.
When used with selecting mode, the specified tests that fit the defined strategy are executed.
However, if none fulfil the strategy criteria, no tests would be executed.
5.6. Skipping Tests
To skip executing the tests for a particular project, Smart Testing aligns with
Maven Surefire/Failsafe
plugins' skip test functionality and respects system properties skipTests, skipITs and maven.test.skip.
6. Registering Custom Strategies
By default Smart Testing comes with a set of strategies such as new, changed, affected, failed or categorized, but you can implement your own strategies and registering them.
To do it you just need to follow next four steps:
-
Implement
org.arquillian.smart.testing.spi.TestExecutionPlannerFactoryJava SPI and registering as such. -
Implement
org.arquillian.smart.testing.spi.TestExecutionPlannerinterface. -
Register strategy on Smart Testing configuration file (optional).
6.1. Implementing TestExecutionPlannerFactory
TestExecutionPlannerFactory is a Java SPI interface that manages the creation of the strategy.
You need to implement three simple methods, two related with the name of the strategy and one that generates the strategy class.
As an example from failed strategy:
import java.io.File;
import org.arquillian.smart.testing.api.TestVerifier;
import org.arquillian.smart.testing.configuration.Configuration;
import org.arquillian.smart.testing.spi.StrategyConfiguration;
import org.arquillian.smart.testing.spi.TestExecutionPlanner;
import org.arquillian.smart.testing.spi.TestExecutionPlannerFactory;
import static org.arquillian.smart.testing.strategies.failed.FailedTestsDetector.FAILED;
public class FailedTestsDetectorFactory implements TestExecutionPlannerFactory {
@Override
public String alias() {
return FAILED;
} (1)
@Override
public boolean isFor(String name) {
return alias().equalsIgnoreCase(name);
}
@Override
public TestExecutionPlanner create(File projectDir, TestVerifier verifier, Configuration configuration) {
return new FailedTestsDetector(projectDir, configuration);
} (2)
@Override
public StrategyConfiguration strategyConfiguration() {
return new FailedConfiguration();
}
}| 1 | Returns the alias of the strategy |
| 2 | Returns an instance of TestExecutionPlanner |
Of course as any other Java SPI, you need to register it in META-INF/services.
org.arquillian.smart.testing.strategies.failed.FailedTestsDetectorFactory6.2. Implementing TestExecutionPlanner
TestExecutionPlanner is the implementation of the strategy.
Apart from the method getName() that returns a name of the strategy there are two important methods:
Collection<TestSelection> selectTestsFromNames(Iterable<String> testsToRun);
Collection<TestSelection> selectTestsFromClasses(Iterable<Class<?>> testsToRun);These methods return a collection of the tests that must be executed.
There is only one call of one of these two methods - it won’t happen that both methods are called at once. The reason for the existence of these two methods is that in some environments there aren’t available Classes for the tests, but just the names.
For example, failed strategy is implemented as:
import java.io.File;
import java.util.Collection;
import org.arquillian.smart.testing.TestSelection;
import org.arquillian.smart.testing.configuration.Configuration;
import org.arquillian.smart.testing.spi.JavaSPILoader;
import org.arquillian.smart.testing.spi.TestExecutionPlanner;
public class FailedTestsDetector implements TestExecutionPlanner {
static final String FAILED = "failed";
private final File projectDir;
private final FailedConfiguration strategyConfig;
public FailedTestsDetector(File projectDir, Configuration configuration) {
this.projectDir = projectDir;
strategyConfig = (FailedConfiguration) configuration.getStrategyConfiguration(FAILED);
}
@Override
public Collection<TestSelection> selectTestsFromNames(Iterable<String> testsToRun) { (1)
return getTests();
}
@Override
public Collection<TestSelection> selectTestsFromClasses(Iterable<Class<?>> testsToRun) { (2)
return getTests();
}
public Collection<TestSelection> getTests() {
InProjectTestReportLoader reportLoader = new InProjectTestReportLoader(new JavaSPILoader(), projectDir);
return TestResultsFilter.getFailedTests(strategyConfig, reportLoader.loadTestResults());
}
@Override
public String getName() {
return FAILED;
}
}| 1 | Method that returns list of tests to execute based on the provided set of fully qualified class names of tests that were previously selected to be executed. |
| 2 | Method that returns list of tests to execute based on the provided set of tests classes that were previously selected to be executed. |
6.2.1. ChangeStorage, ChangeResolver and TestResultParser
Smart Testing provides some services to get some common operations. These services can be used to get information from current project.
ChangeStorage
Change Storage service is used to get the changes (from SCM perspective) that has occurred in current project. This service does not read changes from SCM directly but from a file that has precalculated all changes at the beginning of the Smart Testing process.
Notice that in this way your strategy is more performant since, code does not need to calculate changes for each strategy.
To get it you just need to call new JavaSPILoader().onlyOne(ChangeStorage.class).get().
ChangeResolver
Change Resolver service is like ChangeStorage but in this case the changes are calculated directly from SCM.
To get it you just need to call new JavaSPILoader().onlyOne(ChangeResolver.class).get().
This service is usually not used directly in strategies since we recommend using ChangeStorage service for this purposes and just fall back to this service in case of error.
For example:
final Collection<Change> files = changeStorage.read(projectDir) (1)
.orElseGet(() -> {
logger.warn("No cached changes detected... using direct resolution");
return changeResolver.diff(projectDir, configuration, getName()); (2)
});| 1 | Changes are read from precalculated file |
| 2 | Fallback to calculate directly from SCM |
6.3. Registering strategy
Finally you need to register strategy in Smart Testing configuration file where you set the strategy name as key and the Maven coordinates as value. This file is required so Smart Testing can automatically register the artifacts in build tool script.
As an example of registering previous strategy:
customStrategies:
- smart.testing.strategy.failed=org.arquillian.smart.testing:strategy-failed:1.0.0It is important to notice that the key part must be prefixed with smart.testing.strategy..
If you need to specify any additional coordinate (such as type or classifier), you have to use the whole coordinates format:
groupId:artifactId:packaging:classifier:version:scope
6.4. Alternative Approach
You can still use another approach to register the pair strategy.name=Maven:coordinates without configuring them in Smart Testing configuration file and it is using a System property.
For example previous example would be equivalent as running Smart Testing with next System property:
-Dsmart.testing.strategy.failed=org.arquillian.smart.testing:strategy-failed:1.0.0
Notice again that it must be prefixed with smart.testing.strategy..
7. Surefire Providers
Internally, Smart Testing uses its own implementation of SurefireProvider where calls Smart Testing API. Then, based on the information retrieved from the classpath, it decides which provider should be used for a delegation of the test execution - eg. JUnit4 or TestNG (in the same way as it is done inside of the Surefire plugin itself).
But some projects may use their own surefire provider implementations or can specify one specific known provider that should scan the classpath and execute the test suite.
7.1. Known Surefire Providers
Known Surefire Providers are those implementations that are part of the Surefire project itself plus one implementation distributed as part of JUnit 5. The complete list of known providers:
-
JUnit 4
-
groupId:
org.apache.maven.surefire -
artifactId:
surefire-junit4 -
provider implementation:
org.apache.maven.surefire.junit4.JUnit4Provider
-
-
JUnit 47 and above
-
groupId:
org.apache.maven.surefire -
artifactId:
surefire-junit47 -
provider implementation:
org.apache.maven.surefire.junitcore.JUnitCoreProvider
-
-
JUnit 5
-
groupId:
org.junit.platform -
artifactId:
junit-platform-surefire-provider -
provider implementation:
org.junit.platform.surefire.provider.JUnitPlatformProvider
-
-
TestNG
-
groupId:
org.apache.maven.surefire -
artifactId:
surefire-testng -
provider implementation:
org.apache.maven.surefire.testng.TestNGProvider
-
When one of these providers is specified inside of the Surefire’s dependencies, then Smart Testing is able to automatically detect it and delegate the execution to it.
7.2. Custom Surefire Providers
If some unknown provider (or some other SurefireProvider implementation) is used then it is necessary to set this provider either in the configuration file or using system properties.
The format is a pair of groupId and artifactId with a fully qualified name of the class that implements SurefireProvider interface and that should be loaded. For JUnit 5 provider it would look like this (in case of configuration file):
customProviders:
- org.junit.platform:junit-platform-surefire-provider=org.junit.platform.surefire.provider.JUnitPlatformProviderIf this is specified then Smart Testing will detect the dependency and delegate the test execution to it.
System property dedicated for this usage is smart.testing.custom.providers
8. Reports
8.1. Build Artifacts
To identify which tests were selected to run looking in the logs (or even using some CLI tricks with grep) is not the
most efficient way to get a quick answer to:
-
How tests are ordered?
-
To which strategy they belong?
-
What was the configuration used?
Smart Testing provides concise report in the XML format with all this information at hand.
Here’s how the sample report looks like:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<smart-testing-execution>
<module>core</module>
<executionConfiguration>
<usageMode>selecting</usageMode>
<strategies>
<strategy>new</strategy>
<strategy>changed</strategy>
</strategies>
<properties>
<property name="smart.testing" value="new,changed"/>
<property name="smart.testing.mode" value="selecting"/>
</properties>
</executionConfiguration>
<selection>
<tests>
<test name="org.arquillian.smart.testing.report.ExecutionReporterTest" strategies="new,changed"/>
<test name="org.arquillian.smart.testing.report.ExecutionReporterUsingPropertyTest" strategies="new"/>
</tests>
</selection>
</smart-testing-execution>8.2. Configuration
By default this feature is disabled. You can enable it by setting property smart.testing.report.enable to true.
If you use the property, then you get the generated reports in target/smart-testing/reporting for every module separately.
If you are running Smart Testing in debug mode using property smart.testing.debug or Maven build output in debug mode using -X or --debug then Smart Testing will generate report.
9. Jenkins
So far you’ve seen how to use Smart Testing from developer perspective (running on local machine). But ultimately your software is going to be built on CI/CD server and saving time there means more resources for other projects.
One of important things to take into consideration is that meanwhile on the developer’s machine selecting mode might be the one used most frequently, in CI/CD environment you should consider executing the build in the ordering mode at some point (let it be regular build or a step in the pipeline). You can read more about modes at Modes section.
In case of development machine, you’ll probably want to build simply against the local changes, but in case of CI/CD environment, probably the changes you want to take into consideration are those between the commits you are going to run the build.
Let’s see how to configure Smart Testing in Jenkins.
9.1. Jenkins Freestyle project
Freestyle project is the most basic way of creating builds with Jenkins.
To use Smart Testing in this kind of project, you only need to create a build step of kind Execute Shell/Execute Windows batch command running Maven with required Smart Testing configuration parameters.
mvn -Dsmart.testing="new, affected"
-Dsmart.testing.mode=ordering
-Dscm.range.head=${GIT_COMMIT}
-Dscm.range.tail=${GIT_PREVIOUS_COMMIT}
testOn the next figure you can see the step configuration.
After setting this up you are ready to run the build.
What you will notice in this case is that since the configured mode is ordering, all tests are going to be executed, but the ones marked as important by new and affected strategies are executed first.
9.2. Jenkins Pipeline
Jenkins Pipeline is a group of plugins which support implementing and integrating continuous delivery pipelines into Jenkins.
The definition of a Jenkins Pipeline is typically written into a text file (called a Jenkinsfile) which in turn is checked into a project’s source control repository
To run Smart Testing in Jenkins Pipeline you need to manually call checkout scm process to get access to GIT_COMMIT and PREVIOUS_GIT_COMMIT variables.
This might not be a problem if you are using scripted pipeline but in case of using declarative pipeline, since the checkout process is done automatically. you have no access to GIT_COMMIT and PREVIOUS_GIT_COMMIT variables.
So in next snippet you can see an example of how to use declarative pipeline with Smart Testing:
pipeline {
options {
skipDefaultCheckout()
}
agent any
stages {
stage('Compile and Test') {
steps {
script {
def scmVars = checkout scm
sh "mvn -Dsmart.testing='new, affected' -Dsmart.testing.mode=ordering -Dscm.range.head=${scmVars.GIT_COMMIT} -Dscm.range.tail=${scmVars.GIT_PREVIOUS_COMMIT} test"
}
}
post {
success {
junit 'target/surefire-reports/**/*.xml'
}
}
}
}
}There are few important things you have to keep in mind when adjusting your Jenkinsfile if you are using declarative script approach.
First of all you need to disable automatic checkout. Then you need to manually call checkout and store the result into a variable. Finally you can call Maven, getting Git parameters from scmVars.
9.3. Smart Testing Jenkins Pipeline Shared Library
Instead of running shell script using mvn call, you can use Smart Testing Jenkins Pipeline Shared Library which simplifies the configuration of Smart Testing in Jenkins.
9.3.1. Installing
To install this shared library you basically have two ways:
-
Installing as global shared libraries in
Manage Jenkins>>Configure System>>Global Pipeline Libraries -
Setting repository directly inside
Jenkinsfile. For example@Library('github.com/organization/project@maven') _
You can read more about how to install shared libraries at https://jenkins.io/doc/book/pipeline/shared-libraries/
Using it
To use it, you just need to create the Jenkinsfile and add an stage similar to:
@Library('smart-testing') _ (1)
pipeline {
options { skipDefaultCheckout() } (2)
agent any
stages {
stage('Compile and Test') {
steps {
script {
def scmVars = checkout scm (3)
smartTesting(strategies: 'new, changed', scmRange: scmVars) (4)
}
}
}
}
}| 1 | Refers a global shared library registered inside Jenkins with smart-testing name. |
| 2 | In declarative Jenkins pipeline you need to skip default checkout to get scm info. |
| 3 | Executed the checkout process and store info into scmVars variable. |
| 4 | Calls Maven with Smart Testing parameters. |
Running previous script results as next shell call mvn '-Dsmart.testing=new, changed' -Dsmart.testing.mode=ordering -Dscm.range.head=653317b065ee8c54f9e806bc801b00d4c6371a20 -Dscm.range.tail=653317b065ee8c54f9e806bc801b00d4c6371a20 clean test
9.3.2. Configuration
Next configuration parameters are valid to be set in the smartTesting method.
| Parameter | Default Value | Description |
|---|---|---|
|
|
Sets Maven Home. If not set, then it runs |
|
|
Sets goal of Maven execution. |
|
Sets Maven profile. |
|
|
Sets Smart Testing strategies. |
|
|
last commit |
Map resulting of calling |
|
String that is concatenated before goals. Used to set any custom parameters such as Java system properties. |
10. Reference Card
10.1. Installation
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
<extension>
<groupId>org.arquillian.smart.testing</groupId>
<artifactId>maven-lifecycle-extension</artifactId>
<version>${version}</version>
</extension>
</extensions>
10.2. Configuration
| Property | Description | Default | Possible Values |
|---|---|---|---|
|
Set running mode |
|
|
|
Set strategies in CSV |
- |
|
|
Disable Smart Testing |
|
|
|
Set plugin to apply Smart Testing in CSV |
|
|
|
Enable auto correct of misspelled strategies |
|
|
|
Set custom configuration file |
- |
custom configuration |
10.3. Strategies
| Property | Description | Default | Applicable Strategies |
|---|---|---|---|
|
Set first commit id for inspecting changes |
|
|
|
Set last commit id for inspecting changes |
|
|
|
Set the number of commits from |
|
|
|
Set classes to be included for scanning |
|
|
|
Set classes to be excluded for scanning |
|
|
|
Set transitivity enabled |
true |
|
|
Register custom strategies implementations |
Value is the Maven coordinates of the custom strategy |
|
|
List of categories the strategy should be looking for (can be either whole fully qualified names or only simple class names) |
- |
|
|
List of categories the strategy should exclude from looking for (can be either whole fully qualified names or only simple class names) |
- |
|
|
Sets case sensitivity of the category names. |
|
|
|
Sets if the resolution of categories/tags should be performed only on test classes or also on test methods. |
|
|
|
Sets if the strategy should select the particular failed test methods or take the whole classes |
|
|
11. Developing
| This section will be improved as we continue with the SPI design to let you hook your own logic if needed. |
11.1. Test Bed
Test Bed brings a convenient way of writing functional tests. Here’s how the sample test looks like:
public class LocalChangesMixedStrategySelectionExecutionFunctionalTest {
@ClassRule
public static final GitClone GIT_CLONE = new GitClone(testRepository());
@Rule
public final TestBed testBed = new TestBed(GIT_CLONE);
@Rule
public final SmartTestingSoftAssertions softly = new SmartTestingSoftAssertions();
@Test
public void should_execute_all_new_tests_and_related_to_production_code_changes() throws Exception {
// given
final Project project = testBed.getProject();
project.configureSmartTesting()
.executionOrder(NEW, AFFECTED)
.inMode(SELECTING)
.enable();
final Collection<TestResult> expectedTestResults = project
.applyAsLocalChanges("Single method body modification - sysout",
"Inlined variable in a method", "Adds new unit test");
// when
final TestResults actualTestResults = project.build("config/impl-base").run();
// then
softly.assertThat(actualTestResults.accumulatedPerTestClass())
.containsAll(expectedTestResults)
.hasSameSizeAs(expectedTestResults);
softly.assertThat(project)
.doesNotContainDirectory(TemporaryInternalFiles.getScmChangesFileName())
.doesNotContainDirectory(SMART_TESTING_WORKING_DIRECTORY_NAME);
}
}We are using arquillian-core project (in a form of snapshot and squashed repo) for running those tests. You can find it here.
You can also run particular tests or all tests against defined repo. For the first option simply GitClone rule constructor
public GitClone(String repositoryUrl)or define system property test.bed.repo which will replace the default repository for all the tests which are relying on it.
11.1.1. Design Idea
The main goal of Test Bed module is to automate this process and simplify functional testing of our test optimization tool by providing a programmatic way for these activities.
Key Principles
Project configuration
Easy way of configuring any Maven-based project in the test itself is through simple and self-explanatory fluent API.
project.configureSmartTesting()
.executionOrder(NEW)
.inMode(SELECTING)
.enable();This snippet will enable smart-testing extension in the background. It will also apply test selection criteria
and execution mode (selecting meaning only tests which belong to categories defined in executionOrder).
Config file creation
We have self-explainatory fluent api to create configuration with required options.
final Configuration configuration = new ConfigurationBuilder()
.mode(SELECTING)
.strategies(AFFECTED)
.scm()
.lastChanges("2")
.build()
.build();Also we have fluent api to create configuration file at the root of your maven project using given configuration
project.configureSmartTesting()
.withConfiguration(configuration)
.createConfigFile()
.enable();OR using required options
project.configureSmartTesting()
.executionOrder(FAILED)
.inMode(SELECTING)
.createConfigFile()
.enable();Applying changes
We decided to apply changes to the selected code base by modifying sample repository directly. This is done by creating atomic commits directly rather than applying code changes in the tests themselves (e.g. by using Forge Roaster). This comes with several advantages:
-
Ability to test those changes directly on the project
-
Easier refactoring
-
Ability to analyze execution when tests are failing by simply opening modified project rather than debugging test itself
Creating atomic change, which will be later used for tests, consists of following steps:
-
Making commit with the change
-
Tagging it with additional message
This message is then used as a changeDescription in the project API, for example:
project.applyAsLocalChanges("Single method body modification - sysout",
"Inlined variable in a method");This will look in the repository for a tag with message Single method body modification - sysout and apply as local change (no commits created).
And the same for Inlined variable in a method.
Creating such a tag on the commit is as easy as:
$ git tag affected_01 -m "Single method body modification - sysout"
Defining test result expectations
In order to verify if our tool works, we have to specify which tests are expected to be executed as well as their statuses.
This is done as part of the commit, as at this point we know exactly what we want our tool to execute and what should be the status of our tests based on modifications we made.
For example in arquillian-core we would like to test affected strategy, where we modify ConfigurationRegistrar#loadConfiguration by adding System.out.println to the method.
This should result in finding two related tests ConfigurationRegistrarTestCase and SyspropReplacementInArqXmlTestCase.
Both of them should be executed successfully.
In order to define these expectations we create commit with the following message:
affected: adds system.out to ConfigurationRegistrar#loadConfiguration (1) + org.jboss.arquillian.config.impl.extension.ConfigurationRegistrarTestCase (2) + org.jboss.arquillian.config.impl.extension.SyspropReplacementInArqXmlTestCase
| 1 | The first line describes the change, so it’s up to you to provide meaningful information. |
| 2 | Next two lines define actual expectations. |
For expectations, it’s the list of the executed tests (fully qualified names) prefixed with an expected status.
In this case +, which indicates all test methods in the particular class should be executed successfully.
See org.arquillian.smart.testing.ftest.testbed.testresults.Status for all prefixes.
Executing test optimization
Once we configured the project and applied changes we can build it. This is what we want to really test after all.
Under the hood it’s using Embedded Maven. In addition, after the build is finished, it returns actual test results so we can verify expected and actual test execution. It’s done this way:
final TestResults actualTestResults = project.build().run();
Usage of Git
Before the test class starts "project under test" is cloned and for each test method separated copy is created to execute all the changes and builds in isolation. Folders are named using the following convention:
GIT_REPO_FOLDER + "" + getClass().getSimpleName() + "" + name.getMethodName()
for example:
smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_only_execute_tests_related_to_multiple_commits_in_business_logic_when_affected_is_enabled/
11.1.2. Troubleshooting tests
You can execute embedded build in Test Bed in the debug mode. This gives better analysis of the build execution, as it lets us connect to the build execution to analyze behaviour of our extension in the runtime by debugging both Maven execution as well as Surefire Provider.
Built-in DSL debugging features
You can use project configuration DSL for enabling debug mode directly in the test code:
final TestResults actualTestResults = project
.build()
.options()
.withSystemProperties("scm.range.head", "HEAD", "scm.range.tail", "HEAD~")
.withDebugOutput()
.withRemoteDebugging()
.withRemoteSurefireDebugging()
.configure()
.run();Setting up any of the debugging modes would set an agent in suspend mode for both Maven and Surefire execution, thus giving you time to attach debuggers to both.
Alternatively, this can be also achieved through system properties which you can set as part of your "Run Configuration" in the IDE or build command. Following system properties are available:
test.bed.mvn.remote.debug-
enables remote debugging of the embedded maven build with
suspendset toy(so the build will wait until we attach remote debugger to the port) and port8000to listen). test.bed.mvn.remote.debug.suspend-
trueorfalse(both for maven and surefire) test.bed.mvn.remote.debug.port-
valid port used for remote debugging of maven execution
test.bed.mvn.surefire.remote.debug-
enables remote debugging for surefire
test.bed.mvn.surefire.remote.debug.port-
valid port used for remoting debugging of surefire execution
test.bed.mvn.debug.output-
trueorfalse- sets-Xflag for embedded maven build for more details build log.
|
Configuration precedence
If both system property and the programmatic option is used system property takes precedence. |
Storing project under test
Project Persistence feature is included in Test Bed to store repository used test’s embedded build. This lets you execute the same build outside of test execution for further analysis in case of failure.
By default this feature is enabled to copy repository only in case of test failure.
In order to copy repository for all tests irrespective of test result, explicitly set system property test.bed.project.persist to true.
Persisted project are located in target/test-bed-executions/[current timestamp]/ with name followed by naming convention:
getClass().getSimpleName() + "_" + name.getMethodName()