Smart Testing 0.0.1 Released
Since we wrote this post we didn't laze around. Check our latest announcement.
The Arquillian team is proud to announce the 0.0.1 release of the Smart Testing component!
First release of Smart Testing is here!
It all started as a Google Summer of Code project last year with great ground work of our student Dimcho Karpachev.
Flash forward to this summer, after 222 commits, 69 pull requests merged and insane amount of CI builds we are super excited to announce a new addition to Arquillian Universe we’ve been working so vigorously on.
This time it’s not a new extension to Arquillian Testing Platform, but a brand new tool which we hope will bring a breath of fresh air into your builds, let it be on your local machine or the CI server.
You probably know it best – you write some code and tests, run the build, wait few minutes to only see at the later stage that your changes are breaking it. Of course you can go and grab a coffee, but if you commit early and often that might mean caffeine overdose in a very short time. Running tests before pushing is a good practice, but if it takes too long it simply slows you down.
Moreover having long lasting builds on your CI server makes your CI/CD process not really that continuous and leads to Pull Requests piling up. Bringing new features to production with confidence should be seamless. And for this we need to have faster feedback loops.
Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health.
That’s why we created this cure for you!
Highlights of this release
With this release we provide following strategies which you can use to optimize your build execution:
new– gives higher priority to newly added testschanged– gives higher priority to changed tests (e.g. new test methods or refactorings of components under tests which imply changes in the tests)affected– based on changes in your business logic, gives higher priority to the tests which are exercising themfailed– gives higher priority to the tests which failed in the previous (local) build
In addition you can either decide to run the whole test suite (ordering mode) or only those tests which fall into the selected categories (selecting, which is also a default mode).
new, changed and affected rely on SCM information. For this release we only support Git.
How to get started
If you are using Maven 3.3.x or newer (and you definitely should!) adding Smart Testing to your build is very easy. Create a file .mvn/extensions.xml in the root of project
with the following content:
<?xml version="1.0" encoding="UTF-8"?>
<extensions>
<extension>
<groupId>org.arquillian.smart.testing</groupId>
<artifactId>maven-lifecycle-extension</artifactId>
<version>0.0.1</version>
</extension>
</extensions>and you are all set!
Executing the build with Smart Testing enabled is as simple as adding one system property with strategies of your choice:
$ mvn clean verify -Dsmart.testing=new,changed,affectedThis will execute the build running only those tests which are falling into selected categories, based on your local changes.
For the CI environment, such as Jenkins, you can pass commit hashes using environment variables. For example:
$ mvn verify -Dsmart.testing=affected -Dscm.range.head=GIT_COMMIT -Dscm.range.tail=GIT_PREVIOUS_COMMIT
This will optimize tests based on the changes between the current commit and the one against which the previous build was run.
For more details head over to our documentation backed by awesome Asciidoctor!
We are very excited about Smart Testing but we need you to help us make it even more awesome. We spent tremendous amount of effort building it with the developer experience in mind, but without the community feedback we can only get that far. So give it a try today and tell us what you think! Maven is a first stop for us. Stay tuned! There’s more to come!
Thanks to the whole team for making it happen! You are real heroes!
Also big kudos to Infinitest team for the inspiration! Give it a spin if you are Eclipse or IDEA user.
What is Smart Testing?
Smart Testing is a tool that speeds up the test running phase by reordering test execution plan to increase a probability of fail-fast execution and thus give you faster feedback about your project’s health.
Release details
| Component | Smart Testing |
|---|---|
| Version | 0.0.1 view tag |
| Release date | 2017-09-14 |
| Released by | Bartosz Majsak |
| Compiled against |
Published artifacts org.arquillian.smart.testing
- org.arquillian.smart.testing » core jar javadoc pom
- org.arquillian.smart.testing » surefire-provider jar javadoc pom
- org.arquillian.smart.testing » junit-test-result-parser jar javadoc pom
- org.arquillian.smart.testing » strategy-affected jar javadoc pom
- org.arquillian.smart.testing » strategy-changed jar javadoc pom
- org.arquillian.smart.testing » strategy-failed jar javadoc pom
- org.arquillian.smart.testing » maven-lifecycle-extension jar javadoc pom
- org.arquillian.smart.testing » git-rules jar javadoc pom
- org.arquillian.smart.testing » smart-testing-test-bed jar javadoc pom
Release notes and resolved issues 30
Initial release with Maven integration.
- Component: Core
-
- #143 All working files (surefire reports, changes) should be placed inside one directory
- #124 Surefire fork failures with our provider
- #109 Removes GlobPatterns logic
- #98 Transitivity in Affected strategy
- #81 ArrayIndexOutOfBounds is thrown when using Surefire 2.20
- #77 Ordering process should take into account previous configuration/strategies
- #75 Affected strategy refactoring
- #52 Git properties
- Component: Test Bed
-
- #138 Exposes all test results from Test Bed build
- #137 Extract git rules to its own module
- #104 Unable to run the same (failing) test twice
- #61 Improve build configuration DSL
- #56 Persisting project under test folder for analysis in case of test failure
- #55 Implement functional tests for failed strategy
- #54 Make smart-testing version and git test bed repo configurable in functional tests
- #53 Ability to configure test project to run failsafe
- Component: Maven
-
- #112 Affected tests are not properly resolved for different surefire fork modes
- #91 Debug information
- #83 NullPointerException is thrown when having custom surefire configuration
- #73 Disable extension when test execution is skipped
- #70 Filter selected tests based on testsToRun first instead of filtering it in selectTests method of TestStrategyApplier
- #69 java.lang.NoSuchMethodError is thrown in case of not defining surefire
- #64 Set configuration defaults
- #62 TestNG Tests in the dogfooding repo giving error when trying to run with specified mode.
- #57 Introduce Maven extension to determine tests to run once before the build starts
- #47 Seamlessly store test results from the build execution
- #44 Build time overhead when running with surefire smart test provider
- Component: Documentation
- Component: Selection
Thanks to the following list of contributors: Bartosz Majsak, Alex Soto, Matous Jobanek, Dipak Pawar, Hemani
