End-To-End Integration Testing For Kubernetes and OpenShift

Does the title leave you wondering, why you need integration tests for your deployments ?

If so, let me give you a few examples. Consider you decide to upgrade Kubernetes, deploy a new Zapier
integration or add some authentication capability. In order to ensure your platform works every single
time, all of the above scenarios need to be tested with robust integration tests.

With the advent of microservices and cloud native applications, adding integration or
end-to-end tests to your Kubernetes or OpenShift projects might be challenging and cumbersome especially
when you want to set up the test infrastructure to be as close as possible to production.

Arquillian Community understands this pain, and continuously strive to provide users a seamless
experience for writing integration tests with ease by bringing their tests to the real environment and
deploy with confidence like never before.

With this blog post, we aim at showcasing integration tests for your Kubernetes or OpenShift clusters using just Arquillian Cube. Moving a step further, we also demonstrate building and deploying from scratch, applications with no deployment configuration, leveraging the power of Fabric8 Maven Plugin along with the Arquillian Cube Extension.

The key point here is that if OpenShift or Kubernetes is used as deployable platform in production, your tests are executed in a the same environment as it will be in production, so your tests are even more real than before.

Further, the test cases are meant to consume and test the provided services and assert that the environment is in the expected state.

Deployment Testing Recipes

If, you wish to read no more and see it all in action, head straight to our specially curated
examples to help you get started with ease.

Example 1

Deploying a sample PHP Guestbook application with Redis on Kubernetes from the resource descriptor
manifest file and testing it using Arquillian Cube Extension for Kubernetes and Kubernetes custom assertions.

Source: arquillian-testing-microservices/kubernetes-deployment-testing

Example 2

Deploying a Wordpress and My SQL application to OpenShift from a Template file and testing it using Arquillian
Cube Extension for OpenShift and Fabric8 OpenShift Client.

Source: arquillian-testing-microservices/openshift-deployment-testing

Example 3

Building and deploying a sample SpringBoot GuestBook application with zero deployment configuration using
Fabric8 Maven Plugin and Arquillian Cube Extension .

Fabric8 Maven Plugin aids in building Docker images and creating Kubernetes and OpenShift resource
descriptors for the application that allows for a quick ramp-up with some opinionated defaults and Arquillian
Cube Extension deploys the application from the generated resource descriptors and then executes deployment tests.

Source: arquillian-testing-microservices/zero-config-deployment-test

Step By Step Guide To Writing Deployment Tests

For a more in-depth step by step coverage of deployment testing, continue reading below.

Step 1: Setting Up Kubernetes/OpenShift Cluster

One of the pre-requisites for Arquillian Kube Extension, is to have Kubernetes or OpenShift cluster running on
your host machine.

An easy way to setup Kubernetes or OpenShift cluster locally is using Minikube and Minishift respectively.

Step 2: Adding Arquillian Kube Dependencies to your project

Arquillian Kube Extension provides a black box approach to testing your deployment that neither
mutates the containers (by deploying, reconfiguring etc) nor the Kubernetes/OpenShift resources.

It is used for immutable infrastructure and integration testing, wherein the test cases are meant to,
consume and test the provided services and assert that the environment is in the expected state,
providing you with the confidence that your application will work correctly when deployed on
Kubernetes/OpenShift cluster.

Before we can start writing our tests, we need to define a few dependencies as shown below:

Arquillian Cube BOM – Unified Dependencies

<properties>
    <version.arquillian_cube>${latest_released_version}</version.arquillian_cube>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.arquillian.cube</groupId>
            <artifactId>arquillian-cube-bom</artifactId>
            <version>${version.arquillian_cube}</version>
            <scope>import</scope>
            <type>pom</type>
        </dependency>
    </dependencies>
</dependencyManagement>

Arquillian Cube Requirement

Arquillian optionally provides an `ArquillianConditionalRunner` to control situations under which your tests should be executed by letting you specify the requirements using annotations.
For example, you can use annotations like `@RequiresOpenshift` to skip test when the environment is not prepared to run the tests.

To configure these requirements for your tests, enable the dependency to the following module for your project.

<dependency>
    <groupId>org.arquillian.cube</groupId>
    <artifactId>arquillian-cube-requirement</artifactId>
    <scope>test</scope>
</dependency>

Arquillian Cube Kubernetes (For Kubernetes Deployment)

<dependency>
    <groupId>org.arquillian.cube</groupId>
    <artifactId>arquillian-cube-kubernetes</artifactId>
    <scope>test</scope>
</dependency>

Arquillian Cube OpenShift (For OpenShift Deployment)

<dependency>
    <groupId>org.arquillian.cube</groupId>
    <artifactId>arquillian-cube-openshift</artifactId>
    <scope>test</scope>
</dependency>

Arquillian JUnit

<dependency>
    <groupId>org.jboss.arquillian.junit</groupId>
    <artifactId>arquillian-junit-standalone</artifactId>
    <version>${latest_released_version}</version>
    <scope>test</scope>
</dependency>

Fabric8 OpenShift Client

<dependency>
    <groupId>io.fabric8</groupId>
    <artifactId>openshift-client</artifactId>
    <version>${latest_released_version}</version>
</dependency>

For Fabric8 OpenShift Client, include the above dependency in the pom.xml

Step 3: Writing Deployment Tests

Arquillian Cube extension provides out of the box functionality to create and manage a temporary namespace
per test suite for your tests and then applies all the required kubernetes/openshift resources as defined
in the resource descriptors generated by fabric8 maven plugin for your environment.

Kubernetes/OpenShift resources can then be made accessible within the Test Cases by injecting them using
Arquillian’s @ArquillianResources annotation (see example test below).

@RunWith(Arquillian.class)    (1)
public class ExampleTest {

    @Named("dummy")           (2)
    @PortForward
    @ArquillianResource
    Service dummyService;

    @ArquillianResource       (3)
    OpenShiftClient client;

    @RouteURL("application")  (4)
    @AwaitRoute
    private URL route;

    @Test
    public void service_instance_should_not_be_null() throws Exception {
        assertThat(service).isNotNull();
    }

    @Test
    public void test_at_least_one_pod() throws Exception {
       assertThat(client).pods().runningStatus().filterNamespace(session.getNamespace()).hasSize(1); (5)
    }

    @Test
    public void verify_route_is_configured_and_service_is_accessible() throws IOException {
        assertThat(route).isNotNull();
        OkHttpClient okHttpClient = new OkHttpClient();
        Request request = new Request.Builder().get().url(route).build();
        Response response = okHttpClient.newCall(request).execute();

        assertThat(response).isNotNull();
        assertThat(response.code()).isEqualTo(200);
    }
}

Explained below are the steps for the above snippet of a sample deployment test.

  1. Configuring Arquillian Test Runner
    To setup our test environment, we need to tell junit that this test shall be executed as a arquillian junit
    test. This is done by @RunWith(Arquillian.class).
  2. Injecting Deployment Resources within Test Cases
    Kubernetes/OpenShift resources can then be made accessible within the Test Cases by injecting them
    using Arquillian’s @ArquillianResources annotation.
    The resource providers available, can be used to inject to your test cases the following resources:
    • A kubernetes client as an instance of KubernetesClient.
    • Session object that contains information (e.g. the namespace) or the uuid of the test session.
    • Services (by id or as a list of all services created during the session, optionally filtered by label)
    • Deployments (by id or as a list of all deployments created during the session, optionally filtered by label)
    • Pods (by id or as a list of all pods created during the session, optionally filtered by label)
    • Replication Controllers (by id or as a list of all replication controllers created during the session,
      optionally filtered by label)
    • Replica Sets (by id or as a list of all replica sets created during the session, optionally filtered by label)
  3. The OpenShift extension also provides:
    • An openshift client as an instance of OpenShiftClient.
    • Deployment Configs (by id or as a list of all deployment configs created during the session)
    • Resources can be injected into test cases by id or as a list of all deployments created during the
      session, optionally filtered by label.
  4. Injecting Container Resources to access the deployed service
    Test Enrichers like @RouteURL further aid in injection of container resources like route to the deployed service.
    For farbric8 maven plugin to identify the route, @RouteURL should be set to artifactId of the project by
    default, or explicity configured otherwise.
  5. Adding Test Cases that assert environment is in the expected state.

Step 4: Using Assertion Libraries

Optionally, using Fabric8 Kubernetes Assertions , a nice library based on assert4j, aids in performing
meaningful and expressive assertions on top of the Kubernetes/OpenShift model.

To enable Fabric8 Kubernetes Assertions in your test, include the following dependency in the pom.xml

<dependency>
    <groupId>io.fabric8</groupId>
    <artifactId>kubernetes-assertions</artifactId>
    <version>${latest_released_version}</version>
    <scope>test</scope>
</dependency>

Once everything is ready, Arquillian Kube runs your tests, enriched with resources required to access service
and finally cleaning up everything after the testing is over.

For more details and available configuration options check arquillian kube documentation.

Building Docker Images and creating Resource Descriptors.

Optionally, if you just have a Java application that you wish you to bring to Kubernetes or OpenShift,
use Fabric8 Maven Plugin to create docker images and resource descriptors for you.

Fabric8 Maven Plugin makes Kubernetes/OpenShift look and feel like an application server to a Java
developer by letting you build and deploy your application from maven just like you would with other
maven plugins.

To enable fabric8 on your existing maven project just type fabric8:setup command which adds the
fabric8-maven-plugin to your pom.xml.

Alternatively, you can manually add the following plugin definition to your pom.xml file:

<plugin>
    <groupId>io.fabric8</groupId>
    <artifactId>fabric8-maven-plugin</artifactId>
    <version>3.5.32</version>

    <!-- Connect fabric8:resource and fabric8:build to lifecycle phases -->
    <executions>
        <execution>
            <id>fmp</id>
            <phase>package</phase>
               <goals>
                    <goal>resource</goal>
                    <goal>build</goal>
                </goals>
        </execution>
    </executions>
</plugin>

For more and complete configuration details check out fabric8 maven plugin documentation.

Summary

With some basic features, simple steps and specially crafted examples as highlighted above, you can easily get started with Arquillian Cube extension to test your applications in real production environments, be it Kubernetes or OpenShift clusters, without worrying about setting up the infrastructure and only focusing on the core test logic.

Arquillian OpenShift Container 1.0.0.Beta1 Released

The Arquillian team is proud to announce the 1.0.0.Beta1 release of the Arquillian OpenShift Container component!

Some of the highlights in this release

Better integration for usage in continous integration servers. You can now specify the path to the SSH identity file and disable SSH Strict host checking if needed:

arquillian.xml
<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://jboss.org/schema/arquillian"
  xsi:schemaLocation="
      http://jboss.org/schema/arquillian
      http://jboss.org/schema/arquillian/arquillian_1_0.xsd">

  <container qualifier="openshift">
    <configuration>
      <!-- CAN BE SET VIA SSH_IDENTITYFILE environment variable as well -->
      <property name="identityFile">/absolute/path/to/identity/file</property>
      <property name="disableStrictHostChecking">true</true>
    </configuration>
  </container>

</arquillian>

Archive scanning now does not fail in case of linkage errors.

What is Arquillian?

Arquillian is open source software that empowers you to test JVM-based applications more effectively. Created to defend the software galaxy from bugs, Arquillian brings your test to the runtime so you can focus on testing your application's behavior rather than managing the runtime. Using Arquillian, you can develop a comprehensive suite of tests from the convenience of your IDE and run them in any IDE, build tool or continuous integration environment.

Release details

Component Arquillian OpenShift Container
Version 1.0.0.Beta1 view tag
Release date 2012-03-26
Released by Aslak Knutsen
Compiled against

Published artifacts org.jboss.arquillian.container

  • org.jboss.arquillian.container » arquillian-openshift-express jar javadoc pom

Release notes and resolved issues 3

Feature Request
  • ARQ-823 - Allow OpenShift Container to specify path to ssh keys
Bug
  • ARQ-822 - ByteAssetLoader does not ignore all LinkageError subtypes

Thanks to the following list of contributors: Karel Piwko, Aslak Knutsen

Arquillian OpenShift Container 1.0.0.Alpha2 Released

Since we wrote this post we didn't laze around. Check our latest announcement.

The Arquillian team is proud to announce the 1.0.0.Alpha2 release of the Arquillian OpenShift Container component!

Some of the highlights in this release

Application deployment timeout can be set up by user:

arquillian.xml
<arquillian xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="http://jboss.org/schema/arquillian"
  xsi:schemaLocation="
      http://jboss.org/schema/arquillian
      http://jboss.org/schema/arquillian/arquillian_1_0.xsd">

  <container qualifier="openshift">
    <configuration>
      <property name="deploymentTimeoutInSeconds">150</property>
    </configuration>
  </container>

</arquillian>

Improved archive scanning now honors application.xml and jboss-web.xml for figuring out
deployment context root.

Support for deployment of ZipImport’ed archives.

Arquillian OpenShift Express Container no longer requires JBoss Maven Repository.

What is Arquillian?

Arquillian is open source software that empowers you to test JVM-based applications more effectively. Created to defend the software galaxy from bugs, Arquillian brings your test to the runtime so you can focus on testing your application's behavior rather than managing the runtime. Using Arquillian, you can develop a comprehensive suite of tests from the convenience of your IDE and run them in any IDE, build tool or continuous integration environment.

Release details

Component Arquillian OpenShift Container
Version 1.0.0.Alpha2 view tag
Release date 2012-02-27
Released by Aslak Knutsen
Compiled against

Published artifacts org.jboss.arquillian.container

  • org.jboss.arquillian.container » arquillian-openshift-express jar javadoc pom

Release notes and resolved issues 7

Component Upgrade
  • ARQ-787 - Update Arquillian Core and Ajocado dependencies dependencies
  • ARQ-788 - Update JGit and EJB3 API dependencies
Feature Request
  • ARQ-786 - Enable deploymentTimeout for OpenShift containers
Bug
  • ARQ-621 - openshift container - unable to deploy zip assets
  • ARQ-781 - openshift container - improve deployment url detection
  • ARQ-789 - Deployment scanning for JAR archives is broken

Thanks to the following list of contributors: Karel Piwko, Aslak Knutsen, Jan Papoušek, Jozef Hartinger

Arquillian OpenShift Container 1.0.0.Alpha1 Released

Since we wrote this post we didn't laze around. Check our latest announcement.

The Arquillian team is proud to announce the 1.0.0.Alpha1 release of the Arquillian OpenShift Container component!

What is Arquillian?

Arquillian is open source software that empowers you to test JVM-based applications more effectively. Created to defend the software galaxy from bugs, Arquillian brings your test to the runtime so you can focus on testing your application's behavior rather than managing the runtime. Using Arquillian, you can develop a comprehensive suite of tests from the convenience of your IDE and run them in any IDE, build tool or continuous integration environment.

Release details

Component Arquillian OpenShift Container
Version 1.0.0.Alpha1 view tag
Release date 2011-08-14
Released by Aslak Knutsen
Compiled against

Published artifacts org.jboss.arquillian.container

  • org.jboss.arquillian.container » arquillian-openshift-express jar javadoc pom

Release notes and resolved issues 2

Containers OpenShift - Express support

Task
  • ARQ-515 - Implement OpenShift Express container for Arquillian

Thanks to the following list of contributors: Karel Piwko, Aslak Knutsen