Jenkins

Jenkins
Travis

Travis

1. Prerequisites

You need to have following packages in place:

Assuming that you have all the Golang prerequisites in place (such as $GOPATH), clone the repository first:

$ git clone https://github.com/arquillian/ike-prow-plugins $GOPATH/src/github.com/arquillian/ike-prow-plugins
Have a look how Go Version Manager can help you simplifying configuration and management of different versions of Go.
You can also use setup.sh which takes care of all of the above. Simply execute
$ curl -sSL https://git.io/vxnSk | bash
and you are all set.

2. Development

2.1. Build

In order to compile the project simply execute make build target. This will compile, run tests and put binaries of each plugin in /bin directory in the root of the project. There is also a quick compile option (make compile-only) where go build is only invoked, skipping other targets like tests and checks

You can also automatically format and remove unused imports by invoking goimports tool (make imports).

goimports does not correctly handle import of packages where package name differs from last part of the URL provided. For example import "gopkg.in/h2non/gock.v1" will be removed. To overcome this problem name the import import gock "gopkg.in/h2non/gock.v1". One of the related issues go/issues/9882.

To deploy plugins use make oc-deploy-plugins. This will build images, push them to the repository, generate deployments and apply them on the cluster. This target builds all plugins at once.

You have to be logged in to the cluster first.

2.2. Developing new plugin

Besides creating new folders/packages in plugin folder you will have to register your plugin in the Makefile. Simply add its name to this list and you should be good.

By convention internal PluginName, the directory where the code is located and name of the service are assumed to be the same.

2.3. Testing

We use Ginkgo testing framework and Gomega assertion library for writing tests.

Make sure you have both installed locally in order to execute tests from the CLI:

go get -u github.com/onsi/ginkgo/ginkgo
go get -u github.com/onsi/gomega

2.3.1. Testing hints

  • To run all tests at once execute ginkgo -r on the root of the project.

  • ginkgo watch -r will watch on tests changes recursively starting from the directory the command was invoked. It will re-run the tests whenever change in the code occurs

  • In order to execute one test rename It function to FIt. Ginkgo runner will only execute given test (focused) instead of the whole test suite. This also works with Context and Describe blocks.

  • To exclude prefix any of the blocks with X.

2.3.2. Testing hooks

You can consume GitHub events through webhook pointing to your local cluster. For this purpose use ultrahook. After registering your own API key simply run following command and you are all set:

ultrahook github http://$(oc get route/hook --no-headers=true -o=custom-columns=HOST:spec.host)/hook
If not done before add ultrahook URL as payload URL for webhook configuration (see Setting up the web hook).

Having this set up you will start seeing events triggered by your actions in the repository and corresponding plugins reacting on them. Have a look at pods logs to verify if everything is working expected.

2.4. Local Setup using Minishift

You can use Minishift for developing plugins locally. Follow official installation guide to set it up.

This setup is based on following versions:

minishift v1.13.1
openshift v3.7.1
kubernetes v1.7.6

To start your Minishift cluster you can use following command (you might want to customize CPU and memory limits):

minishift start --profile prow --memory 8GB --cpus 4

2.4.1. Initial setup

Before you start developing new plugins or evolving existing ones you need to deploy set of basic Prow services and configuration.

For this purpose use login to your OpenShift cluster using oc login and execute make oc-init-project target which will:

  • Create new project named ike-prow-plugins

  • Add config maps for Prow configuration (config.yaml) and its plugins (plugins.yaml)

  • Add required secrets (see GitHub settings for details)

At this point you have new Openshift project created and before you will be able to deploy plugins you need to run the target oc-deploy-hook that will deploy hook service responsible for listening to GitHub events and dispatching them to relevant plugins.

2.4.2. Pushing to local Minishift repository

To speed up development cycle (and avoid pushing work-in-progress) images to Docker Hub you can use built-in docker repository available in Minishift. Makefile and templates are adjusted to use it, so the only thing you have to do is to set up few environment variables.

Before executing docker login make sure you are logged in using token instead of regular password. That’s returned by oc whoami -t and is used as password by Docker daemon. To do that open OpenShift Console (e.g. xdg-open https://$(minishift ip):8443/console &>/dev/null), click on your username in the upper-right corner and then Copy Login Command. Use it to login from the terminal and you are ready to go.
Setting up Docker to use local Minishift registry
eval $(minishift docker-env)
docker login -u $(oc whoami) -p $(oc whoami -t) $(minishift openshift registry)
export REGISTRY=$(minishift openshift registry)
export DOCKER_REPO=$(oc project -q)

2.4.3. Hints

You can simplify your setup by seeding configuration files

Seeding tokens
echo "${GH_WEBHOOK_SECRET}" > config/hmac.token (1)
echo "${GH_TOKEN}" > config/oauth.token (2)
echo "${SENTRY_DSN}" > config/sentry.dsn (3)
1 Your GitHub Webhook secret. See for more details GitHub settings.
2 Your user (or bot account) GitHub OAuth Token. See for more details GitHub settings.
3 Sentry DSN in https://<key>:<secret>@<sentry_host>/<project> format.
If you are using zsh you can take a look at autoenv or direnv to automate this setup every time you access project’s directory. Just make sure to add it to your global .gitignore to avoid sharing it through PRs (e.g. git config --global core.excludesfile ~/.gitignore).
Some handy aliases
alias oc-console='xdg-open https://$(minishift ip):8443/console &>/dev/null'
alias uh="ultrahook github http://$(oc get route/hook --no-headers=true -o=custom-columns=HOST:spec.host)/hook"
alias gdoc="./.asciidoctor/generate.sh && xdg-open gh-pages/index.html &>/dev/null"

2.5. Sentry Logging

Plugins in this repository can be integrated with Sentry error tracking tool. Use following flags when starting a plugin service to enable it:

  • sentry-dsn-file - path to the file containing complete Sentry DSN. Defaults to /etc/sentry-dsn/sentry.

  • sentry-timeout - Sentry server timeout in ms. Defaults to 1 second.

  • env - environment name plugin is running in. Used e.g. by Sentry for tagging.

You will have to add config/sentry.dsn file in this repository so built-in Makefile targets will be able to populate DSN correctly to plugin pods.

2.6. Test suite API

Since the plugins require a lot of communication with GitHub we use Gock for mocking the necessary requests in our tests. However, some cases require more complex communication that has to be mock and this can lead to very cluttered lengthy code with several code duplication. To avoid this, we have a complex fluent API that creates the mocks instead of you.

The API has two main entry points:

  • MockPr that creates an instance of the MockPrBuilder which is then responsible for collecting all information that is necessary for the mock creation. By calling the next method that follows this one you specify the way the PR information should be loaded from - there are possible these options:

    • LoadedFrom(jsonPath string)

    • LoadedFromDefaultJSON()

    • LoadedFromDefaultStruct()

  • NewMockPluginTemplate that takes a plugins name and then provides the same method MockPr as mentioned above. This is for test cases when it is necessary to know which plugin is used - eg. for mocking status calls

2.6.1. MockPrBuilder

MockPrBuilder is the main building block of the whole API - it offers a set of methods - eg:

  • WithTitle

  • WithDescription

  • WithFiles

  • WithoutFiles

  • WithComments

  • WithoutComments

  • WithReviews

  • WithLabels

  • WithUsers

  • …​etc…​

These methods set the required information to the builder and when the method Create() is called, then based on the provided data it sets the necessary Gock mocks.

2.6.2. Expectation

For checking any of the POST, PATCH and DELETE requests, the MockPrBuilder offers method Expecting where you can define the required matcher for the sent payload. The methods responsible for this are:

  • Comment

  • ChangedComment

  • Status

  • RemovedLabel

  • AddedLabel

  • ChangedTitle

2.6.3. PrMock

The method Create returns an instance of PrMock that offers additional methods such as:

  • CreateCommentEvent

  • CreatePullRequestEvent

  • PermissionForUser

The values returned from these methods can be used for triggering/calling the plugin’s logic.

The overall usage can look like:

// given
prMock := mocker.MockPr().LoadedFromDefaultJSON().
  WithFiles(LoadedFrom("test_fixtures/github_calls/prs/without_tests/changes.json")).
  WithoutConfigFiles().
  WithoutMessageFiles("test-keeper_without_tests_message.md").
  WithoutComments().
  Expecting(
    Status(ToBe(gh.StatusFailure, keeper.NoTestsMessage, keeper.NoTestsDetailsPageName)),
    Comment(ContainingStatusMessage(keeper.WithoutTestsMsg))).
  Create()

// when
err := handler.HandlePullRequestEvent(log, prMock.CreatePullRequestEvent("opened"))

2.6.4. Config mock API

There is also a possibility to not use the whole MockPrBuilder but only part of it (with the usage of SubMockBuilder). An example of such a usage is mock for config files:

mocker.AddConfig(
  ConfigYaml(Containing(
    Param("test_patterns", "['*my', 'test.go', 'pattern.js']"),
    Param("skip_validation_for", "['pom.xml', 'regex{{*\\.adoc}}']")))).
  ToChange(change)

3. Plugins

3.1. Configuration

In order to enable certain plugins for your repository or organization, you should edit plugins.yaml file adding:

external_plugins:
  bartoszmajsak/wfswarm-booster-pipeline-test:
  - name: test-keeper (1)
    events: (2)
      - pull_request
      - issue_comment
1 The test-keeper plugin will be applied to specified repository. You can also specify plugins only for the whole organization. Be aware however, that adding an external plugin both to the repository and its organization will result in an error.
2 You can limit events dispatched by hook to your plugin.

3.1.1. GitHub settings

You will need two secrets to be able to integrate with GitHub. The config/hmac.token file should contain the token that you give to GitHub for validating webhooks. You can generate it using any reasonable randomness-generator, for example random.org.

The config/oauth.token is an OAuth2 token that has read and write access to the GitHub account. Generate it from here. Depending on your plugin needs the scopes might vary, but it is important to have following scopes enabled: admin:org_hook and admin:repo_hook.

Both of these files are ignored by git (see .gitignore) so you can keep them in your repository, as some make targets rely on them.

3.1.2. Setting up the web hook

In order to setup webhook for your repository go to https://github.com/{org}/{repo}/settings/hooks/new and provide:

  • payload URL

  • content type needs to be application-json

  • secret the same as defined in config/hmac.token file

  • customize types of hooks you would like to receive (or use all to start)

For hints how to test web hook against your local setup head over to Testing hooks section.

More details about GitHub hooks can be found in the official developer documentation.

3.2. Test Keeper plugin

This plugin can help you stick to the rule that every feature you ship comes with automated way of assuring it works - by using automated tests. If it won’t find any tests in the Pull Request, it adds a comment with a description and marks its check as Failure, or as Success otherwise (see the screenshots below):

Success
Figure 1. Success status
Failure
Figure 2. Failure status

This check is done based on the file name patterns - for more information head over to Plugin Configuration part.

The plugin is triggered when the Pull Request is opened/reopened or updated by new or removed commit.

If, for whatever reason, you want to bypass this check - simply comment using /ok-without-tests command. If you are an admin user or requested PR reviewer but not a creator of the PR you will see the Success status. If the comment will be later removed the check is triggered again.

3.2.1. How does it work?

Test Keeper looks into the files in your Pull Request and checks if any tests were added or modified based on common naming patterns (we don’t analyze source code yet…​).

In addition, if a PR contains only "non-production" changeset, it won’t be checked for test presence. By "non-production" changes we mean things like:

  • documentation

  • build assets (dependency files, build definition)

  • shell scripts

  • visual assets (images, styles, HTML templates)

  • common configuration files

Of course, all of it is configurable.

We have few reasonable defaults, which you can check here.

If we missed some important patterns feel free to open an issue or better yet - a Pull request!

3.2.2. Plugin Configuration

To configure Test Keeper plugin place test-keeper.yml (or test-keeper.yaml) file inside of the directory .ike-prow/ in your project and use properties described below.

test-keeper.yaml
test_patterns:
  - '**/__test.go'                         (1)
skip_validation_for:
  - 'regex{{.*\.[xsd]?html$}}'             (2)
combine_defaults: false                     (3)
1 Defines set of file patterns which will be used to match files changeset and determine if Pull Requests comes with any changed test files.
2 These file patterns will be used against changeset in the Pull Requests to exclude files that don’t have to be verified by any test. If only such files exists the check will be marked as "Success" as no tests are expected for such a PR.
3 Allows you to decide if you want to combine your patterns with predefined defaults (true by default).
The configuration file is always loaded from the HEAD of the Pull Request.
File patterns

Both inclusions and exclusions can be specified in two formats - either in a wildcard format or in a regex.

Wildcard format

Wildcard format represents the whole file path using a combination of a star wildcard:

**

wildcard for expressing "any directory"

*

wildcard for expressing "any file name"

Examples

**/*_test.go

matches any subdirectory containing a go file whose name ends with _test

*_test.go

is same as the previous one - just a shorter version

pkg/**/*_test.go

matches all pkg subdirectories containing a Go file whose name ends with _test

vendor/

matches directory vendor and all its subdirectories and files located in it

Regular expressions
Don’t use a regular expression inside of the wildcard format. We don’t support it.

If the wildcard format is not suitable for you, you can still use regex, but it has to be surrounded by regex{{…​}}.

Both formats can used together in list of patterns, e.g.:

test_patterns:
  - '**/__test.go'
  - 'regex{{.*test\.ts[x]?}}'

3.2.3. Status message

When there is a PR submitted without any test logic changed, then plugin (apart form setting the failure status) adds a comment explaining what is wrong and what the developer/reviewer can do. If the PR is then either

  • updated by a commit that contains a test

or

  • updated so it contains only those files which the validation should be skipped for

then the status message in the comment is updated respectively to reflect the latest state of the PR.

Custom status message

Any of the status messages can be changed by putting the required custom message to any of the following files:

  • test-keeper_without_tests_message.md for the case when no test is added

  • test-keeper_with_tests_message.md for the case when PR is updated by a commit containing a test

  • test-keeper_only_skipped_message.md for the case when PR is updated so it contains only those files which the validation should be skipped for

All of them has to be located in the directory .ike-prow/

3.2.4. Status details

In this section, you can find status details description applicable for each state of the test-keeper plugin.

Success - test is present

Your Pull Request has been approved because the plugin detected a test file that has been added or changed in the PR.

It means that changeset in this PR has files matching any of the default test patterns, those you configured or by a combination of both.

For more information about the behavior and what the default file patterns are, see How does it work? section. If you need to reconfigure the plugin then read the section Plugin Configuration.

Success - no test needed

Your Pull Request has been approved because it seems that it doesn’t need to have additional tests. All changed files in the changeset match patterns for which the validation should be skipped.

For more information about the behavior and what the default file patterns are, see How does it work? section. If you need to reconfigure the plugin then read the section Plugin Configuration.

Success - approved by

Your Pull Request has been approved by any of the administrators or reviewers despite the fact that it has no test.

If the PR contains only "non-production" changeset and some of them haven’t been detected by any file patterns the validation should be skipped for, you can add them in your configuration file.

For more information see How does it work? and Plugin Configuration sections.

Failure

Your Pull Request has been rejected because the plugin wasn’t able to find any added or changed test file in the change-set.

Automated tests give us confidence in shipping reliable software. Please add some as part of this change.

If you are an admin and you are sure that no test is needed then you can use a command /ok-without-tests as a comment to make the status green.

For more information about the behavior, how the test files are detected and what the default file patterns are, see How does it work? section. If you need to reconfigure the plugin then read the section Plugin Configuration.

3.3. Work In Progress Plugin

This plugin can help reviewers with focusing on work that really matters so they don’t need to spend time with Pull Requests that are still in progress unless they were requested to do partial reviews.

The plugin checks the PR for the presence of "work in progress" prefix in title and the GitHub label to manage the status of the PR in progress and or manage the addition/deletion of the label or the prefix respectively as described in the following cases:

Case 1: The PR is has a "work in progress" label but the "work in progress" prefix is missing in the title.

  • The PR status is marked as Failure and the title is left as it is.

Case 2: The PR title contains the "work in progress" prefix but no "work in progress" label is included.

  • The PR status is marked as Failure and the "work in progress" label is added automatically.

Case 3: The PR title is updated and "work in progress" prefix is removed.

  • The PR status is changed to Success and the "work in progress" label is removed automatically.

Case 4: The PR is updated and the "work in progress" label is removed.

  • The PR status is changed to Success and the "work in progress" prefix is removed automatically.

where the PR title’s "work in progress" prefix can have any of the default prefixes (WIP, DO NOT MERGE, DON’T MERGE, WORK-IN-PROGRESS with any combinations of the flavors such as ( ), [ ], { }, ! !,* *, etc…​) or any of the user configured prefixes as described below (non-case sensitive) and the "work in progress" GitHub label can have the default value (work-in-progress) or any other user configured value as described below as valid values.

3.3.1. Plugin Configuration

To configure Work In Progress plugin place work-in-progress.yml (or work-in-progress.yaml) file inside of the directory .ike-prow/ in your project and use properties described below.

work-in-progress.yaml
title_prefixes:          (1)
  - 'work in progress'
combine_defaults: true   (2)
gh_label: wip            (3)
1 Defines set of custom-prefixes which will be used to match titles prefixes in changeset and determine if Pull Requests are still in progress.
2 Allows you to decide if you want to combine your patterns with the list (WIP, DO NOT MERGE, DON’T MERGE, WORK-IN-PROGRESS) of predefined defaults (true by default).
3 Defines the custom name to be used for the GitHub label for the "work in progress pull request" (work-in-progress by default).

3.3.2. Status details

In this section, you can find status details description applicable for each state of the work-in-progress plugin.

Success

Your Pull Request has been approved as the plugin didn’t detect any sign of work in progress (the PR title doesn’t start with any of the "work in progress prefixes" or contain any "work-in-progress" label (both of them are configurable)).

Failed

Your Pull Request has been rejected because the plugin detected that either the PR title starts with one of the "work in progress prefixes" or has the "work-in-progress" label (both of them are configurable), so it seems that there is still an ongoing work.

When the PR is done and ready for review and merge, then remove the prefix from the title or the "work-in-progress" label to make the status green. Removing the prefix also triggers removing the added GitHub "work-in-progress" label from the PR and vice-versa.

3.4. PR Sanitizer Plugin

This plugin enforces writing effective pull requests that confirm with proper naming conventions and description to clearly convey its purpose and provide better collaboration, traceability and maintainability by giving a clear and concise context to the reviewer of what’s being solved at a high level, taking off an extra step for the reviewer.

If any of the pr-sanitizer checks finds out that the pull request doesn’t comply with any of the predefined conventions then it marks the plugin’s check as Failure, or as success otherwise.

3.4.1. Title Verification

This ensures that the PR title conforms with the semantic commit message style. Conformance with the semantic commit message style not only makes your changelog and git history clean and easy to navigate but also encourages making atomic pull requests that address only a single concern, better reflect the change within and are more clearly understood.

The format below shows a rigid semantic commit message style.

<type>(<scope>): <subject>

  • where <scope> is optional.

  • and each of <type> categorizes the PR according to the change introduced (such as feat for introducing a new feature and fix for patching a bug in codebase). The default types are feat, fix, docs, style, refactor, test, and chore.

3.4.2. Description Verification

The description verification validates two things:

  • link to an issue the pull request is related to

  • the length of the description

Issue link check verifies that the PR description contains an issue link used with any of the GitHub keywords that ensure closing the issue when the pull request is merged. The supported keywords are closes, closed, fix, fixes, fixed, resolve, resolves, resolved. More information about closing issues using keywords can be found here.

Content Length Check

Content length check verifies that the PR description has a minimal number of characters (50 by default). With this check, it is possible to verify (to some degree) that the description contains more elaborate information about the changes proposed in the PR.

If the description contains an issue link used with any of the GitHub keywords (e.g. close #1) then this part is not taken into consideration when the length of the description is being measured.

3.4.3. Plugin Configuration

To configure PR Sanitizer plugin place pr-sanitizer.yml (or pr-sanitizer.yaml) file inside of the directory .ike-prow/ in your project and use properties described below.

pr-sanitizer.yaml
type_prefixes:            (1)
  - ':star:'
combine_defaults: true   (2)
description_content_length: 40  (3)
1 Defines set of custom types which will be used to match titles prefixes in changeset and determine if Pull Requests Title conforms with the semantic message style. (Eg. above code snippet configures emoji style semantic commit message.)
2 Allows you to decide if you want to combine your patterns with the list (feat, fix, refactor, docs, test, chore, style) of predefined default types (true by default).
3 Set Number of characters to be verified in PR description content.

3.4.4. Status message

When there is a PR that doesn’t conform with the conventions, then plugin (apart form setting the failure status) adds a comment explaining what is wrong and what the developer should do. If the PR is modified then the status message in the comment is updated respectively to reflect the latest state of the PR.

Custom status message

Any of the status messages can be changed by putting the required custom message to any of the following files:

  • pr-sanitizer_failed_message.md for the case when the PR doesn’t conform with the conventions

  • pr-sanitizer_success_message.md for the case when the PR is modified so it conforms with the conventions

Both of them has to be located in the directory .ike-prow/

3.4.5. Status details

In this section, you can find status details description applicable for each state of the pr-sanitizer plugin.

Success

Your Pull Request has been approved as it complies with the PR conventions given by the pr-sanitizer plugin.

Failure

Your Pull Request has been rejected because the plugin detected that your pull request doesn’t comply with the predefined PR conventions. For more information see the status message that should have been added to the PR as a comment. The possible causes of the check failure are following:

Title missing semantic prefix

The pull request title does not start with any of the "type prefixes", so it seems that it does not conform with the semantic commit message style.

Edit the PR title so it conforms with the semantic commit message style by prefixing it with one of the "type prefixes".

For more information see Title Verification and Plugin Configuration sections.

Description content is too short

The pull request description (excluding issue link and it’s keywords) has less than the minimal number of characters (50 by default).

Update the PR description by adding more information so it contains at least the minimal number of characters.

For more information see Description - Content Length Verification

The pull request description doesn’t contain any issue link used in a combination with any of the GitHub keywords that are able to automatically close the related issue.

Update the PR description by linking a issue.

For more information see Description - Issue Link Verification

3.5. Trigger Plugins on Demand

In order to trigger plugin on demand, just add /run plugin-name(e.g. /run test-keeper) comment on pull request. If you want to trigger only specific set of plugins, you can trigger it by adding comment /run plugin-A plugin-B(e.g. /run test-keeper work-in-progress). However if you want to run all plugins configured for your repository, just add /run all comment on pull request.