Micro-benchmarking Framework for Jenkins Plugins

I have been working on improving the performance of the Role Strategy Plugin as a part of my Google Summer of Code project.
Since there was no existing way to measure performance and do benchmarks on Jenkins Plugins,
my work for the first phase of the project was to create a framework for running
benchmarks in Jenkins plugins with a Jenkins instance available. To make our job a bit easier,
we chose Java Microbenchmark Harness for running these benchmarks. This
allows us to reliably measure performance of our time-critical functions and will help make Jenkins perform faster
for everyone.

The micro-benchmarking framework was recently released in the Jenkins Unit Test Harness 2.50.
The blog post below shows how to run benchmarks in your plugins.

Introduction

The framework runs works by starting a temporary Jenkins instance for each fork of the JMH benchmark,
just like JenkinsRule from Jenkins Test Harness. Benchmarks are run directly from your JUnit Tests which allows
you to fail builds on the fly and easily run benchmarks from your IDE, just like unit tests. You can easily
configure your benchmarks by either using your Java methods, or by using Jenkins Configuration-as-Code plugin
and passing the path to your YAML file.

To run benchmarks from your plugins, you need to do the following:

bump up the minimum required Jenkins version to 2.60.3 or above
bump Plugin-POM to a version ≥ 3.46 or manually upgrade to Jenkins Test Harness ≥ 2.51.

Now, to run the benchmarks, you need to have a benchmark runner that contains a @Test so it can run
like a JUnit test. From inside a test method, you can use the OptionsBuilder provided by JMH to
configure your benchmarks. For example:

public class BenchmarkRunner {
    @Test
    public void runJmhBenchmarks() throws Exception {
        ChainedOptionsBuilder options = new OptionsBuilder()
                .mode(Mode.AverageTime)
                .forks(2)
                .result("jmh-report.json");

        // Automatically detect benchmark classes annotated with @JmhBenchmark
        new BenchmarkFinder(getClass()).findBenchmarks(options);
        new Runner(options.build()).run();
    }
}

Sample benchmarks

Now, you can write your first benchmark:

Without any special setup

@JmhBenchmark
public class JmhStateBenchmark {
    public static class MyState extends JmhBenchmarkState {
    }

    @Benchmark
    public void benchmark(MyState state) {
        // benchmark code goes here
        state.getJenkins().setSystemMessage("Hello world");
    }
}

Using Configuration as Code

To use configuration as code, apart from the dependencies above you also need to add the following
to your pom.xml:


    io.jenkins
    configuration-as-code
    1.21
    
true


    io.jenkins
    configuration-as-code
    1.21
    tests
    test

Now configuring a benchmark is as simple as providing path to your YAML file and specifying the class
containing the benchmark state.

@JmhBenchmark
public class SampleBenchmark {
    public static class MyState extends CascJmhBenchmarkState {
        @Nonnull
        @Override
        protected String getResourcePath() {
            return "config.yml";
        }

        @Nonnull
        @Override
        protected Class getEnclosingClass() {
            return SampleBenchmark.class;
        }
    }

    @Benchmark
    public void benchmark(MyState state) {
        Jenkins jenkins = state.getJenkins(); // jenkins is configured and ready to be benchmarked.
        // your benchmark code goes here...
    }
}

More Samples

As a part of this project, a few benchmarks have been created in the Role Strategy Plugin which show
configuring the instances for various situations. You can find them
here.

Running Benchmarks

Running benchmarks from Maven

To easily run benchmarks from Maven, a Maven profile to run the benchmarks has been created
and is available starting Plugin-POM version 3.45. You can then run your benchmarks from the
command line using mvn test -Dbenchmark.

Running benchmarks on ci.jenkins.io

If you have your plugins hosted on ci.jenkins.io, you can easily run benchmarks directly from your Jenkinsfile
by using the runBenchmarks() method after the buildPlugin() step in your which is now available in
Jenkins Pipeline library.
This function also accepts the path to your generated JMH benchmark reports as an optional
parameter and archives the benchmark results. Running benchmarks in pull request builds allows you to constantly
monitor the performance implications of a given change. For example, the Jenkinsfile from Role Strategy Plugin:

buildPlugin()
runBenchmarks('jmh-report.json')

Visualizing benchmark results

Benchmark reports generated (in JSON) can be visualized using the either the JMH Report Plugin
or by passing the benchmark reports to the JMH visualizer web service. As an example, here is
a visualized report of some benchmarks from the Role Strategy Plugin:

Role Strategy Plugin benchmarks visualized by JMH Visualizer

These improvements seen above were obtained through a small pull request
to the plugin and shows how even seemingly small changes can bring major performance improvements. Microbenchmarks
help to find these hot-spots and estimate the impact of changes.

Some tips and tricks

Since BenchmarkRunner class name in the example above does not qualify as a test according to Maven surefire plugin’s
naming conventions, the benchmarks will not interfere with your JUnit tests.
Benchmark methods need to be annotated by @Benchmark for JMH to detect them.
Classes containing benchmarks are found automatically by the BenchmarkFinder
when annotated with @JmhBenchmark.
A reference to the Jenkins instance is available through either JmhBenchmarkState#getJenkins() or through
Jenkins.getInstance() like you would otherwise do.
JmhBenchmarkState provides setup() and tearDown() methods which can be overridden to configure the
Jenkins instance according to your benchmark’s requirements.
The benchmark builds on ci.jenkins.io are currently throttled because of the limited availability of highmem nodes.
The benchmark framework was made available in Jenkins Test Harness 2.50, it is recommended to use version 2.51 as it includes some bug fixes.

Links and Feedback

If you have any feedback, comments or questions, please feel free to reach out to me through either
the Role Strategy Plugin Gitter chat or through
the Jenkins Developer Mailing list.

Presentation slides
Demo at Platform SIG meeting
Documentation for the micro-benchmark framework:

Originally posted on Jenkins Blog
Author: abhyudayasharma