Why Your Java Benchmarks Lie — And How JMH's 10 Patterns Fix That

I used to measure Java code with a loop and a stopwatch. I would call a method one million times and divide the total time by one million. The numbers looked precise. They made me feel like a scientist. Then I changed the order of two benchmarks and the results flipped. Sometimes the first benchmark made the second one look faster. Sometimes the compiler got rid of the whole loop because the result was never used. I stopped believing my own numbers. JMH changed that.

JMH stands for Java Microbenchmark Harness. The OpenJDK team maintains it. It is a small framework that runs your benchmark method in a controlled way, handles warmup, stops the compiler from deleting your code, and gives you a confidence interval at the end. In this article I walk through ten patterns I use every time I need a reliable measurement. If you are new to JMH, these patterns will save you hours of painful debugging.

1. Use a Dedicated Benchmark Class with @Benchmark Methods

A microbenchmark measures one tiny unit of work. The unit might be a single method call, a small calculation, or a short loop. You need a class that JMH can scan, and you need at least one method marked with @Benchmark.

import org.openjdk.jmh.annotations.*;

public class StringBench {

    @Benchmark
    public String buildWithBuilder() {
        return new StringBuilder("a").append("b").append("c").toString();
    }
}

JMH looks for methods with @Benchmark and creates a harness around them. The harness calls the method many times, measures each call, and prints a report. The class itself does not need to extend anything. The method can be public or package-private.

You need two JMH dependencies in your build file. With Maven, add this to your pom.xml:

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.37</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.37</version>
</dependency>

You also need a way to run it. The simplest approach is to build a jar with a main class. The Maven Shade plugin works well for this:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.5.1</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals><goal>shade</goal></goals>
            <configuration>
                <finalName>benchmarks</finalName>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>org.openjdk.jmh.Main</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

Build with mvn clean package, then run with java -jar target/benchmarks.jar.

The important rule is to make each benchmark method represent a single unit of work. Do not wrap your code in a loop that runs ten thousand times unless you want to measure the loop itself. If your method creates a StringBuilder, let each call create one object. JMH will call the method in a controlled loop for you.

2. Prevent Dead Code Elimination with Blackhole

The Java runtime is smart. If you write code that calculates a value and then never uses it, the runtime can remove that code. This is called dead code elimination. It is a great optimization for real programs. It is terrible for benchmarks.

Here is a benchmark that looks like it measures a sum:

@Benchmark
public void sumWithoutUsingResult() {
    int total = 0;
    for (int i = 0; i < 1000; i++) {
        total += i;
    }
}

The method computes total but returns nothing and stores nothing. The runtime may decide the whole loop is pointless and skip it. Your benchmark will show near-zero time, even though the loop is doing real work when the result is used.

JMH gives you a tool called Blackhole. A Blackhole is a special object that consumes values. When you pass a value to blackhole.consume(), the runtime believes the value will be used later, so it keeps all the computation needed to create that value.

import org.openjdk.jmh.infra.Blackhole;

@Benchmark
public void sumWithBlackhole(Blackhole blackhole) {
    int total = 0;
    for (int i = 0; i < 1000; i++) {
        total += i;
    }
    blackhole.consume(total);
}

I use Blackhole whenever my benchmark method would normally return a value. If the method has a return type, JMH treats the returned value as consumed automatically. So this version is also safe:

@Benchmark
public int sumWithReturn() {
    int total = 0;
    for (int i = 0; i < 1000; i++) {
        total += i;
    }
    return total;
}

When you have multiple values you need to keep alive, call blackhole.consume for each one. Do not try to be clever and combine them into one object. The runtime may optimize away one field and keep another. Use separate calls.

3. Carry Test Data in @State Objects

Literal constants are another trap. If your benchmark method uses a fixed value, the compiler can fold it into the calculation at compile time. The runtime may compute the answer before the benchmark even runs.

Here is an example of a constant that can ruin your measurement:

@Benchmark
public long sumFixedNumbers() {
    long total = 0;
    for (int i = 1; i <= 100; i++) {
        total += i;
    }
    return total;
}

The runtime knows that the sum of 1 through 100 is 5050. It can replace the whole loop with return 5050. You would measure the time needed to return a constant, not the time needed to add numbers in a loop.

The fix is to keep test data in a @State object. JMH creates the state object for you and passes it to the benchmark method. The state object can hold an array, a list, a random seed, or any other input.

import java.util.Random;

@State(Scope.Thread)
public class ArrayData {
    public int[] values = new Random().ints(10_000).toArray();
}

@Benchmark
public long sumState(ArrayData data) {
    long total = 0;
    for (int v : data.values) {
        total += v;
    }
    return total;
}

The @State annotation has a scope. Scope.Thread means every worker thread in the benchmark gets its own copy of the state. Scope.Benchmark means all threads share one state object. Scope.Group means threads within a group share state. I use Scope.Thread most of the time because it avoids contention and keeps each thread from interfering with another.

The important part is that the data is created outside the timed benchmark method. JMH does not count the time to create the state object. The benchmark only measures the loop working with that data.

4. Exercise Multiple Input Sizes with @Param

One benchmark method is nice. Ten benchmark methods for ten input sizes are annoying. JMH solves this with @Param. The annotation tells JMH to run the same method once for each value you give it.

@State(Scope.Thread)
public static class Params {
    @Param({"10", "1000", "100000"})
    public int length;
}

@Benchmark
public long sumFirstN(ArrayData data, Params params) {
    long total = 0;
    int limit = Math.min(params.length, data.values.length);
    for (int i = 0; i < limit; i++) {
        total += data.values[i];
    }
    return total;
}

JMH will run sumFirstN for length 10, then 1000, then 100000. The report will show three separate lines. This makes it easy to see how performance changes as the input grows.

You can also use @Param directly on the state object to control the size of the array:

@State(Scope.Thread)
public static class SizedData {
    @Param({"100", "10000"})
    public int size;

    public int[] values;

    @Setup(Level.Trial)
    public void setup() {
        values = new Random().ints(size).toArray();
    }
}

I like @Param because it forces me to think about scaling. A method can be fast for a tiny input and slow for a large one. Without parameters, I might only measure the tiny case and make wrong assumptions.

5. Choose the Right Execution Mode

JMH has several execution modes. The default mode is throughput. Mode.Throughput counts how many operations complete in a second. This is useful for batch processing jobs.

Mode.AverageTime measures the average time for one operation. This is the mode I use when I care about latency. If I want to know how many nanoseconds a JSON parser takes for a single message, I use AverageTime.

Mode.SampleTime records individual operation times and builds a distribution. This is good for tail latency. You can see if most calls are fast but a few are slow.

Mode.SingleShotTime runs the benchmark exactly once per invocation. This is useful for cold code or one-off initialization, but it is harder to use correctly.

You set the mode with @BenchmarkMode.

@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public long avg() {
    return someOperation();
}

You can also set the mode from the command line when running the jar:

java -jar target/benchmarks.jar -bm avgt

I use AverageTime for latency measurements and Throughput for everything else. Sample time is more advanced, so I run it separately when I need to understand the distribution.

6. Configure Warmup Iterations to Reach Steady State

The Java runtime does not compile your code all at once. It starts by interpreting the bytecode, then compiles methods when they become hot. The compilation takes time. If you start measuring immediately, your numbers will be noisy because the code is still changing.

JMH runs warmup iterations before it starts measuring. The warmup iterations do not appear in the final report. They exist only to let the JIT compiler finish its work and let the code reach a steady state.

Use @Warmup to tell JMH how many warmup iterations to run and how long each one lasts.

@Benchmark
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(2)
public void steadyState() {
    // code goes here
}

The default unit for time is seconds, so time = 1 means one second. Five seconds of warmup is often enough for simple methods. Complex code with many calls and inlining may need twenty or thirty seconds.

I always look at the score column when I increase warmup. If the score keeps dropping, the code has not warmed up enough. If the score stays flat across iterations, the code is stable. A benchmark value that changes wildly between runs is a sign that the JIT is still doing work.

7. Fork JVMs to Isolate Benchmarks

JMH can run each benchmark in a fresh Java process. This is called forking. The @Fork annotation controls how many times JMH starts a new JVM for the benchmark.

@Fork(3)
public class MyBenchmark {
    // benchmarks here
}

I always fork when comparing two implementations. If I run two benchmarks in the same JVM, the first one can affect the second. It might load classes into the code cache, trigger compilation, or leave static state behind. The second benchmark then starts with a warm code cache and looks faster than it should.

A fresh JVM gives each benchmark a clean starting point. The tradeoff is time. Forking three JVMs means the benchmark takes three times longer. But the results are much more trustworthy.

You can also fork from the command line:

java -jar target/benchmarks.jar -f 3

If you have many benchmarks, running all of them with three forks can take a while. That is okay. Reliable numbers beat fast but unreliable numbers.

8. Use Built-In Profilers to Find Where Time Goes

Timing data tells you which benchmark is faster. It does not tell you why. JMH includes several profilers that run alongside the benchmark and collect extra information.

The gc profiler is my first stop. It shows the allocation rate, the number of garbage collections, and the time spent in GC. A slow benchmark is often slow because it creates too many objects.

java -jar target/benchmarks.jar "StringBench.*" -prof gc

The output includes columns like Alloc rate and GC count. I once found that a method using String.format allocated much more than a method using string concatenation. The timing difference made sense after I saw the allocation numbers.

The stack profiler samples the call stack and produces a text-based profile. It shows which methods are on the hot path.

The perfasm profiler shows generated assembly code for very low-level analysis. I do not use it often because it requires Linux and a performance tool. When I do use it, I can see exactly which instructions are on the hot path.

These profilers add overhead. Do not include them in the same run you use for final timing numbers. Run the benchmark once without profilers to get the score, then run it again with a profiler to understand the cause.

9. Keep the Measured Code Inside the Benchmark Method

Every line inside the benchmark method is measured. If you create a random array inside the method, JMH will measure the array creation time along with the operation you care about. If you open a file or print to the console inside the method, the benchmark becomes a test of file I/O or console writing, not your code.

Put setup work in a @Setup method. JMH runs setup outside the timed region. The @Setup method runs once before a trial, before each iteration, or before each invocation, depending on the level.

import java.util.Random;
import org.openjdk.jmh.annotations.*;

@State(Scope.Thread)
public class SearchBench {

    private int[] values;

    @Setup(Level.Trial)
    public void setup() {
        values = new Random().ints(100_000).toArray();
    }

    @Benchmark
    public boolean findMiddle() {
        for (int v : values) {
            if (v == 50_000) {
                return true;
            }
        }
        return false;
    }
}

Level.Trial runs setup once for the whole benchmark. Level.Iteration runs setup before each measurement iteration. Use Level.Iteration when the state must be reset between iterations.

The benchmark method should look like a clean call to the operation you want to measure. If the operation needs an object, create that object in setup and store it in a field. If the operation needs to clear a data structure after each call, use @Setup(Level.Iteration) to reset it.

10. Report and Compare Results with Confidence Intervals

JMH prints a report that includes the score, the error, and the units. The error is a confidence interval. It tells you how much the measurement could vary due to random noise.

Here is an example report:

Benchmark                    Mode  Cnt   Score    Error  Units
MyBench.optionA             avgt   20  15.234 ± 0.125  ns/op
MyBench.optionB             avgt   20  15.411 ± 0.310  ns/op

The first benchmark has a score of 15.234 nanoseconds per operation. The error is plus or minus 0.125. The second benchmark has a score of 15.411 nanoseconds per operation. The error is plus or minus 0.310.

I compare the intervals. The first interval runs from about 15.109 to 15.359. The second interval runs from about 15.101 to 15.721. They overlap near 15.1 to 15.359. Because the intervals overlap, I cannot say with confidence that option A is faster than option B. The difference in the mean scores could be random noise.

If the intervals do not overlap, the result is more convincing. Even then, I run a few forks to be sure.

I also keep my benchmark environment stable. I close other applications, disable sleep mode, and use the same JVM flags for every comparison. If I compare two branches of code, I run them with the same command-line arguments and same machine.

I learned this lesson when I compared two sorting methods. The first benchmark appeared faster. Then I noticed the first benchmark had a much larger error bar. I ran it again with more forks, and the two methods were actually equal. My initial conclusion was wrong because I only looked at the mean score.

A Few Final Words of Advice

Start small. Pick one method you suspect is slow. Write a benchmark for it using the patterns above. Run it with warmup, forks, and a blackhole. Look at the confidence interval. Then add the profiler to see why it is slow.

JMH is not magic. It cannot measure a program that is already running while other programs eat CPU time. It cannot tell you what will happen on a production server with different hardware. It can tell you how your code behaves under controlled conditions. That is exactly what a microbenchmark should do.

When you use these ten patterns, the numbers you get will be closer to the truth. You will stop chasing ghosts caused by JIT compilation and dead code. I know I did. The first time I ran a JMH benchmark and saw a stable score that matched my intuition, I felt like I finally had a measuring stick I could trust.