public abstract class Benchmark
extends GWTTestCase

A type of GWTTestCase which specifically records performance results. Benchmarks have additional functionality above and beyond GWT's JUnit support for standard TestCases.

Reporting

In a single JUnit run, the results of all executed benchmarks are collected and stored in an XML report viewable with the benchmarkViewer.

Permutations

GWT supports test methods that have parameters. GWT will execute each benchmark method multiple times in order to exhaustively test all the possible combinations of parameter values. All of your test method parameters must be annotated with a Range annotation such as RangeField or RangeEnum. For example,

 
 public void testArrayListRemoves(
   @RangeEnum(Position.class) Position where, 
   @RangeField("insertRemoveRange") Integer size) { ... 
 }
 

Timing

• GWT automatically removes jitter from your benchmark methods by running them for a minimum period of time (150ms).
• GWT supports time limits on the maximum duration of each permutation of a benchmark method. With this feature, you can supply very high upper bounds on your ranges (such as Integer.MAX_VALUE), which future-proofs your benchmarks against faster hardware.
• GWT supports Setup and Teardown methods which separate test overhead from the actual work being benchmarked. The timings of these lifecycle methods are excluded from test results.

Notes

Please note that Benchmarks do not currently support asynchronous testing mode. Calling GWTTestCase.delayTestFinish(int) or GWTTestCase.finishTest() will result in an UnsupportedOperationException.

Examples of benchmarking in action

A simple benchmark example

AllocBenchmark is an example of a basic benchmark that doesn't take advantage of most of benchmarking's advanced features.

public class AllocBenchmark extends Benchmark {

  private static final int numAllocs = 1000;

  @Override
  public String getModuleName() {
    return "com.google.gwt.examples.Benchmarks";
  }

  /**
   * Allocates java.lang.Object in a for loop 1,000 times.
   *
   * The current version of the compiler lifts the declaration of obj outside
   * of this loop and also does constant folding of numAllocs.
   * Also, this loop allocs the GWT JS mirror for java.lang.Object
   * <em>NOT</em> an empty JS object, for example.
   *
   */
  public void testJavaObjectAlloc() {
    for ( int i = 0; i < numAllocs; ++i ) {
      Object obj = new Object();
    }
  }

  /**
   * Compares GWT mirror allocations of java.lang.Object to an empty JS object.
   */
  public native void testJsniObjectAlloc1() /*-{
    for (var i = 0; i < @com.google.gwt.examples.benchmarks.AllocBenchmark::numAllocs; ++i ) {
      var obj = {}; // An empty JS object alloc
    }
  }-*/;

  /**
   * Like version 1, but also folds the constant being used in the iteration.
   */
  public native void testJsniObjectAlloc2() /*-{
    for (var i = 0; i < 1000; ++i ) {
      var obj = {}; // An empty JS object alloc
    }
  }-*/;

  /**
   * Like version 2, but hoists the variable declaration from the loop.
   */
  public native void testJsniObjectAlloc3() /*-{
    var obj;
    for (var i = 0; i < 1000; ++i ) {
      obj = {}; // An empty JS object alloc
    }
  }-*/;

  /**
   * Like version 3, but counts down (and in a slightly different range).
   */
  public native void testJsniObjectAlloc4() /*-{
    var obj;
    for (var i = 1000; i > 0; --i ) {
      obj = {}; // An empty JS object alloc
    }
  }-*/;
}

An advanced benchmark example

ArrayListBenchmark is a more sophisticated example of benchmarking. It demonstrates the use of Setup and Teardown test methods, parameterized test methods, and time limits.

public class ArrayListBenchmark extends Benchmark {

  private static final int PRIME = 3001;

  /**
   * The various positions that data can be inserted into a list.
   */
  protected enum Position {

    BEGIN("at the beginning"), EXPLICIT_END("explicitly at the end"), IMPLICIT_END(
        "implicitly at the end"), VARIED("in varied locations");

    private String label;

    /**
     * Constructor for <code>Position</code>.
     * 
     * @param label a not <code>null</code> label describing this
     *          <code>Position</code>.
     */
    Position(String label) {
      this.label = label;
    }

    /**
     * Returns the textual description for the position.
     * 
     * @return a not <code>null</code> description.
     */
    @Override
    public String toString() {
      return label;
    }
  }

  protected final List<Position> explicitPositions = Arrays.asList(
      Position.BEGIN, Position.EXPLICIT_END, Position.VARIED);

  protected final IntRange insertRemoveRange = new IntRange(64,
      Integer.MAX_VALUE, Operator.MULTIPLY, 2);

  protected final IntRange baseRange = new IntRange(512, Integer.MAX_VALUE,
      Operator.MULTIPLY, 2);

  List<String> list;

  int index = 0;

  @Override
  public String getModuleName() {
    return "com.google.gwt.emultest.EmulSuite";
  }

  /**
   * Appends <code>size</code> items to an empty {@code List}.
   * 
   * @param size the size of the {@code List}
   */
  @Setup("beginListAdds")
  public void testListAdds(@RangeField("baseRange")
  Integer size) {
    int num = size.intValue();
    for (int i = 0; i < num; i++) {
      list.add("hello");
    }
  }

  // Required for JUnit
  public void testListAdds() {
  }

  /**
   * Performs <code>size</code> gets on a {@code List} of size,
   * <code>size</code>.
   * 
   * @param size the size of the {@code List}
   */
  @Setup("beginListGets")
  public void testListGets(@RangeField("baseRange")
  Integer size) {
    int num = size.intValue();
    for (int i = 0; i < num; i++) {
      list.get(i);
    }
  }

  // Required for JUnit
  public void testListGets() {
  }

  /**
   * Performs <code>size</code> inserts at position, <code>where</code>, on
   * an empty <code>List</code>.
   * 
   * @param where Where the inserts happen
   * @param size The size of the <code>List</code>
   * 
   */
  @Setup("beginListInserts")
  public void testListInserts(@RangeEnum(Position.class)
  Position where, @RangeField("insertRemoveRange")
  Integer size) {
    insertIntoCollection(size, where, list);
  }

  // Required for JUnit
  public void testListInserts() {
  }

  /**
   * Performs <code>size</code> removes at position, <code>where</code>, on
   * an ArrayList of size, <code>size</code>.
   * 
   * @param where Where the inserts happen
   * @param size The size of the <code>List</code>
   */
  @Setup("beginListRemoves")
  public void testListRemoves(@RangeField("explicitPositions")
  Position where, @RangeField("insertRemoveRange")
  Integer size) {
    removeFromCollection(size, where, list);
  }

  // Required for JUnit
  public void testListRemoves() {
  }

  /**
   * Creates a new empty List.
   * 
   * @return a not <code>null</code>, empty List
   */
  protected List<String> newList() {
    return new ArrayList<String>();
  }

  void beginListAdds(Integer size) {
    list = newList();
  }

  void beginListGets(Integer size) {
    createList(size);
  }

  void beginListInserts(Position where, Integer size) {
    list = newList();
    index = 0;
  }

  void beginListRemoves(Position where, Integer size) {
    beginListInserts(where, size);
    testListInserts(where, size);
  }

  private void createList(Integer size) {
    beginListAdds(size);
    testListAdds(size);
  }

  private void insertIntoCollection(Integer size, Position where, List<String> l) {
    int num = size.intValue();
    for (int i = 0; i < num; i++) {
      if (where == Position.IMPLICIT_END) {
        l.add("hello");
      } else if (where == Position.BEGIN) {
        l.add(0, "hello");
      } else if (where == Position.EXPLICIT_END) {
        l.add(l.size(), "hello");
      } else if (where == Position.VARIED) {
        l.add(index, "hello");
        index += PRIME;
        index %= l.size();
      }
    }
  }

  private int removeFromCollection(Integer size, Position where, List<String> l) {
    int num = size.intValue();
    for (int i = 0; i < num; i++) {
      if (where == Position.IMPLICIT_END) {
        throw new RuntimeException("cannot remove from the end implicitly");
      } else if (where == Position.BEGIN) {
        l.remove(0);
      } else if (where == Position.EXPLICIT_END) {
        l.remove(l.size() - 1);
      } else if (where == Position.VARIED) {
        l.remove(index);
        index += PRIME;
        int currentSize = l.size();
        if (currentSize > 0) {
          index %= l.size();
        }
      }
    }
    return index;
  }
}