Simulator 0.5 released!

Today we released version 0.5 of the Hazelcast Simulator tool. It is a production simulator used to test Hazelcast and Hazelcast based applications in clustered environments. You can use Hazelcast Simulator for the following use cases:

• In pre-production to simulate the expected throughput/latency of Hazelcast with your specific requirements.
• To test if Hazelcast behaves as expected when you implement a new functionality in your project.
• As part of your test suite in your deployment process.
• When upgrading your Hazelcast version.

Please read “Simulator 0.4 released” for a general introduction or have a look at the Hazelcast Simulator Documentation. You can download Hazelcast Simulator here.

@RunWithWorker

The biggest change of this release is the new @RunWithWorker atation for tests. We wanted to reduce some boilerplate code which we found in almost all tests, e.g. the ThreadSpawner to create multiple worker threads. We also wanted to have built-in probes and OperationSelector support. The new interface IWorker and the annotation provides that.

A very minimal test (without verification) could look like this:

package com.hazelcast.simulator.tests;
    
    import com.hazelcast.core.HazelcastInstance;
    import com.hazelcast.core.IMap;
    import com.hazelcast.simulator.test.TestContext;
    import com.hazelcast.simulator.test.annotations.RunWithWorker;
    import com.hazelcast.simulator.test.annotations.Setup;
    import com.hazelcast.simulator.test.annotations.Teardown;
    import com.hazelcast.simulator.worker.selector.OperationSelectorBuilder;
    import com.hazelcast.simulator.worker.tasks.AbstractWorker;
    
    public class ExampleTest {
    
      private enum Operation {
        PUT,
        GET
      }
    
      // properties
      public double putProb = 0.5;
    
      private final OperationSelectorBuilder<Operation> builder
          = new OperationSelectorBuilder<Operation>();
    
      private IMap<Integer, String> map;
    
      @Setup
      public void setUp(TestContext testContext) throws Exception {
        HazelcastInstance targetInstance = testContext.getTargetInstance();
        map = targetInstance.getMap("exampleMap");
    
        builder
            .addOperation(Operation.PUT, putProb)
            .addDefaultOperation(Operation.GET);
      }
    
      @RunWithWorker
      public Worker createWorker() {
        return new Worker();
      }
    
      private class Worker extends AbstractWorker<Operation> {
    
        public Worker() {
          super(builder);
        }
    
        @Override
        protected void timeStep(Operation operation) {
          int key = randomInt();
          switch (operation) {
            case PUT:
              map.put(key, "value" + key);
              break;
            case GET:
              map.get(key);
              break;
            default:
              throw new UnsupportedOperationException(
                  "Unknown operation" + operation);
          }
        }
      }
    }

The annotated method has to return a new instance of an IWorker implementation. By using @RunWithWorker you get a built-in probe injected, per default a WorkerProbe. It provides throughput values during runtime, but creates no output file. You can easily overwrite the probe type (and the worker thread count) in your test properties:

ExampleTest@class = com.hazelcast.simulator.tests.ExampleTest
    ExampleTest@workerProbeType = hdr
    ExampleTest@threadCount = 20

The available probe types are: disabled, worker, throughput, latency, maxlatency and hdr.

IWorker, AbstractWorker and AbstractMonotonicWorker

The AbstractWorker<O> is a ready to use implementation of the IWorker interface. You just have to implement the timeStep(O operation) method and provide an OperationSelectorBuilder<O> for the constructor. You can override additional methods like beforeRun() and afterRun() or access a Random instance via randomInt() or getRandom(). Please have a look at the class and its JavaDoc for more information.

If you have a single operation test the AbstractMonotonicWorker is the better fit. It extends the AbstractWorker, but just requires a timeStep() method to be implemented, which has no operation parameter.

HdrProbe

We fixed the HdrProbe and enhanced our Visualizer tool. The HdrProbe makes use of HdrHistogram to record and analyze latency information. Our Visualizer tool displays the recorded values of one or more probes to compare results from different test runs:

The result of an IntIntMapTest compared to a StringStringMapTest:

You can see a lower throughput and a slightly higher latency with the IMap<String, String> in comparison with the IMap<Integer, Integer> (please focus on the relative changes, since the tests were run in a single local VM and the absolute numbers are not very meaningful).

MapStreamer

To speed up the warmup phases you can fill a map with the new MapStreamer:

private IMap<Integer, Integer> map;
    
    @Warmup
    public void warmup() throws InterruptedException {
      Random random = new Random();
      MapStreamer<Integer, Integer> streamer
          = MapStreamerFactory.getInstance(map);
    
      keys = generateIntKeys(keyCount, Integer.MAX_VALUE,
          KeyLocality.RANDOM, hazelcastInstance);
      for (int key : keys) {
        int value = random.nextInt(Integer.MAX_VALUE);
        streamer.pushEntry(key, value);
      }
      streamer.await();
    }

With Hazelcast version 3.5 or newer it does use asynchronous IMap operations so it’s extremely fast, but it has own back-pressure and doesn’t rely on back-pressure provided by Hazelcast. For older Hazelcast versions a synchronous version is created by the factory.

Small improvements and fixes

Beside that we upgraded jclouds to version 1.9 to support new EC2 machine classes. The utility classes have been moved to a separate module. The Agent binds to 0.0.0.0 now, so you can use Hazelcast Simulator on 127.0.0.1 (hostnames are not yet supported). You can define test properties for TestRunner, which is used to run a test locally during test development.

With the help of CheckStyle, FindBugs and SonarQube we resolved a lot of smaller issues and increased the code coverage of Hazelcast Simulator by about 10% since the last release. We fixed some Maven issues and the total performance count in PerformanceMonitor. The AgentSmokeTest and AgentRemoteServiceTest run on Jenkins now. We fixed TestRunner.withDuration() and fail fast on an empty agents.txt file. At last we fixed a NPE during runtime when Hazelcast Simulator was built without a git repository.