GridGain/Apache Ignite have released a benchmark which purports to show that they are faster than Hazelcast. This follows on from our Hazelcast/Ignite Benchmark using Yardstick, their own benchmark, which shows Hazelcast 3.6-EA was much faster than Ignite 1.4.1.
In looking into it we discovered that they had faked the test. Instead of configuring the tests the same they were using:
sync backups in Hazelcast for replicas while configuring their own backups as async.
map.put() instead of map.set() in Hazelcast, while using put() is equivalent to our set() operation because their put() does not have a return value.
Async vs Sync
They configure the test using the following script:
-b 1 stands for 1 backup
–sm FULL_SYNC stands for synchronization mode full sync.
In the Benchmark as published on 16 February, they are using 1 backup and sync mode PRIMARY_SYNC.
This means that you wait for response of the primary and not the ack’s of the backups. This is comparable to Hazelcast async-backup. However, they are comparing their PRIMARY_SYNC to our sync backup and that is unfair. So if they want to make a fair comparison they should select FULL_SYNC (waiting for all primaries + backups).
And this is exactly what did they yesterday:
As you can see, they changed sync mode from PRIMARY_SYNC to FULL_SYNC. However, they have not updated their performance report. In Hazelcast switching to async has a huge impact on performance compared to sync; async is almost as fast as having no backup at all.
put void vs put with return value
They are comparing map.put (which returns a value) with their cache:
This is the Ignite test: [Benchmark removed by Apache Ignite]
Here is the definition of the cache: https://github.com/apacheignite/yardstick-ignite/blob/master/src/main/java/org/apache/ignite/yardstick/cache/IgniteCacheAbstractBenchmark.java
It is an IgniteCache. For documentation of the put:https://ignite.apache.org/releases/1.5.0-b1/javadoc/org/apache/ignite/IgniteCache.html#put%28K,%20V%29
As you can see it is a void.
The Hazelcast test being used is: HazelcastPutBenchmark: https://github.com/gridgain/yardstick-hazelcast/blob/master/src/main/java/org/yardstickframework/hazelcast/HazelcastPutBenchmark.java
As you can see the put is being called. See the below link for the javadoc for put; which shows a return value.
So as you clearly can see, they are comparing a put without a return value, to a put with a return value. Since the value is small, I don’t think this will make a huge impact.
An Apples to Apples Test
We ran a comparison using our own benchmark tool correcting for these issues. And we were way faster. Just as we were against Apache Ignite 1.4.1.
We believe this behavior is unbecoming of an Apache project. In fact it is a breach of the Apache Incubator Code of Conduct which prohibits conduct which “Is libelous, defamatory, knowingly false or misrepresents another person”. We believe their benchmark is an intentional misrepresentation.
We worked very hard to increase our performance over the past year, and would like to see competitors improving performance in their projects and products rather than faking tests.
4 nodes, 20 clients, 64 threads in each client that produces load
4 Servers (both nodes and clients are run on 4 servers)
2x Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz – with 10 cores per processor. Total 20 cores, 40 with hyper threading enabled.
768GB 2133 MHz memory 24x HP 32GB 4Rx4 PC4-2133P-L Kit
Heap Configuration member: “-Xms500m -Xmx1g” client: “-Xms100m -Xmx500m”
Hazelcast XML Configuration
<map name="hzAsyncOnheap*"> <in-memory-format>BINARY</in-memory-format> <backup-count>0</backup-count> <async-backup-count>1</async-backup-count> </map>
GridGain/Apache Ignite XML Configuration
<bean class="org.apache.ignite.configuration.CacheConfiguration"> <property name="name" value="atomic"/> <property name="cacheMode" value="PARTITIONED"/> <property name="atomicityMode" value="ATOMIC"/> <property name="swapEnabled" value="false"/> </bean>
Hazelcast is 20-50% faster.
99 Percentile Latency Results
Much lower latencies for Hazelcast
Red: Hazelcast 3.6 Blue: Gridgain/Apache Ignite 1.5