Hazelcast IMDG 3.9 EA is available

Hello Hazelcasters!

First off I would like to introduce myself – I started contributing to the project about 4 years ago, later on I joined as a performance engineer and in last 2 years I led the core engineering team. I am very excited to be stepping to the position of head of engineering at here at Hazelcast.

Now on to the BIG news. We just released an Early Access version of Hazelcast IMDG 3.9! The whole Engineering team worked incredibly hard to make it happen and in this post I would like to introduce some of the new exciting features.

User Code Deployment from Clients

In Hazelcast 3.8, we introduced an option for automatic distribution of your domain classes across cluster members. This greatly simplifies a deployment process as you don’t need to copy JARs across your cluster members. It also allows you to add new classes without a cluster restart. We received a lot of feedback on this feature; probably the most common feature request was to allow clients to push new classes into a running cluster. Well, here is the good news– it’s part of the IMDG 3.9 Release! It’s disabled by default for security reasons, but it only takes 1 switch to enable it.

Dynamic Data Structure Configs

Dynamic configuration of data structure is another popular feature request that we implemented. In older Hazelcast versions, you would have to provide all configurations upfront. There was no supported way to submit e.g., a new IMap configuration into a running cluster. Some people applied wildcards in configuration names as a workaround, but we have always felt it was an unnecessary limitation.

In IMDG 3.9, this is no longer necessary. You can now submit new configurations into a running cluster. It’s as easy as:

    MapConfig mapConfig = new MapConfig("myDynamicNewMap");
    mapConfig.setSomeOption();
    mapConfig.setAnotherOption();
    hazelcastInstance.getConfig().addMapConfig(mapConfig);

And Hazelcast will automatically distribute this configuration to all cluster members.

Store-by-Reference in Near Cache

Near Cache is often used to serve a small subset of very hot items with minimal latency. When an entry is found in a Near Cache, it can then be immediately returned to the caller and no network communication is involved. Speed is the “raison d’être” of Near Cache, and in Hazelcast 3.9 we made it even faster. In older versions, we used to store Near Cache keys as serialized blobs. It has some nice properties, but it also means we had to serialize a key with each get request. This proved to be a bottleneck and IMDG 3.9 allows to you use deserialized objects as near-cache keys.

Finer-Grained Anti-Entropy

We invest a lot of our energy and brain-power into making Hazelcast IMDG rock stable– even under unhappy circumstances. We have had an anti-entropy mechanism for automatic correction of out-of-the-sync backups for a long time. When the mechanism detects an inconsistency between primary replica and backups, it will then start a reconciliation process.

This is invisible to users as it is considered to be an internal concern. However, it can still have negative side-effects: When partitions are bulky, it can cause quite burden on the system. In older versions of Hazelcast, a single partition was the minimal reconciliation unit. Even when a single entry was missing in a single map, anti-entropy mechanism would always copy the whole partition.

In IMDG 3.9, we are now making a step towards a finer grained reconciliation and migrate only the maps & caches with missing records. This reduces latency spikes caused by this process and contributes to a smooth operational experience.

Lite Member Promotion

Lite cluster members act as regular members, but they do not own any data. You can use them as pure computation nodes or as an alternative to clients. In 3.9 we implemented a feature request we received over our Gitter chat: You can start a Lite member and promote it later to a full member if needed. This is useful when your application is starting, but you want to delay data migrations until the application is fully initialized.

Manual Migration Control

When a member joins or leaves a cluster then Hazelcast automatically triggers data re-balancing. This makes the cluster nicely elastic, but data re-balancing is a quite expensive process, it usually involves sending big chunks of data over network. We are now introducing a new cluster state NO_MIGRATION – when where we do not trigger re-balancing when a new members are joining or leaving. This has a few use-cases. One of them is adding a bunch of new nodes and post-pone the re-balancing when all nodes are joined.

Client Connection Strategies

You can now configure how a client behaves when it has no connection to a cluster. Clients can now support three different behaviors:

1. Silent Reconnect. This blocks all operations until it reconnects.
2. Abort all in-flight operations and reconnect.
3. Do not reconnect at all.

High-Density Memory Store Indexes (Enterprise only)

High-Density (HD) IMap is great way to achieve a consistent performance regardless of your data size. It massively reduces impact of JVM Garbage Collections (GC) on your system. In older versions, it did not play nicely with search indexes; the indexes were stored in a regular Java Heap and therefore they contributed to GC pauses. In Hazelcast 3.9, HD IMap always stores indexes in HD memory. It means you can have a nice and predictable performance and still use indexes to speed-up queries.

Rolling Upgrade from 3.8 (Enterprise only)

In IMDG 3.8, we introduced an infrastructure to support Rolling Upgrade to a new minor release. IMDG 3.9 is the first version using this infrastructure; You can now upgrade your IMDG 3.8 cluster into 3.9 without any outage! Note, this is supported only for final versions. Rolling Upgrade from IMDG 3.8 to 3.9 Early Access still has a few bugs that we will resolve before Hazelcast 3.9 General Availability (final) is release.

WAN with Discovery Service Provider Interface (Enterprise only)

We’ve had the Discovery Service Provider Interface (SPI) for a number of IMDG releases. There are providers for Kubernetes, Consul, Eureka, and many others. It’s a simple and popular way for cluster member discovery. In IMDG 3.9, we use the very same SPI also for WAN discovery. This means you can use the same providers in WAN configuration and different clusters will discover each other automatically!

OpenSSL Integration (Enterprise only)

We are proud of our zero-dependency policy. It makes Hazelcast very “embeddable” and it eliminates JAR hell. However, during our bench marking we discovered the SSL engine in most Java implementations can be a performance bottleneck. Hence in IMDG 3.9, we are adding an option to use the native OpenSSL library for traffic encryption. When you do not configure or the library is unavailable, it will then use the Java implementation as in older versions.

Disclaimer

We do not recommend running Early Access versions in production, but we encourage everyone to grab it and play with the new features. Lets us know what you think. We love to hear your feedback, our developers tend to hang out at Gitter so feel free to stop by and contribute!

Download Hazelcast IMDG 3.9 EA Now!