Hazelcast Blog

See you at JavaOne

hazelcast — Fri, 21 Sep 2012 14:31:31 +0000

Hazelcast has two talks at JavaOne 2012:

Don’t forget to mark these talks in your JavaOne Calendar.

Please visit us at booth 5105 in the exhibition hall. Say hi! and see our very cool and fun demo. We also have tweet-shirt surprise for you: Send a tweet containing JavaOne and Hazelcast words during JavaOne conference, get your Hazelcast t-shirt from Hazelcast booth. Just for fun.

We hope to see you all at JavaOne!

Gartner Selected Hazelcast as a “Cool Vendor”

hazelcast — Fri, 04 May 2012 15:28:07 +0000

Each year, Gartner, leading industry analyst firm, selects the cool vendors in key technology areas and creates a report by evaluating the innovative vendors. We are honored to be selected as a “Cool Vendor” by Gartner in the Application and Integration Platforms for the year 2012. This confirms what we have been hearing from the Hazelcast community for a long time. Gartner’s recognition, by the report authored by Massimo Pezzini and Jess Thompson, is a reflection of our “cool” customers and users so we thank them for creating “cool” applications using Hazelcast.

Hazelcast will continue to innovate and be ahead of the game. We believe that developing distributed, cloud-ready applications should be simple and fun. Join Hazelcast community to experience the ultimate sophistication and joy.

Resources:

Learn more about Hazelcast
Obtain/buy the Gartner Report on Hazelcast
Add ‘Hazelcast’ skill to your Linkedin Profile
Join Linkedin Hazelcast Hackers Group
Join Hazelcast Mailing List

Disclaimer:
Gartner does not endorse any vendor, product or service depicted in our research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Hazelcast 2.0 Released: Big Data In-Memory

hazelcast — Thu, 08 Mar 2012 12:08:01 +0000

With 2.0 release, Hazelcast is ready for caching/sharing terabytes of data in-memory. Storing terabytes of data in-memory is not a problem but avoiding GC and being resilient to crashes are big challenges. Among several others, there are two major features added to tackle these challenges. Elastic Memory (off-heap storage) and Distributed Backups.

1. Elastic Memory

By default, Hazelcast stores your distributed data (map entries, queue items) into Java heap which is subject to garbage collection. As your heap gets bigger, garbage collection might cause your application to pause tens of seconds, badly effecting your application performance and response times. Elastic Memory is Hazelcast with off-heap memory storage to avoid GC pauses. Even if you have terabytes of cache in-memory with lots of updates, GC will have almost no effect; resulting in more predictable latency and throughput.

For each map storage type (off-heap/on-heap) can be configured. If map’s storage type is off-heap then primary, backup and even near-cached (if available) entries are all stored off the heap and your off-heap storage space will dynamically scale as you add more nodes.

Elastic Memory implementation doesn’t require any defragmentation. Here is how things work: User defines the number of GB storage to have off the heap per JVM, let’s say it is 40GB. Hazelcast will create 40 DirectBuffers, each with 1GB capacity. If you have, say 50 nodes, then you have total of 2TB off-heap storage capacity. Each buffer is divided into configurable chunks (blocks) (default chunk-size is 1KB). Hazelcast uses a queue of available (writable) blocks. 3KB value, for example, will be stored into 3 blocks. When the value is removed, these blocks are returned back into the available blocks queue so that they can be reused to store another value.

2. Distributed Backups

In dynamically scalable partitioned storage systems, whether it is a NoSQL database, filesystem or in-memory data grid, changes in the cluster (adding or removing a node) can lead to big data moves in the network to re-balance the cluster. Re-balancing will be needed for both primary and backup data on those nodes. If a node crashes for example, dead node’s data has to be re-owned (become primary) by other node(s) and also its backup has to be taken immediately to be fail-safe again. Shuffling MBs of data around has a negative effect in the cluster as it consumes your valuable resources such as network, CPU and RAM. It might also lead to higher latency of your operations during that period.

Imagine a cluster of 50 machines, each holding 40GB data; 20GB primary and 20GB backup. When a node, say node5, crashes, obviously we should make sure that primary (owned) data by the dead node5 has to be owned by at least one other node in the cluster. So 20GB will be re-owned and its backup will be taken. But where was that 20GB backed up before? Let’s say it was backed up on node7 before, then node7 will be the new owner of that 20GB. This means node7 owns currently two times the other nodes. which also means, node7 will burn more CPUs than others to handle twice as much requests. In addition, node7 will have to backup this 20GB on another node, say node9. This means node7 will send 20GB to node9. This is really bad for the network and for the CPUs of these two nodes. Also node9 will need enough memory space to store 20GB backup-data! It is huge memory waste. Such limitations will force you allocate (buy) more memory so that you don’t go out of memory upon crashes. Say you have enough memory, to balance the load, node7 and node9 will have to migrate its data onto other nodes overtime.

2.0 is designed to tackle all these big-data challenges. With this version, data owned by a node will be evenly backed up by all the other nodes. In other words, every node takes equal responsibility to backup every other node. This leads to better memory usage and less influence in the cluster when you add/remove nodes. Say you have 2TB data on 50 node cluster; each node storing 20GB primary, 20GB backup data. Let’s focus on one of the nodes, say node3. 20GB primary data that node3 has will be backed up by all 49 other nodes each backing up 1/49th of 20GB. If node3 dies, each node will own 1/49th of its data; notice that no migration is needed and cluster is still well-balanced! So backup mechanism is designed in a way that there will no need to rebalance after crashes. Say you added 5 more nodes. There is no immediate action to be taken by the cluster; because existing nodes are already in proper state. Only thing Hazelcast will do is to slowly migrate some of the data to the new nodes so that all nodes can eventually be equally loaded.

Hazelcast team is working on a demo to show this. Plan is to store 1 billion entries, total of 4TB data on 100 nodes at Amazon EC2. Recorded video of the demo will be posted online so that we all can see it in action.

Hazelcast 2.0 is coming! What is new?

hazelcast — Mon, 20 Feb 2012 16:59:58 +0000

Hazelcast 2.0 is a huge step forward in building the best IMDG and making Hazelcast experience even more pleasant.

As always this release contains many fixes, enhancements and improvements. But there are different reasons that make 2.0 very special. We have many big changes in the internals of Hazelcast. Many of these changes are made to prepare Hazelcast for the era of “BigData In-Memory”!

Distributed Backups:
Do you remember; the next member is your backup… thing? Now it is time to forget that. With 2.0 the backups are spanned into the cluster evenly. We call this, distributed backup implementation. Data owned by a member will be evenly backed up by all the other members. In other word, every member takes equal responsibility to backup every other node. This leads to better memory usage and less influence in the cluster when you add/remove nodes. The new backup system makes it possible to form backup-groups so that you can have backups and owners fall into different groups. We will have another blog entry explaining this in detail.

Parallel IO:
Prior to 2.0, regardless of the cluster size, each member had one -In- and one -Out- Thread to handle the communication to other members using NIO channels. With the new Parallel IO implementation we have combined In and Out thread into a single IO thread. But now it is possible to have many IO threads, each handling communication to different set of members. So in a 100 node cluster each member may have 10 IO threads and each thread dealing with roughly 10 members. This is great if you have many cores on the node. The motivation behind Parallel IO is to utilize more CPU, more network and achieve lower latency. We made the number of IO threads configurable so you can have one or much more depending on your environment.

Connection Management:
Hazelcast 2.0 is more tolerant to connection failures. On connection failure it tries to repair it before declaring the member as dead. So now it is ok to have short socket disconnections… No problem if your virtual server migrates to a new host.

Adding Listeners and Indexes via Configuration:

You were able to add Map/Queue/Topic/Membership/Migration listeners after Hazelcast instance is started. So you were missing all events happened before you add your listener(s). With 2.0, you can declare your listeners in configuration. You can also add indexes for your maps via configuration so that

1. You don’t have to have IMap.addIndex in your code anymore.

2. If you have a MapLoader to pre-populate your map, all these pre-populated entries can be indexed.

New Event Objects:
Event Listeners for Queue/List/Set/Topic were delivering the item itself on event methods. That’s why the items had to be deserialized by Hazelcast Threads before invoking the listeners. Sometimes this was causing class loader problems too. With 2.0, we have introduced new event containers for Queue/List/Set and Topic just like Map has EntryEvent. The new listeners now receive ItemEvent and Message objects respectively. The actual items are deserialized only if you call the appropriate get method on the event objects. This is where we brake the compatibility with the older versions of Hazelcast. Sorry for any inconvenience this many cause but in the long-term, we all win.

Client Config:
We had tons of factory methods to instantiate a HazelcastClient instance. 2.0 RC2 contains a ClientConfig API to get rid of all those factory methods: HazelcastClient.newHazelcastClient(ClientConfig). Nice and clean.

SSL Support:
Hazelcast was able to encrypt the communication over socket both with symmetric and asymmetric keys. Now it supports SSL communication also.

Other updates:

Distributed MultiMap value collection can be either List or Set.
SuperClient is renamed to LiteMember to avoid confusion. Be careful! It is a member, not a client.
New IMap.set (key, value, ttl, TimeUnit) implementation, which is optimized put(key, value) operation as set doesn’t return the old value.
HazelcastInstance.getLifecycleService().kill() will forcefully kill the node. Useful for testing.
forceUnlock, to unlock the locked entry from any Node and any thread regardless of the owner.
Enum type query support.. new SqlPredicate (“level = Level.WARNING”) for example

Issues:
Up to now the following issues have been fixed. We are trying to fix more until the final .
430, 459, 471, 567, 574, 629, 632, 646, 666, 686, 669, 690, 692, 693, 695, 698, 705, 710, 711, 712, 713, 714, 715, 719 , 721, 722, 724, 727, 728, 729, 730, 731, 732, 733, 738, 740, 747, 751, 756, 758, 759, 761, 765, 767, 770, 773, 779, 781, 783, 790

Documentation is not up to date but will be completed until the final. The ETA for 2.0 final is end of February. Go and grab a fresh RC and start Hazelcasting 2.0!

Hazelcast USA Trip, EastBayJUG and Coffee

hazelcast — Wed, 30 Nov 2011 14:13:02 +0000

Fuad and I will be in USA. Our schedule is below. I am sure we can
squeeze a coffee break somewhere if you are near-by and would like to
meet. Please drop an email (talip@hazelcast.com).

We will also speak at SF EastBayJUG on 13th. Here is the link to sign
up: http://www.meetup.com/eastbayjug/events/42096972/

Dec 4-6 Atlanta, GA
Dec 7-8 Nashville, TN
Dec 9-10 Los Angeles
Dec 11-13 San Francisco
Dec 14-18 New York

Hazelcast 1.9.4 is released!

Talip Ozturk — Tue, 06 Sep 2011 17:20:37 +0000

1.9.4 is now final. We all have been waiting for this release long enough. Now it is time to enjoy it!

Other news:

Our new product: Hazelcast Elastic Memory: Off-heap memory storage to avoid GC pauses.
New features are added into our Management Center product such as map config change in-runtime and TimeTravel.

We will keep putting a lot of work into Elastic Memory and Management Center while making Hazelcast even stronger, faster and easier.

1.9.4 Release Notes:

New WAN Replication (synchronization of separate active clusters).
New Data Affinity (co-location of related data and computation) feature.
New EC2 Auto Discovery for your Hazelcast cluster running on Amazon EC2 platform.
New Distributed Semaphore implementation.
New Distributed CountDownLatch implementation.
Improvement: Distribution contains HTML and PDF documentation besides Javadoc
Improvement: Better TCP/IP and multicast join support.
Improvement: Memcache protocol: Better integration between Java and Memcache clients.Put from memcache, get from Java client.
200+ commits 25+ bug fixes and several other enhancements

Hazelcast Elastic Memory

Talip Ozturk — Fri, 02 Sep 2011 17:38:52 +0000

We made pretty good progress in our new product, Hazelcast Elastic Memory: Hazelcast with off-heap memory storage to avoid GC pauses. We have been working on it for months and we are already quite impressed with the results. Even if you have terabytes of cache in-memory with lots of updates, GC will have almost no effect; resulting in more predictable latency and throughput. Our test results will be published here from now on. So stay tuned.

Elastic Memory is an add-on to the existing Hazelcast but will only be part of Hazelcast Enterprise Edition (EE), our new commercial offering.

Elastic Memory is currently in alpha stage. If you want to test it before everybody else, just let us know via email (info@hazelcast.com).

Exciting times ahead!