Posts by Mehmet Dogan

GO: Call me maybe, Java!

When you need to pass boundaries of managed java environment, you find yourself playing (or fighting :) with C/C++ code (headers, compiler flags, linker flags …).With version 1.5, Go is coming to save us, C/C++ is not your only option anymore…

Story of a Java 8 Compiler Bug (JDK-8064803)

Story begins with a cleanup done on Hazelcast IQueue interface by one of my colleagues. What’s done actually was just removing unneeded/redundant overridden methods those are already defined in java.util.Queue and java.util.concurrent.BlockingQueue interfaces. It was a very simple cleanup, but…

Full commit diff can be seen here: 2d9eabe82a0a0568d2c92c7d53a5383efb32cf7d#diff-fc20765c5b3aa6eafd6348c5a004447b

Then suddenly, we started to see build errors on our Jenkins CI compatibility builds which are using java 8. There was no compile errors but only a single test using a parameterized queue was failing with java.lang.NoSuchMethodError: com.hazelcast.core.IQueue.poll()Ljava/lang/String;.

Here is the failing test method which has nothing special:

@Test
public void testQueueItemListener() throws InterruptedException {
    final CountDownLatch latch = new CountDownLatch(8);
    final String value = "hello";
    final HazelcastInstance instance = createHazelcastInstance();
    IQueue<String> queue = instance.getQueue("testQueueItemListener");

    queue.addItemListener(new ItemListener<String>() {
        public void itemAdded(ItemEvent<String> itemEvent) {
            assertEquals(value, itemEvent.getItem());
            latch.countDown();
        }

        public void itemRemoved(ItemEvent<String> itemEvent) {
            assertEquals(value, itemEvent.getItem());
            latch.countDown();
        }
    }, true);

    queue.offer(value);
    assertEquals(value, queue.poll()); // <---- error is thrown here
    queue.offer(value);
    assertTrue(queue.remove(value));
    queue.add(value);
    assertEquals(value, queue.remove());
    queue.put(value);
    assertEquals(value, queue.take());

    assertTrue(latch.await(5, TimeUnit.SECONDS));
    assertTrue(queue.isEmpty());
}

My gut feeling was saying that, this is certainly a compiler bug. But I needed to prove that using a simple and easily reproducible test case. At the beginning I wrote a dummy implementation of IQueue interface and executed the same test against it. Result was the same failure again.

Then I decided to emulate the Hazelcast IQueue implementation using a few single abstract method interfaces. There were two parent interfaces ParentA and ParentB replacing java.util.Queue and java.util.concurrent.BlockingQueue. And the actual interface Child which is substitute of IQueue. And an empty implementation of Child interface:

public interface ParentA<T> {
    T process() throws Exception;
}

public interface ParentB<T> {
    T process() throws Exception;
}

public interface Child<T> extends ParentA<T>, ParentB<T> {
}

public class ChildImpl<T> implements Child<T> {
    @Override
    public T process() {
        return null;
    }
}

Test method was very simple, just instantiate a ChildImpl with String generic type by assigning it to a Child reference and call empty process() method:

public class Test {
    public static void main(String[] args) throws Exception {
        Child<String> child = new ChildImpl<String>();
        String result = child.process();
        System.err.println(result);
    }
}

Compiled these with javac 8 and ran the main method. Result was as expected: failure!

Exception in thread "main" java.lang.NoSuchMethodError: Child.process()Ljava/lang/String;

When I disassembled the generated bytecode using javap tool (javap -c Test.class), output was:

public class Test {
  public Test();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String[]) throws java.lang.Exception;
    Code:
       0: new           #2                  // class ChildImpl
       3: dup
       4: invokespecial #3                  // Method ChildImpl."<init>":()V
       7: astore_1
       8: aload_1
       9: invokeinterface #4,  1            // InterfaceMethod Child.process:()Ljava/lang/String;
      14: astore_2
      15: getstatic     #5                  // Field java/lang/System.err:Ljava/io/PrintStream;
      18: aload_2
      19: invokevirtual #6                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
      22: return
}

Code was trying to invoke an interface method which is returning a String, see this line

9: invokeinterface #4,  1   // InterfaceMethod Child.process:()Ljava/lang/String;

Problem was Child interface doesn’t have a process() method returning a String. It’s a generic method and because of type erasure it simply returns a plain Object. Compiler itself should add a cast instruction where needed.

Then I compiled the same interfaces/classes using javac 7. When I disassembled the Test.class again, it was invoking the righ interface method and checking returned type using checkcast bytecode:

public class Test {
  public Test();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String[]) throws java.lang.Exception;
    Code:
       0: new           #2                  // class failure/ChildImpl
       3: dup
       4: invokespecial #3                  // Method failure/ChildImpl."<init>":()V
       7: astore_1
       8: aload_1
       9: invokeinterface #4,  1            // InterfaceMethod failure/Child.process:()Ljava/lang/Object;
      14: checkcast     #5                  // class java/lang/String
      17: astore_2
      18: getstatic     #6                  // Field java/lang/System.err:Ljava/io/PrintStream;
      21: aload_2
      22: invokevirtual #7                  // Method java/io/PrintStream.println:(Ljava/lang/String;)V
      25: return
}

See these two lines:

9: invokeinterface #4,  1            // InterfaceMethod failure/Child.process:()Ljava/lang/Object;
14: checkcast     #5                  // class java/lang/String

It was known that everything was OK before we cleaned up overridden interface methods from IQueue interface. To verify that, I added an overriding method declaration to Child interface, which then became:

public interface Child<T> extends ParentA<T>, ParentB<T> {
    T process() throws Exception;
}

As I was expecting this version generated the correct bytecode and worked fine using java 8 too.

To prevent this issue happening for our java 8 users, we added back some of the overriding generic methods to the IQueue interface with commit 81581a2d11dfcc535c2549a222a8dbd054f8669d. Now our IQueue looks like:

public interface IQueue<E> extends BlockingQueue<E>, BaseQueue<E>, ICollection<E> {

    /*
     * Added poll(), poll(long timeout, TimeUnit unit) and take()
     * methods here to prevent wrong method return type issue when
     * compiled with java 8.
     *
     * For additional details see;
     *
     * http://mail.openjdk.java.net/pipermail/compiler-dev/2014-November/009139.html
     * https://bugs.openjdk.java.net/browse/JDK-8064803
     *
     */

    E poll();

    E poll(long timeout, TimeUnit unit) throws InterruptedException;

    E take() throws InterruptedException;

    /**
     * Returns LocalQueueStats for this queue.
     * LocalQueueStats is the statistics for the local portion of this
     * queue.
     *
     * @return this queue's local statistics.
     */
    LocalQueueStats getLocalQueueStats();
}

I also reported this issue to compiler-dev mailgroup. See http://mail.openjdk.java.net/pipermail/compiler-dev/2014-November/009139.html. My report was replied back by Maurizio Cimadamore promptly. It was definitely a javac bug and he filed an issue on openjdk issue tracker, JDK-8064803. Issue was solved in a few days for JDK-9 target but sadly this bug still exists in javac 8 as of version java 1.8.0_31.


*** This post was originally published at
http://blog.dogan.io/2015/03/02/java-8-compiler-bug/

Installing IBM JDK for newbies

Installing IBM JDK on a linux box is a trivial task or should be a trivial task. But when you get a message like

The installer cannot run on your configuration. It will now quit.

you find yourself staring at console without a clue.

When you search above not-really-helping message on your favourite search engine, you’ll see that most of the answers are just saying to run installer as root.

If that solves your issue, please don’t read after this point. We are done!

But… You already know how to install a package. You’re not a newbie linux user anymore, right?

After some more search I saw an IBM support ticket, which just says installer requires /etc/mailcap and /etc/mime.types
files and either install Mailcap package or create dummy /etc/mailcap and /etc/mime.types files.

Local fix

To make install operation successful either of the following
options can be applied.

  • Install mailcap RPM package before attempting the JDK
    installation.

  • Create dummy files /etc/mailcap and /etc/mime.types files
    manually before installing the JDK

Problem summary

The problem is caused when /etc/mailcap & /etc/mime.types file
are not present in the machine where the installer is being run.
These files are used to write the java web start entries during
jdk installation. The installer will fail silently if these
files are not present in the /etc directory of the system.


*** This post was originally published at
http://blog.dogan.io/2014/11/05/installing-ibm-jdk-for-newbies/

Don’t call us, Hazelcast will call you!

Sending tasks or application logic to the data is a good step to achieve greater scalability, generally you need to inject some local dependencies into remote tasks before they are executed. In Hazelcast, there are a few ways of injecting…

More than just JCache

JSR107, Java Temporary Caching API, also known as JCache is finally out after an effort of 13 years. It's like the only everlasting JSR of the JCP history. But long story short, it's completed now and companies in NoSQL/in-memory-data-grid/big-data world are competing each other to release their own implementations first.

Give me some GC stats…

While analysing application memory usage and inspecting allocation/garbage creation patterns, we generally need to know garbage collection count, time, rate etc. Although most of the profilers give this information out-of-the-box or JVM already has builtin flags to enable GC logging in many details, sometimes we want to access GC information programmatically.

Fortunately, there’s a formal/public API in JDK to gather some useful information: java.lang.management.GarbageCollectorMXBean.

GarbageCollectorMXBean gcMxBean = ...;

// name of the memory manager,
// aka name of the garbage collector
gcMxBean.getName();

// total collection count since process start
gcMxBean.getCollectionCount();

// total (approximate) collection elapsed time (ms)
gcMxBean.getCollectionTime()

Above code gives us only information about a garbage collector in system. But in a JVM there may be more than one garbage collector running for different memory regions, like eden space, tenured generation etc.

From this raw data, we need to gather some useful information. First we need to distinguish which memory manager belongs to which region. To be able to do this seperation, we need to know GC names in JVM. There are a few major JVM implementations, like Hotspot, JRockit or IBM J9. I’ll do this for 3 most known Hotspot GC types;

static final Set<String> YOUNG_GC = new HashSet<String>(3);
static final Set<String> OLD_GC = new HashSet<String>(3);

static {
    // young generation GC names
    YOUNG_GC.add("PS Scavenge");
    YOUNG_GC.add("ParNew");
    YOUNG_GC.add("G1 Young Generation");

    // old generation GC names
    OLD_GC.add("PS MarkSweep");
    OLD_GC.add("ConcurrentMarkSweep");
    OLD_GC.add("G1 Old Generation");
}

private static void printGCMetrics() {

    long minorCount = 0;
    long minorTime = 0;
    long majorCount = 0;
    long majorTime = 0;
    long unknownCount = 0;
    long unknownTime = 0;

    List<GarbageCollectorMXBean> mxBeans
        = ManagementFactory.getGarbageCollectorMXBeans();
    for (GarbageCollectorMXBean gc : mxBeans) {
        long count = gc.getCollectionCount();
        if (count >= 0) {
            if (YOUNG_GC.contains(gc.getName())) {
                minorCount += count;
                minorTime += gc.getCollectionTime();
            } else if (OLD_GC.contains(gc.getName())) {
                majorCount += count;
                majorTime += gc.getCollectionTime();
            } else {
                unknownCount += count;
                unknownTime += gc.getCollectionTime();
            }
        }
    }

    StringBuilder sb = new StringBuilder();
    sb.append("MinorGC -> Count: ").append(minorCount)
            .append(", Time (ms): ").append(minorTime)
            .append(", MajorGC -> Count: ").append(majorCount)
            .append(", Time (ms): ").append(majorTime);

    if (unknownCount > 0) {
        sb.append(", UnknownGC -> Count: ").append(unknownCount)
                .append(", Time (ms): ").append(unknownTime);
    }

    System.out.println(sb);
}

This is sample output from above method;

MinorGC -> Count: 3, Time (ms): 39, MajorGC -> Count: 0, Time (ms): 0

Now you need to find out why in the hell those precious CPU cycles are spent on doing GC.

Give me some GC stats…

While analysing application memory usage and inspecting allocation/garbage creation patterns, we generally need to know garbage collection count, time, rate etc. Although most of the profilers give this information out-of-the-box or JVM already has builtin flags to enable GC logging in many details, sometimes we want to access GC information programmatically.

How many bytes can I store here, sir?

Every operating system has its own special procedure to access memory information and statistics. In Linux one can read /proc filesystem, in nearly every OS one can fork a subprocess to execute a system command to find out memory information or call a system dependent API.

But since Java has the motto of write once run everywhere, how can one access memory information in Java?

Memory belonging to JVM process

Reading process memory usage and free process memory is already available using JDK public API. There are a few ways of achieving this;

  • Using java.lang.Runtime:
Runtime runtime = Runtime.getRuntime();
// maximum memory this JVM process can allocate
runtime.maxMemory();
// total allocated memory by JVM process
runtime.totalMemory();
// total free space in allocated memory
runtime.freeMemory();
  • Using java.lang.management.MemoryMXBean:
MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();

MemoryUsage heapMemoryUsage = memoryMXBean.getHeapMemoryUsage();
// total allocated *heap* memory by JVM process
heapMemoryUsage.getCommitted();
// total used *heap* memory
heapMemoryUsage.getUsed();
// maximum *heap* memory this JVM process can allocate
heapMemoryUsage.getMax();

// also there's another *MemoryUsage* bean to access
// non-heap memory allocated by JVM (See DirectByteBuffer)
MemoryUsage nonHeapMemoryUsage = memoryMXBean.getNonHeapMemoryUsage();
...

Physical And Swap Memory

There’s no public API to access physical memory information (at least I don’t know yet :). But you can guess that there should be an internal method hiding somewhere inside JDK. No need to go far, answer is in com.sun.management package.

com.sun.management.OperatingSystemMXBean bean has many useful methods to access physical and swap memory information (in contrast to not-very-useful java.lang.management.OperatingSystemMXBean).

com.sun.management.OperatingSystemMXBean operatingSystemMXBean
        = (com.sun.management.OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();

// total physical memory available
operatingSystemMXBean.getTotalPhysicalMemorySize();
// free physical memory available
operatingSystemMXBean.getFreePhysicalMemorySize();
// total swap space available
operatingSystemMXBean.getTotalSwapSpaceSize();
// free swap space available
operatingSystemMXBean.getFreeSwapSpaceSize();

Biggest problem in your head should be what if this class is not available in my environment. First solution pops out from our minds is using reflection. But there’s also a standard-ish way of accessing this data: querying MBeanServer

public static long getTotalPhysicalMemorySize() {
    return queryPhysicalMemory("TotalPhysicalMemorySize");
}

public static long getFreePhysicalMemorySize() {
    return queryPhysicalMemory("FreePhysicalMemorySize");
}

public static long getTotalSwapSpaceSize() {
    return queryPhysicalMemory("TotalSwapSpaceSize");
}

public static long getFreeSwapSpaceSize() {
    return queryPhysicalMemory("FreeSwapSpaceSize");
}

private static long queryMBeanServer(String type) {
    MBeanServer mBean = ManagementFactory.getPlatformMBeanServer();
    try {
        ObjectName name = new ObjectName("java.lang", "type",
            "OperatingSystem");
        Object attribute = mBean.getAttribute(name, type);
        if (attribute != null) {
            return Long.parseLong(attribute.toString());
        }
    } catch (Exception ignored) {
        // means this information is not available
    }
    return -1L;
}

Go fill all emty space! It’s yours!

Spurious wakeups are real!

Most probably you heard spurious wakeup term many times, you read it in many API docs. But have you actually seen it in the wild? Is it real?

Here is wikipedia definition:

Spurious wakeup describes a complication in the use of condition variables as provided by certain multithreading APIs such as POSIX Threads and the Windows API.

Even after a condition variable appears to have been signaled from a waiting thread’s point of view, the condition that was awaited may still be false.

Simply, it means a thread can wakeup from its waiting state without being signaled or interrupted or timing out. To make things correct, awakened thread has to verify the condition that should have caused the thread to be awakened. And it must continue waiting if the condition is not satisfied.

For example;

  • using Object.wait() / Object.notify():
final long timeoutMillis = someTimeoutValue;
final Object mutex = new Object();
volatile Object response;

// Thread-1
synchronized (mutex) {
    long remainingWait = timeoutMillis;
    long waitStart = System.currentTimeMillis();
    // loop with condition is guard against spurious wakeup
    while (response == null && remainingWait > 0) {
        mutex.wait(remainingWait);
        remainingWait = timeoutMillis - (System.currentTimeMillis() - waitStart);
    }
}

// Thread-2
synchronized (mutex) {
    response = value;
    mutex.notify();
}
  • using Condition.await() / Condition.signal():
final long timeoutMillis = someTimeoutValue;
final Lock lock = new ReentrantLock();
final Condition noResponse = lock.newCondition();
volatile Object response;

// Thread-1
lock.lock();
try {
    long remainingWait = timeoutMillis;
    long waitStart = System.currentTimeMillis();
    // loop with condition is guard against spurious wakeup
    while (response == null && remainingWait > 0) {
        noResponse.await(remainingWait, TimeUnit.MILLISECONDS);
        remainingWait = timeoutMillis - (System.currentTimeMillis() - waitStart);
    }
} finally {
    lock.unlock();
}

// Thread-2
lock.lock();
try {
    response = value;
    noResponse.signal();
} finally {
    lock.unlock();
}

Let’s go back to actual question; is sprurious wakeup a theoretical incident or is it practically real?

I have to say that, it’s (un)fortunately a real/living phenomenon that one must take care of. We (at Hazelcast) struggled/wrestled with an operation timeout bug for hours, days and weeks. And finally we found that the bug we were trying to hunt was actually just a spurious wakeup.

See OperationTimeoutException bug report and the patch fixing it.