Java, Integration and the virtues of source

Have you ever used java.util.concurrent.CountDownLatch ??

It’s a very convenience class to achieve synchronization between two or more threads, where allows one or more threads to wait until a set of operations being performed in other threads completes (check javadoc and this post). CountDownLatch can save your time in suitable cases and you have to be aware of this class.

As always synchronization of threads can raise deadlocks if code is not good. And as concurrency use cases can be very complex, developers must be very careful. I will not describe here a complex concurrency problem, but a simple problem that you may face it if you use CountDownLatch careless.

Assume you have 2 threads (Thread-1 and Thread-2) that share a single java.util.concurrent.ArrayBlockingQueue and you want to synchronize them using a CountDownLatch. Check this simple example:

package gr.qiozas.simple.threads.countdownlatch;

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.CountDownLatch;

public class DeadlockCaseCDL {

    public static void main(String[] args) throws InterruptedException {
		CountDownLatch c = new CountDownLatch(1);
		ArrayBlockingQueue b = new ArrayBlockingQueue(1);

		new Thread(new T1(c, b)).start();
		new Thread(new T2(c, b)).start();
    }

    private static class T1 implements Runnable {
        private CountDownLatch c;
        private ArrayBlockingQueue b;
        private T1(CountDownLatch c, ArrayBlockingQueue b) {
            this.c = c; this.b = b;
        }
        public void run() {
		  try {
		    b.put(234);
		    b.put(654);
		    doWork(T1.class);
		    c.countDown();
		    doWork(T1.class);
		  } catch (InterruptedException ex) {}
	   }
    }

    private static class T2 implements Runnable {
        private CountDownLatch c;
        private ArrayBlockingQueue b;
        private T2(CountDownLatch c, ArrayBlockingQueue b) {
            this.c = c; this.b = b;
        }
        public void run() {
		  try {
		    doWork(T2.class);
		    c.await();
		    doWork(T2.class);
		    System.out.println("I just dequeue "+b.take());
		    System.out.println("I just dequeue "+b.take());
		  } catch (InterruptedException ex) {}
	   }
    }

    private static void doWork(Class classz) {
        System.out.println(classz.getName()+" do the work");
    }
}

What you see above is that main thread creates a CountDownLatch with count “1” and an ArrayBlockingQueue with capacity “1” and afterwards spawns the “2 threads”. The ArrayBlockingQueue will be used for the real “work” (enqueue and dequeue) and the CountDownLatch will be used to synchronize the threads (enqueue must be done before dequeue).

Thread-1 enqueues 2 messages and Thread-2 wants to dequeue them, but only after Thread-1 has enqueued both messages. This synchronization is guaranteed by CountDownLatch. Do you believe that this code is OK??
No, it is not as causes a deadlock!!!

If you see carefully line 10, you will see that I initialize ArrayBlockingQueue with capacity equal to “1”. But Thread-1 wants to enqueue 2 messages and then release the lock (of CountDownLatch), in order to be consumed afterwards by Thread-2. The capacity “1” of queue blocks Thread-1 until another thread dequeue one message from queue, and then tries again to enqueue the 2nd message. Unfortunately, only Thread-2 dequeues messages from queue, but because Thread-1 hold the lock of CountDownLatch, the Thread-2 cannot dequeue any message and so it blocks. So, we really have a deadlock as both threads are blocked (waiting to acquire different locks). Thread-1 waits for ArrayBlockingQueue lock and Thread-2 for CountDownLatch lock (you can see it also in the related Thread Dump below).

If we increase the capacity of the queue then this code will run without problems, but this is not the point of this article. What you have to understand is that CountDownLatch must be used with care, in order to avoid such dangerous cases. You have to know the functional cases of your class, elaborate to other developers of team for this functionality, write useful javadoc and always write code that is robust in extreme cases, not only for happy paths.

Another point that you may help you is that this deadlock is not detected by modern JVMs. Try it.

As you may know, modern JVMs (both Hotspot and JRockit) are able to detect simple deadlocks and report them on Thread Dump. See a simple deadlock example that detected from Hotspot JVM:

Found one Java-level deadlock:
=============================
"Thread-6":
waiting to lock monitor 0x00a891ec (object 0x06c616e0, a java.lang.String),
which is held by "Thread-9"
"Thread-9":
waiting to lock monitor 0x00a8950c (object 0x06c61708, a java.lang.String),
which is held by "Thread-6"

You can try DeadlockCaseCDL and get a Thread Dump (on GNU/Linux run just “kill -3 ‹jvm_pid›”). You will see that thread dump looks normal and no deadlock is detected by JVM, but you are on a deadlock!!! So, be aware that this kind of deadlock is not detected by JVM.

Check this Thread Dump example from my local execution:

Full thread dump Java HotSpot(TM) Server VM (17.1-b03 mixed mode):

"DestroyJavaVM" prio=10 tid=0x0946e800 nid=0x5382 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Thread-1" prio=10 tid=0x094b1400 nid=0x5393 waiting on condition [0x7c79a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for   (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
	at gr.qiozas.simple.threads.countdownlatch.DeadlockCaseCDL$T2.run(DeadlockCaseCDL.java:50)
	at java.lang.Thread.run(Thread.java:662)

"Thread-0" prio=10 tid=0x094afc00 nid=0x5392 waiting on condition [0x7c7eb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for   (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
	at gr.qiozas.simple.threads.countdownlatch.DeadlockCaseCDL$T1.run(DeadlockCaseCDL.java:29)
	at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x0947f800 nid=0x5390 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x7cfa9000 nid=0x538f waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x7cfa7000 nid=0x538e waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x7cfa5800 nid=0x538d waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x7cf96000 nid=0x538c in Object.wait() [0x7cd15000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on  (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked  (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x7cf94800 nid=0x538b in Object.wait() [0x7cd66000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on  (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked  (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x7cf92000 nid=0x538a runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x09475c00 nid=0x5383 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x09477000 nid=0x5384 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x09478800 nid=0x5385 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0947a000 nid=0x5387 runnable

"VM Periodic Task Thread" prio=10 tid=0x09489800 nid=0x5391 waiting on condition

JNI global references: 854

Heap
 PSYoungGen      total 14976K, used 1029K [0xa2dd0000, 0xa3e80000, 0xb39d0000)
  eden space 12864K, 8% used [0xa2dd0000,0xa2ed1530,0xa3a60000)
  from space 2112K, 0% used [0xa3c70000,0xa3c70000,0xa3e80000)
  to   space 2112K, 0% used [0xa3a60000,0xa3a60000,0xa3c70000)
 PSOldGen        total 34304K, used 0K [0x815d0000, 0x83750000, 0xa2dd0000)
  object space 34304K, 0% used [0x815d0000,0x815d0000,0x83750000)
 PSPermGen       total 16384K, used 1739K [0x7d5d0000, 0x7e5d0000, 0x815d0000)
  object space 16384K, 10% used [0x7d5d0000,0x7d782e90,0x7e5d0000)

Regards,
Adrianos Dadis.
—
Democracy requires Free Software

Performance is very important in high load multithreading applications. Developers must be aware of concurrency issues in order to achieve better performance. When we need concurrency we usually have a resource that must be shared by two or more threads. In such cases we have a race condition, where only one of the threads will acquire the lock (on resource) and all the other threads that want the lock will block. This synchronization feature does not come for free; Both JVM and OS consume resources in order to provide you with a valid concurrency model. The three most fundamental factors that makes concurrency implementation resource intensive are:

• Context switching
• Memory synchronization
• Blocking

In order to write optimized code for synchronization you have to be aware of these 3 factors and how to decrease them. There are many things that you must watch out when writing such code. In this article I will show you a technique to decrease these factors by reducing lock granularity.

Starting with the basic rule: Do not hold the lock longer than necessary.

Do whatever you need to do before acquire the lock, use the lock only to act on synchronised resource and release it immediately. See a simple example:

public class HelloSync {
	private Map dictionary = new HashMap();
	public synchronized void borringDeveloper(String key, String value) {
		long startTime = (new java.util.Date()).getTime();
		value = value + "_"+startTime;
		dictionary.put(key, value);
		System.out.println("I did this in "+((new java.util.Date()).getTime() - startTime)+" miliseconds");
	}
}

In this example we violate the basic rule, because we create two Date objects, call System.out.println(), and do many String concatenations. The only one action that needs synchronization is action: “dictionary.put(key, value);” Alter the code and move synchronization from method scope to this single line. A slightly better code is this:

public class HelloSync {
	private Map dictionary = new HashMap();
	public void borringDeveloper(String key, String value) {
		long startTime = (new java.util.Date()).getTime();
		value = value + "_"+startTime;
		synchronized (dictionary) {
			dictionary.put(key, value);
		}
		System.out.println("I did this in "+((new java.util.Date()).getTime() - startTime)+" miliseconds");
	}
}

Above code can written even better, but I just want to give you the idea. If wondering how check java.util.concurrent.ConcurrentHashMap.

So, how can we reduce lock granularity? With a short answer, by asking for locks as less as possible. The basic idea is to use separate locks to guard multiple independent state variables of a class, instead of having only one lock in class scope. Check this simple example that I have seen it in many applications.

public class Grocery {
	private final ArrayList fruits = new ArrayList();
	private final ArrayList vegetables = new ArrayList();
	public synchronized void addFruit(int index, String fruit) {
		fruits.add(index, fruit);
	}
	public synchronized void removeFruit(int index) {
		fruits.remove(index);
	}
	public synchronized void addVegetable(int index, String vegetable) {
		vegetables.add(index, vegetable);
	}
	public synchronized void removeVegetable(int index) {
		vegetables.remove(index);
	}
}

The grocery owner can add/remove fruits and vegetables in/from his grocery shop. This implementation of Grocery guards both fruits and vegetables using the base Grocery lock, as the synchronization is done on method scope. Instead of this fat lock, we can use two separate guards, one for each resource (fruits and vegetables). Check the improved code below.

public class Grocery {
	private final ArrayList fruits = new ArrayList();
	private final ArrayList vegetables = new ArrayList();
	public void addFruit(int index, String fruit) {
		synchronized(fruits) fruits.add(index, fruit);
	}
	public void removeFruit(int index) {
		synchronized(fruits) {fruits.remove(index);}
	}
	public void addVegetable(int index, String vegetable) {
		synchronized(vegetables) vegetables.add(index, vegetable);
	}
	public void removeVegetable(int index) {
		synchronized(vegetables) vegetables.remove(index);
	}
}

After using two guards (splitting the lock) we will see less locking traffic than the original fat lock would have. This technique works better when we apply it on locks that have medium lock contention. If we apply it on locks that have slight contention, then the gain is small, but still positive. If we apply it on locks that have heavy contention, then the result is not always better and you must be aware of this.

Please use this technique with conscience. If you suspect that this is a heavy contention lock then please follow these steps:

1. Confirm the traffic of your production requirements, multiple it by 3 or 5 (or even 10 even if you want to be prepared).
2. Run the appropriate tests on your testbed, based on the new traffic.
3. Compare both solutions and only then choose the most appropriate.

There are more techniques that can improve synchronization performance, but for all techniques the basic rule is one: Do not hold the lock longer than necessary.
This basic rule can be translated to “asking for locks as less as possible” as I have already explained you, or to other translations(solutions) which I will try to describe them in future articles.

Two more important advices:

• Be aware of classes in java.util.concurrent package (and subpackages) as there are very clever and useful implementations.
• Concurrency code most times can be minimized by using good design patterns. Always have in mind Enterprise Integration Patterns, they can save your nights.

Regards,
Adrianos Dadis.
—
Democracy requires Free Software