ConcurrentModificationException occurs under load in DefaultScriptSessionManager.checkTimeouts()

Description

While the changes in 2.0.7 (see DWR-60) helped address the deadlocks we were seeing in 2.0.6 we are now experiencing CME when our server load reaches over 1200 users.

Here is the stacktrace we are seeing:

java.util.ConcurrentModificationException
at java.util.HashMap$AbstractMapIterator.checkConcurrentMod(HashMap.java:122)
at java.util.HashMap$AbstractMapIterator.makeNext(HashMap.java:127)
at java.util.HashMap$ValueIterator.next(HashMap.java:212)
at org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:129)
at org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)
at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:196)

Activity

Show:
David Marginian
March 23, 2012, 11:12 AM
Edited

For the thread dump, are you manually capturing it? Is the "DEADLOCK DETECTED:" string part of the dump or something you added? From my initial analysis invalidate() and setPageForScriptSession() both lock on pageSessionMap but either is holding another lock at the time so we don't have a lock-ordering Deadlock. Have you tried capturing another thread dump a few minutes later? Do you see the same threads blocked?

Rob Moore
March 23, 2012, 11:55 AM

We manually captured this dump – I believe it was from the JConsole. I'll have to try repeated dumps in future but I can tell you once we experience this error performance really suffers to the point the application is unusable.

David Marginian
March 23, 2012, 12:09 PM

Have you looked at our source and come to the same conclusion that I have? I followed all of the threads execution flow based on the provided thread dump and I don't see any of them holding a lock when they enter invalidate or setPageForScriptSession. This leaves me thinking that this is not an issue with DWR. Are you sure your server can handle the load you are pushing to it?

Djokic Marjan
October 29, 2012, 3:23 AM

Hi,
I see this issue is closed but I experience the same problem with version 2.0.9 and under high load (10-15 requests/sec).

Here is part of thread dump log:

"TP-Processor3166" daemon prio=10 tid=0x00002aaabc693000 nid=0x39ef waiting for monitor entry [0x00000000677f7000]      java.lang.Thread.State: BLOCKED (on object monitor)      at org.directwebremoting.impl.DefaultScriptSessionManager.invalidate(DefaultScriptSessionManager.java:136)      - waiting to lock <0x00000006b7a2add8> (a java.util.Collections$SynchronizedMap)      at org.directwebremoting.impl.DefaultScriptSession.invalidate(DefaultScriptSession.java:109)      at org.directwebremoting.impl.DefaultScriptSessionManager.checkTimeouts(DefaultScriptSessionManager.java:200)

DWR is used to call methods which execution lasts not more than 30sec.

Can somebody confirm that this problem is solved in version 2.0.9?

David Marginian
October 29, 2012, 5:22 AM

Marjan, to avoid confusion this issue is for a Concurrent Modification exception. You appear to be having a separate issue.

Assignee

David Marginian

Reporter

Rob Moore

Labels

None

Documentation Required

No

Components

Fix versions

Affects versions

Priority

Normal
Configure