high app cpu and db pool count caused by stuck thread of forum delete

UMich recently has a server restart because of high CPU usage and high db pool count.

In our thread dump file, we found hundreds of threads (>600) associated with forum delete (see attached).

In the app log file, there are multiple SQLNestedException for the permission checks, which was issued from the forum delete action.

2012-09-30 19:10:53,138 [TP-Processor235] WARN org.sakaiproject.db.impl.BasicSqlService - Sql.dbRead: sql: select SRR.ROLE_NAME from SAKAI_REALM_RL_GR SRRG inner join SAKAI_REALM SR on SRRG.REALM_KEY = SR.REALM_KEY inner join SAKAI_REALM_ROLE SRR on SRRG.ROLE_KEY = SRR.ROLE_KEY where SR.REALM_ID = ? and SRRG.USER_ID = ? and SRRG.ACTIVE = '1' /site/2588f17d-5d1f-40e8-b357-dc9a968c80d8 bc5f3970-742f-4726-a2c7-c6c9d3e29558
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

We are still trying to decide why the forum action ends up in hundreds thread.

We are running in our own patched forum branch based on v3.0: -r109162 https://source.sakaiproject.org/svn/msub/umich.edu/ctools/PATCHED/msgcntr-3.0/branches/CTools-2.7.2