Possible issue with static log4j AgentFacade NoClassDefFoundError

We're lately seeing very random/rare problems with Samigo specifically where after a random period of time (usually a few days) it stops working. Usually the reports are that some user is just using the tool and it stops for the entire instance resulting in a reboot. It's not regularly reproducible and generally can take well over a week to come up.

The stack trace is a NoClassDefFoundError on a class (currently always AgentFacade) that is present and works fine all of the rest of the time. AgentFacade is in shared/lib/samigo-services.jar.

In researching there's the likely problem is that it has a problem initializing static variables or blocks. I don't see any static blocks but it has 3 static variables.

private static Log log = LogFactory.getLog(AgentFacade.class);
private static final AgentHelper helper = IntegrationContextFactory.getInstance().getAgentHelper();
private static final boolean integrated = IntegrationContextFactory.getInstance().isIntegrated();

I saw this bit of text on the log4j page that says that having a static log4j object is more efficient, but it's not safe for objects deployed in a shared classloader since you can have issues with threading. So that's the likely suspect for me here. http://commons.apache.org/proper/commons-logging/guide.html#Developing_With_JCL

There are other classes that have a static Log in shared (samigo) so I'll probably just remove them all.

I think all instances of log4j Log/LogFactory in classes in shared should not be static as per the recommendation. While it might not eliminate this problem, I don't see any real negatives.

In the rest of Sakai, log4j is occasionally static and occasionally not. But I haven't seen this issue anywhere other than Samigo in production.

Stacktrace:
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:455)
caused by: javax.servlet.ServletException: java.lang.NoClassDefFoundError: Could not initialize class org.sakaiproject.tool.assessment.facade.AgentFacade
at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:845)
caused by: java.lang.NoClassDefFoundError: Could not initialize class org.sakaiproject.tool.assessment.facade.AgentFacade
at org.sakaiproject.tool.assessment.facade.PublishedAssessmentFacadeQueries.getBasicInfoOfAllPublishedAssessments2(PublishedAssessmentFacadeQueries.java:1547)