[SAK-31608] Sakai has startup warnings on Tomcat 8.0.39+ Created: 02-Aug-2016  Updated: 02-Jun-2017  Resolved: 21-Feb-2017

Status: CLOSED
Project: Sakai
Component/s: Global
Affects Version/s: 11.0
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Matthew Buckett Assignee: Unassigned
Resolution: Incorporated Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Incorporate
is incorporated by SAK-25750 Upgrade Castor and cleanup related code CLOSED
Relate
relates to SAK-31823 Update Master POM Tomcat version to r... CLOSED
is related to SAK-32176 Upgrade to latest version of tomcat (... CLOSED

 Description   

With an unconfigured Tomcat 8.0.36 Sakai fails to startup.

It looks like Tomcat 8.0.39 starts up fine but it has a lot of warnings during startup unless conf files are modified.



 Comments   
Comment by Matthew Buckett [ 02-Aug-2016 ]

The stack traces look similar to this:

02-Aug-2016 11:19:23.170 WARNING [localhost-startStop-1] org.apache.tomcat.util.scan.StandardJarScanner.scan Failed to scan [file:/opt/tomcat-8.0.36-sakai11/lib/jndi_1.2.1.jar] from classloader hierarchy
 java.io.FileNotFoundException: /opt/tomcat-8.0.36-sakai11/lib/jndi_1.2.1.jar (No such file or directory)
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:219)
        at java.util.zip.ZipFile.<init>(ZipFile.java:149)
        at java.util.jar.JarFile.<init>(JarFile.java:166)
        at java.util.jar.JarFile.<init>(JarFile.java:130)
        at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:60)
        at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:43)
        at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:322)
        at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:272)
        at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262)
        at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:106)
        at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:103)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5292)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:940)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1816)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Comment by Matthew Buckett [ 02-Aug-2016 ]

The workaround for this is to add this to your tomcat configuration:

tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*.jar
Comment by Matthew Buckett [ 02-Aug-2016 ]

Matthew Jones mentioned that it's everything after 8.0.34 is broken without the workaround.

Comment by Matthew Jones [ 02-Aug-2016 ]

It was stated at https://groups.google.com/a/apereo.org/d/msg/sakai-dev/BqPb6qQKb7g/JGrr3G8yBQAJ that the "specific" jarsToSkip additions needed were:

tomcat.util.scan.StandardJarScanFilter.jarsToSkip=xerces-J_1.4.0.jar,jdbc-se2.0.jar,jndi_1.2.1.jar,jta1.0.1.jar,cglib-full-2.0.2.jar,commons-logging.jar

I'm not sure where these are coming from now:
I saw this issue but the response from Mark Thomas was to use the Users Mailing List: https://bz.apache.org/bugzilla/show_bug.cgi?id=59617

I didn't see anything on the user mailing list discussing this.

My best guess is it was broken by the change on https://bz.apache.org/bugzilla/show_bug.cgi?id=59226

I'm not sure if 8.0.37 will fix, this involves nested jar scanning whih seems like where these are picked up from? https://bz.apache.org/bugzilla/show_bug.cgi?id=59862

Comment by Seth Theriault [ 02-Aug-2016 ]

We previously recommended the use of the tomcat.util.scan.StandardJarScanFilter.jarsToSkip= setting in SAK-22183 and on the lists, although primarily for speed.

Comment by Matthew Jones [ 02-Aug-2016 ]

Yeah right, but I believe in Tomcat 8 it changed to needing to be configured in the context.xml (https://groups.google.com/a/apereo.org/d/msg/sakai-dev/cjtYGxd6hG0/6Zv3kUaxNsMJ) instead.

But having it skip all the jars was safe. It's just unfortunate right now that you have to configure anything in there or you get a startup error. Previously it was for an optional performance improvement, at the moment it looks like a requirement.

Comment by Hendrik Steller [ 11-Aug-2016 ]

During my first steps with Sakai 11, adding "*.jar" was NOT safe anymore.
Running with that setting broke at least some of the faces-based tools which would throw an Exception complaining about not being properly configured.
That happened with both Tomcat 8.0.36 and 8.5.4

Comment by Yuanhua Qu [ 21-Sep-2016 ]

Attempt to run on tomcat 8.0.37 with work around suggestion

tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*.jar

It does removed the ERRORs related to jar scanning, but still have this error

nested exception is java.lang.IncompatibleClassChangeError

Switching to nightly server version tomcat 8.0.32, server started cleanly without the workaround.

Comment by Reinier Post [ 26-Sep-2016 ]

So what is the present workaround, except downgrading to 8.0.32?

I'm using 8.0.37, setting tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*.jar causes Faces applications to fail.
Instead, I added the JARs that generate warnings to its existing value: xerces-J*jar,jdbc-se*.jar,jndi*.jar,cglib-full-*.jar. I don't know if this breaks anything, and it eliminates only eliminates 3 of the 4 warnings: I still get

26-Sep-2016 12:51:19.409 WARNING [localhost-startStop-1] org.apache.tomcat.util.scan.StandardJarScanner.scan Failed to scan file:/home/tomcat8s11/tomcat8/lib/xerces-J_1.4.0.jar from classloader hierarchy

What is the recommended way to go? Just leave the warnings be? (Sakai works as far as I can see.) I'm not a Sakai developer.

Comment by Earle R Nietzel [ 27-Sep-2016 ]

Generally the version that should be used with sakai is the version that comes from the master/pom.xml, currently it is:
<sakai.tomcat.version>8.0.20</sakai.tomcat.version>

Generally it is fine to use newer versions of tomcat but there can be issues like this one.

(this comment is to address the question of what tomcat version to use for end users)

Comment by Neal Caidin [ 30-Sep-2016 ]

I'm surprised to hear that our pom.xml is at 8.0.20 while our nightly QA servers are at 8.0.32. Ideally shouldn't they be the same for consistency's sake?

Comment by Matthew Jones [ 30-Sep-2016 ]

Neal Caidin Yeah they probably should, create a jira and update it. The only thing that depends on that version anymore with pack-demo removed is webdav.

Comment by Steve Swinsburg [ 03-Oct-2016 ]

Now that we have drag and drop in Resources, can we remove webdav from Sakai?

Comment by Earle R Nietzel [ 03-Oct-2016 ]

Yeah that was one of the original goals of adding drag n drop uploads was to eventually remove webdav.

We should make sure that there are no features webdav has that drag n drop doesn't first.

I would expect eventually that people will just want to connect their remote storage https://remotestorage.io/

Comment by Neal Caidin [ 03-Oct-2016 ]

The POM is updated - SAK-31823

Comment by Reinier Post [ 03-Oct-2016 ]

I use WevDAV as a Sakai admin to survey and manage our resources with a WebDAV drive mapping on Linux (i.e. I mount the resources as a directory). This is not really an advertised feature, as far as I'm aware, but it is very handy. E.g. version control on resources becomes easier this way.

Comment by Shawn Foster [ 03-Oct-2016 ]

Polling some of my colleagues for WebDAV use cases, I've heard about a few different uses of WebDAV that the drag-and-drop in Resources doesn't address, such as connecting via WebDAV to quickly rearrange or organize files and folders (that is more efficient than using the web interface); uploading a bunch of files with structure that didn't work well when extracting from an uploaded zipped file; and using the WebDAV connection as a mounted network drive to pull down files.

I think we need to redesign the Resources tool (as was proposed at Open Apereo 2016) to address these issues and use cases before we can remove WebDAV.

There might be a limited number of people at each institution that are using WebDAV, but those people probably have very specific or important use cases that require them to use WebDAV to accomplish the task.

Comment by Matthew Jones [ 22-Dec-2016 ]

So the webdav is an unrelated discussion here and I don't know how that got involved.

Sakai does startup in 8.0.39 but it starts up with ugly warnings at the start. So far there's 2 fixes I've found.

1) In conf/catalina.properties append xerces-J_1.4.0.jar,jdbc-se2.0.jar,jndi_1.2.1.jar,jta1.0.1.jar,cglib-full-2.0.2.jar,commons-logging.jar to the end of tomcat.util.scan.StandardJarScanFilter.jarsToSkip
2) Modify context.xml add

 <JarScanner scanClassPath="false"/>

FYI: If you have <JarScanFilter> still you should remove it as this causes errors.

But it would still seems like good practice if the StandardJarScanner checked if a file existed before looking it up. We can file a bug for them.

Comment by Matthew Jones [ 10-Feb-2017 ]

Adding a context.xml to the META-INF of every project fixes it. I feel like there's probably only a select group of projects that needs this, but I don't see any negative effects adding it everywhere. (Which is essentially what changing in in context.xml does)

Comment by Matthew Buckett [ 21-Feb-2017 ]

The problem of the missing JARs like:

 | 21-Feb-2017 17:43:20.351 WARNING [localhost-startStop-3] org.apache.tomcat.util.scan.StandardJarScanner.scan Failed to scan [file:/opt/tomcat/sakai-lib/cglib-full-2.0.2.jar] from classloader hierarchy
app_1  |  java.io.FileNotFoundException: /opt/tomcat/sakai-lib/cglib-full-2.0.2.jar (No such file or directory)
app_1  | 	at java.util.zip.ZipFile.open(Native Method)
app_1  | 	at java.util.zip.ZipFile.<init>(ZipFile.java:219)
app_1  | 	at java.util.zip.ZipFile.<init>(ZipFile.java:149)
app_1  | 	at java.util.jar.JarFile.<init>(JarFile.java:166)
app_1  | 	at java.util.jar.JarFile.<init>(JarFile.java:130)
app_1  | 	at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:60)
app_1  | 	at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:48)
app_1  | 	at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:334)
app_1  | 	at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:284)
app_1  | 	at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262)
app_1  | 	at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:106)
app_1  | 	at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:101)
app_1  | 	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5303)
app_1  | 	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
app_1  | 	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
app_1  | 	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
app_1  | 	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
app_1  | 	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:940)
app_1  | 	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1816)
app_1  | 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
app_1  | 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
app_1  | 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
app_1  | 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
app_1  | 	at java.lang.Thread.run(Thread.java:745)

is caused by castor-1.0.jar which has a META-INF/MANIFEST.MF of:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.6.3
Created-By: 1.4.2_08-b03 (Sun Microsystems Inc.)
Specification-Title: Castor XML, JDO, DAX, DSML
Specification-Vendor: Intalio Inc.
Specification-Version: 0.8
Implementation-Title: Castor
Implementation-Vendor: Intalio Inc.
Implementation-Version: 1.0
Class-Path: xerces-J_1.4.0.jar jdbc-se2.0.jar jndi_1.2.1.jar jta1.0.1.
 jar cglib-full-2.0.2.jar commons-logging.jar

Name: org/exolab/castor
Sealed: true

Comment by Matthew Buckett [ 21-Feb-2017 ]

On SAK-25750 it didn't look like anyone knew why castor was in shared.

Generated at Sun Sep 22 08:18:29 CDT 2019 using Jira 8.0.3#800011-sha1:073e8b433c2c0e389c609c14a045ffa7abaca10d.