Ray Davis - 12-Dec-2007 11:55 Another possible feature is some way to create and include test-specific components _before_ a test runs. Writing tests with "mock components" is currently a two-step process when using Test Harness: first I build the integration-test module with testing turned off (so that the component artifacts will get made without waiting for tests to run), then I do a "mvn clean test". No big deal, but it would nicer as a single-step process.

Aaron Zeckoski - 18-Dec-2007 10:39 Just to have it in the public record, I fully support this work and view it as critical to having the ability to produce a stable Sakai release. Being able to run the tests in maven/eclipse would be a great addition to the functionality of test-runner. I think it might be best to put the final work into the test-harness project but that is just my thought on it.

Ray Davis - 05-Jan-2008 06:56 REQUIREMENTS (add more if you have them) 1) Run as a normal unit test integrated with Maven's "test" goal, acting as if the test code was in a deployed Sakai web application calling component service methods. 2) Optionally be runnable inside a deployed component while Tomcat is running. 3) Optionally be runnable inside a deployed web application while Tomcat is running. 4) Allow building and "deployment" of test-specific components before the test is run. Undo "deployment" at end of test. 5) Allow optional test-specific configurations of the component manager. 6) Simple configuration of dependencies. 7) Autowiring of component bean singletons. 8) Easy control over whether the component manager should stay running between tests, should shut down and restart between tests, or should be completely unloaded from memory (clearing out static variables) between tests.

Ray Davis - 05-Jan-2008 06:57 Goals 2 and 3 should be meetable using the Maven goals of Aaron's TestRunner. Goal 4 could be met by making the "component" and "shared" paths loaded by Josh's TestHarness configurable.

Ray Davis - 22-Feb-2008 06:54 Since it seems likely that this will be built on top of the current "testrunner" code, another requirement comes to mind: 9) It should be possible to run tests while keeping test libraries (notably a particular version of JUnit) out of deployed builds. Because testrunner is currently aimed at Tomcat-initiated tests running inside a real Sakai deployment, it installs the test drivers and JUnit in CATALINA_HOME/shared/lib. This won't be necessary for automated and command-line testing, and so the current module packaging will have to be restructured a bit to add deployment options (probably via Maven profiles).

Jon Gorrono - 22-Feb-2008 13:18 Regarding profiles: what should be the default active profile's behavior... deploy to shared or not?

Ray Davis - 22-Feb-2008 13:37 IMO, the default should be to not deploy any test material to the site. "First, do no harm" and all that. But we can always revisit the issue later if that makes it too difficult for the in-server testers.

Aaron Zeckoski - 22-Feb-2008 13:46 Good question. It definitely will not work if you don't deploy the stuff into shared right now (and probably ever given the way junit has no APIs and we use it's classes in our API). That said, it cannot even try to run without junit in shared so I am not sure how you would want to have it setup.

Ray Davis - 22-Feb-2008 14:09 "cannot even try to run without junit in shared" -- Sure it can, same as the test-harness runs: by Maven (or other test running code) putting dependent libraries into the classpath. JUnit only needs to be in Tomcat's "shared/lib" if unit tests are being run inside Tomcat. integration-test : command line with programmatic control over the classpaths integration-test-component : run inside a deployed Sakai component WAR (with classpaths determined by Tomcat) integration-test-webapp: run inside a deployed web application WAR (with classpaths determined by Tomcat)

Ray Davis - 22-Feb-2008 14:15 I should confess that I'm not sure profiles are even needed. That was one likely mechanism that came to mind, but we might get the desired results by test-component and test-webapp POMs specifying different dependencies from a command-line integration test POM. So I should stop kibbitzing now. :)

Jon Gorrono - 30-Mar-2008 20:00 I created a page in the Sakai-dev space where we can all kibbitz: http://confluence.sakaiproject.org/confluence/x/OgAlAQ

Ray Davis - 05-Jun-2008 16:21 I was pulled off this yet again to work on a Gradebook conversion issue. As part of that job, I finished a bit of tidying I've wanted to do for a while: breaking the component container emulation out of test-harness's JUnit subclass and into a utility class of its own. I checked that into trunk at revision: 47237. As soon as I'm able to get back to Jon's branch, I'll merge the change in there.

Ray Davis - 08-Jul-2008 11:21 Added BOF notes from Paris: http://confluence.sakaiproject.org/confluence/display/CONF09/Requirements+for+test-harness

Ray Davis - 11-Jul-2008 14:04 I've added some small improvements to the test-harness and ComponentContainerEmulator before tackling the harder stuff. Revision: 47280 - Improve exception handling and allow access to top application context Revision: 47577 - Fix CATALINA_HOME handling Revision: 47601 - Default sakai.home correctly Revision: 48623 - Eliminate Eclipse warnings Revision: 48635 - Make it easier to pass relevant system properties to integration tests; update user integration tests to reflect improvements (and possible defaults for master POM) Revision: 48649 - Make it easier to use test-specific configurations; update integration tests to reflect improvements One logical next step is to get a default configuration for test-harness into the master POM so that developers don't have to go searching for the best one to copy-and-paste. That gets us into the question of goal and phase, though, since these defaults aren't applicable to non-integration tests or to integration tests that don't use the test-harness (e.g., HttpUnit or Selenium front-to-back functional tests). I guess as a temporary stopgap I'll just document the recommendation in the README or on a Confluence page....

Ray Davis - 11-Jul-2008 15:51 Revision: 48651 - Updated documentation and let Eclipse fall back to TEST_CATALINA_HOME or CATALINA_HOME environment variables since it doesn't have direct access to the Maven system variables

Ray Davis - 17-Jul-2008 13:34 - edited Revision: 48793 - Added ability to autowire Sakai services (a la AbstractDependencyInjectionSpringContextTests) while using the test-harness. As a side-effect, I also finally added integration tests for the test-harness itself. For a quick idea of the outcome, see: https://source.sakaiproject.org/svn/test-harness/trunk/integration-test/src/test/java/org/sakaiproject/test/ChildContextDependencyInjectionTest.java

Ray Davis - 19-Jul-2008 08:51 After meeting with Jon and doing some tests, I've confirmed that using the "integration-test" phase doesn't completely clear up the task of handling test-tailored components. The difficulty lies in Maven's and Eclipse's understanding of project classloaders. In other words, given this sort of project layout: proj-itest/src/main/java/..../TestTailoredProviderImpl.java proj-itest/src/test/java/.../SomeDependentServiceTest.java We can easily set up our pom.xml so that when we say "mvn clean integration-test", a "sakai:deploy" of the test-tailored component is done before the tests run. But when the tests ask the component manager for the provider service, the TestTailoredProviderImpl that runs will be fully visible to the test's classloader (rather than hidden away inside the component's classloader) and will be the class that's local to the project rather than the one that's deployed. Neither Maven nor Eclipse conceive of blocking access to a project's classes from the tests which are *part* of the project. (Which is pretty understandable, actually.) This is only a problem in special circumstances. Most service-level integration testing doesn't have to worry about tailoring components specifically for the tests themselves -- I just happen to work on two of the exceptions, the User Directory Service and the Component Manager. And even then the classloader fibbing isn't *always* a problem. But *sometimes* it will be.

Ray Davis - 11-Sep-2008 10:07 Upcoming tasks (possibly) for 2.6: * Change documentation to reflect the tests that were moved as part of the K1 transition. * See what's up with the SAK-14063 patch. * Promote a standard Maven profile name to trigger longer-running tests (including most integration tests), such as "all-tests". * Add a utility class to centralize some frequently needed (and frequently more-or-less copied-and-pasted) emulation logic (act as user, pretend to be in a site context). I reverted my first try because it introduced "surprise" dependencies into test writers' projects. But now that all the relevant service APIs are in one Kernel project, it should be viable.

Ray Davis - 11-Sep-2008 10:08 I see a possible way to get Test-Runner-level classloading realism for tests run from Maven or Eclipse. But it'll likely require a new subclass of the Sakai component manager and a new Maven plugin, and so there's not much chance of getting it done before the 2.6 freeze. The end scenario would go: - Build (but do not necessarily deploy) a TestRunner-style component which embeds the current project's integration test suite. - Before starting Sakai, point the component manager at the built target as an extra component to load. (This is where the Component Manager subclass comes in -- Ian's doing something similar in Kernel 1.) - Have Maven run the specified test-component's standard test suite through a standard TestRunner-style API. (This is where a new plugin is likely, since I don't yet see a way to keep the Maven-built component classes out of the Surefire classpath.) This should give the same end result as TestRunner's current deployment of JUnit and the TestRunner to Tomcat "shared" and test-embedding components to Tomcat "components", except for having to run Tomcat.

Ray Davis - 12-Sep-2008 14:55 As of rev 52204, documentation has been updated, SAK-14063 has been applied, and I'm trying out "mvn -DskipLongTests=false test" as a standard way to trigger longer-running tests.