Affects Version/s: None
Fix Version/s: None
Property addition/change required:Yes
This works builds on SAK-13324. For some background read, https://confluence.sakaiproject.org/display/TERRA/Sakai+Session+Clustering+Problems+and+Ideas?src=search
This work is an exploration of creating a rather light weight way to replicate sakai sessions across a cluster. Sakai has suffered from this problem for a long time. Without some type of session failover, upgrades can't occur without downtime, simple configuration changes require downtime, and when nodes fail users loose their sessions. In addition any attempt at elastic scaling is impossible, mainly because its impossible to remove nodes from a cluster without impacting users. (Adding nodes isn't such a big deal). Attempting to bleed users off nodes is difficult to orchestrate and takes way too long.
I have been dusting off the work Unicon started and settled on looking at a distributed cache to help with this. This idea has been looked at before but due to serialization issues and possible performance issues Terracotta was selected. Things have progressed since 2008. Things like memcached and redis are pretty commonly used for things like this. They are significantly faster than say a distributed ehcache layer was 5+ years ago. In addition, most of the client api's allow you to push data to the cache servers asynchronously to eliminate any session writing performance hit from the user.
In terms of serialization, Java serialization isn't the only option any more. New frameworks that don't rely on the Serializable interface available, such as kryo, https://github.com/EsotericSoftware/kryo. The performance is great and the size of the output is small.
I have a working prototype that persists the session in memcached and uses kryo for serialization. The initial work involves focusing on the main session. The tool session and context session are next in line. The serialization is taking 2-6 millis to complete and storing in memcached takes 4-7 ms (locally anyway, over the network would expect some delay). There are indeed classloading issues to deal with which is why Terracotta got selected in the first place. I'm experimenting with some ideas, it may not work out, it remains to be discovered. Even if we couldn't get the full session in all of its 3 incarnations to serialize and replicate, I think just having the top level session working, its still a huge step forward. I have this working already.
There is more to work out, here which I'll itemize as this develops. This started out as a fast and loose discovery project. Now that i've proven what is possible we can discuss a design and improve on things. I'm not married to anything here, its just raw discovery, but I've convinced myself this is indeed possible.
There is virtually no configuration needed for this setup, except for telling sakai where your memcached servers are. There is no reliance on tomcat specifics or proprietary software. Memcached is very easy to install and manage, its virtually one yum command in linux. I'm not married to it, we could decide to use redis or make the storage pluggable, whatever.
I'm putting my work into a branch, and intend to work closely with whoever wishes to contribute to this effort.
IMHO, the biggest architectural issues with Sakai right now are: no session replication, no distributed cache (make db usage higher and requires expensive app nodes with bazillion amounts of RAM), and the lack of a language independent cluster friendly messaging system (cache invalidation, warehousing, chat, data integration, etc, etc). These problems make Sakai more expensive to scale and manage, and make cloud offerings difficult. As I see it, this work is a step towards beginning to address these issues. Getting something like memcached or redis integrated has the potential to actually address all of these issues in the longer term.
Currently the only changes are in kernel: https://source.sakaiproject.org/svn/kernel/branches/SAK-25440
So you can just pull that branch. I setup a whole top level thing in case I need to start working in other areas.
Basic Design To Date
1. The request filter writes the serialized session object to memcached in postProcessResponse(). This involves using the Kryo library to serialize the MySession object, and the spymemcached library to store in memcached. Since Kryo allows you to create serializer objects, the serialization work doesn't involve messing with existing code (for the most part).
2. In SessionComponent.getSession() if the session can't be found in the local instance, memcached will be consulted, if the session is found it will be deserialized into a new MySession object on the new node.
So the idea is you are still going to be sticky to a tomcat. You change your load balancer to support failover, so in the case of a tomcat node failing you'll get sent to a new tomcat, and your session comes back. For right now that means you are logged on. The tool state carrying forward is a tool by tool process. There is no silver bullet for that.
enable -Dsakai.cluster.terracotta=true as a system prop. This puts any non portable objects into a special map that will not be serialized. You'll get a bunch of errors about not being able to find the Terracotta named classloaded but you ignore those for now.
enable -Dsakai.cluster.memcached=true. This enables the get/set memcached calls to read and write the serialized session.
- set to the location of your memcached server if you aren't going with the default as shown
For a demo: http://screencast.com/t/LXG2EyAZ