Transition from soaplib to rpclib

MethodContext

The main difference between soaplib and rpclib for our use cases, is that every public SOAP method are given a MethodContext as the first argument. This object is unique per call and contains the SOAP environment, SOAP headers, the current request, and it even contains a User Defined Context (udc) that we could use to anything we want. We still don't have access to twisted's HTTP environment, but we can now use SOAP headers for the session ID instead. If we're still in need of twisted's environment, I guess it's easier to hack it in than in soaplib.

Events

Events are local functions that could be called at every SOAP request, response and at exceptions. This could be used to e.g. find the session ID from the SOAP headers and validate it and do stuff around it. Another example is that it could check the session and raise an AuthException if the client is not logged in in the web service.

Service classes are not instantiated anymore

Since rpclib makes use of MethodContext, it doesn't instantiate the Service classes anymore. This prevents us from storing data in the instances. Instead we have to use the MethodContex instance.

A tutorial for how rpclib works can be found at http://arskom.github.com/rpclib/manual/metadata.html, and example code for our use is put in https://github.com/plq/rpclib/blob/master/examples/authentication/server_soap.py. A short summary:

class SessionHeader(ComplexModel):
    """Our requested SOAP header"""
    session_id = Mandatory.String

class AuthenticationService(ServiceBase):
    """The Service for authentication"""
    @rpc(String, String, _returns=String, _throws=AuthenticationError)
    def authenticate(ctx, user_name, password):
        ret = ctx.udc.cerebrum.authenticate(user_name, password)
        if ret:
            return ctx.udc.site.makeSession(ret)
        raise AuthenticationError()

class GroupService(ServiceBase):
    """The main Service"""
    __in_header__ = SessionHeader

    @rpc(Mandatory.String, _returns=Iterable(String))
    def get_group_members(ctx, group_name):
        return ctx.udc.group.search_members(group_name)

def _on_method_call(ctx):
    """Event for validating session ID, which should be in SOAP header."""
    assert ctx.in_object is not None
    if ctx.in_header.session_id not in session_db:
        raise AuthenticationError()

# Every call to methods in GroupService must first go through
# _on_method_call, which validates the session ID. This makes the
# authentication part.
GroupService.event_manager.add_listener('method_call', _on_method_call)

# run twisted almost as before with soaplib

Forgotten-password service

The forgotten-password service has been tested by going through every step in the process of both getting out the usernames and setting a new password.

• Getting out username: Worked, by giving both student number and fnr.
• Getting error message for unknown person: Worked, by giving a bogus fnr.
• Sending bogus data through forms: Ampersand is now working, but not utf8-data outside of latin1. Have added a check on the client side that gives the user a message if given data is out of latin1. This is not the best solution, but is what we have the time for. After this fix it worked as it should.
• Getting error message for unknown person on the password page: Worked, got error message by giving bogus fnr, username and phone number.
• Getting same error message for known person, but unknown phone: Worked, same error message.
• Authenticating at the password site: Worked, got an SMS with the token.
• Adding wrong token: Worked, got an error message.
• Adding wrong token maximum times: Worked, token was removed and was sent back to the start page.
• Adding correct token: Worked, was given the password form.
• Trying to set a short password: Worked, got an error message about it being to short.
• Trying to give utf8 characters and bogus data to the password checker: Worked as it should, gave the correct error message.
• Set a valid password: Worked, password got set.

To test the threading part, the same strategy as for when the original CIS was tested for parallell bugs was used. A script is running in an endless loop, running different calls to the website. Note that the captchas are turned off to be able to test this. The script runs different calls for asking if a user exists or not, and checks that the result is as expected. By running more than 11 processes with this script from different IP addresses, started at different times, we stress the service enough to make use of all its threads, which is 10 by default in twisted. This is then run overnight, and the logs are then checked for errors and warnings the next morning.

The stress test was run over the weekend 12th-13th of May with 13 processes from the hosts melk.uio.no, login1.uio.no, login2.uio.no, cere-utv01.uio.no, morpheus.uio.no and umbra.uio.no. The service gave no errors.

Note that this is not a proper stress test, but is just enough to be able to verify that the threads is not creating any major problems for each others. We could still have bugs related to the threads.

Postmasterservice

The postmaster service was quickly tested:

• Logging on (through bofhd): Worked.
• Asking for a list of some affiliations: Found a bug, which got fixed. Worked afterwards.
• Asking for a list of some affiliations and by OUs: Worked, got a list of e-mail addresses.

This service has not been stresstested, as it uses the same framework as the forgotten-password service.

1. Merging

The branch soaplib-to-rpclib has to be merged into trunk first, before the sysadmins wants to deal with it. This was done in svn revision 15951 .

2. Installing rpclib

The rpclib package needs to be installed in the production environment.

See the rpclib page for more information, or the page at python.org: http://pypi.python.org/pypi/rpclib. Note that the latest version has not been tested yet.

3. Deploying Cerebrum code

Note that this requires some downtime, as the update requires that the web client also gets updated.

Note that we have two such web services at UiO (Individuation and Postmaster), but both clients depends on the same code files, so have to deploy both at the same time.

Related files:

• Cerebrum/Errors.py
• Cerebrum/modules/cis/ (everything under here)
• Cerebrum/modules/no/uio/PostmasterCommands.py
• servers/cis/ (everything under here)
• cisconf (everything under here) - Note that some instances are not updated with this yet, which means that we have to move their config out of cereconf and into cisconf files. This is not done yet, to avoid getting in conflict with other deployments. You could for instance talk to jokim for doing this when the given instance is about to be deployed.

How to deploy:

1. Update all related files.
2. Run a python-shell with the command: import rpclib; rpclib.__version__
3. Restart the SoapPostmasterService.py job.
4. Make sure the job runs and check the log for errors: /cerebrum/uio/var/log/cerebrum/cis_postmaster.log
5. Restart the SoapIndividuationService.py job.
6. Make sure the job runs and check the log for errors: /cerebrum/uio/var/log/cerebrum/cis_individuation.log
7. Deploy the PHP clients, as they don't work at this moment.

4. Deploying the PHP clients

We have two PHP clients that must be updated, the forgotten-password service and the postmaster service, but the last is only for UiO. At UiO is both placed underneath Brukerinfo. The other instances have their own locations for the forgotten-password service.

Related files:

• clients/web/phplib/ (everything under here) - This is the shared code and is on curumo placed in the directory phplib/
• clients/web/postmaster/ (everything under here) - This is for the postmaster site, and is in production placed in the directory src/postmaster/ (only for UiO).
• clients/web/individuation/ (everything under here) - This is for the forgotten-password site, and is in production placed in the directory src/individuation/
• cerebrum_sites/etc/$INSTANCE/cis_passord.web/ - This is the config directory, which might be updated for some instances. Most likely, this is not the case.

How to deploy for UiA

UiA has their own PHP client, which they have modified for their own purposes. The communication part has changed, so they need to update the client's phplib code at the same time as we update our own code. We therefore can't update their Cerebrum code until they say go.

All the related updates are in: phplib/model/CICom.php

We could give them a test server to use for testing, depending on much changes they have to do. We then have to use a port at cere-utv01 that is open for UiA, or ask nett-drift@usit.uio.no to open one for us.

UiA has their own fictive student accounts for testing such scenarios as these, so they could test the password service themselves.

Cleaning up

After every instance is updated and is using rpclib, the old soaplib packages could be uninstalled.

Author: jokim