CIS service for Group Publishing

This is a webservice to serve group information for UIO instance. Currently, the service provides one function to list flat group members. As some groups have a lot of members, more than bofhd will show to non-superusers, this service may be useful. It does not limit the number of members returned. Note that this is the first CIS web service to incorporate authentication. It exposes methods to authenticate and deauthenticate the users.

Proceed to Deployment section if you do not want to read development stuff

Information about groups are available through bofhd. Output of group_list and group_list_expanded commands is limited to a predefined maximum number to make bofhd more responsive. The webservice shall provide group information much like bofhd does, but it shall list all members of the groups without size limits .

Requirements

  1. Support authentication and authorisation. Username and password authentication will be supported and other methods might be provided later on. Note that this webservice is thought to be used in batch jobs, i.e. there will be a script trying to retreive group information through the service and providing username and passwords is cumbersome then since the credentials must be stored in the script if not entered by the user of the script.
  2. Use SOAP protocol. The CIS webservices used soaplib for this purpose, which complicated authentication. The porting to rpclib fixed this issue.

Components

Test client

Developing a client for the service is not a part of the requirements, but it was needed for testing. The Individuation service works with the PHP client specially developed for that purpose AFAIK. For the Group Publishing service test we can use a simple suds client. To use it at cere-utv01 install python-suds

The client code can look something like this

import sys
import logging
from suds.client import Client
service_url = 'https://cere-utv01.uio.no:11011/SOAP/?wsdl'
client = Client(service_url, cache=None)
in_header = client.factory.create('InHeader')
in_header.id = 'myinheader'
client.set_options(soapheaders=in_header)
result = client.service.<remote method name>

Source of a possible client is saved in cerebrum_sites/hacks/uio/cis_groupservice_test_client.py.

Server

The server is based on Twisted and uses rpclib for SOAP support. Core functionality of CIS services resides in Cerebrum/modules/cis/SoapListener.py. This module defines classes to start up the Twisted server with or without encryption, to prepare rpclib and to handle soap faults. EndUserFault occurs when client provides wrong data, UnknownFault occurs when client provided valid data but something wrong happened on the server side.

More specific functionality can be defined in service modules. Twisted setup is handled in TwistedSoapStarter for an unencrypted server and in TLSTwistedSoapSarter for an encrypted version. SoapListener was implemeted when developing the *Individuation* project (*the forgotten password* service).

Authentication

Authentication is implemeted as a service in a separate application module Cerebrum/modules/cis/auth.py. To let a service make use of authentication it needs to add the auth module to the list of applications for the SOAP server:

applications = [auth.PasswordAuthenticationService, GroupService]

The PasswordAuthenticationService has methods, authenticate and deauthenticate. The authenticate method is a copy of bofhd's method * bofhd_login*, which contains functionality for both verifying that the username and password is correct, and that the user is allowed to log on, e.g. by checking for quarantines.

Note that this means that every active user in Cerebrum authenticates through this service, in the same way as in bofhd. We can add authorization later if need be.

The authentication process:

  1. The client authenticates, by running authenticate(username, password)

  2. If the server verifies this a new, random session-ID is returned to the client. This session-ID is also added to the SOAP-headers in the response message.

  3. When the client is calling a new method, e.g. get_members(groupname) , it needs to have the session-ID in its SOAP- headers.

  4. The webservice is first validating that the given session-ID exists, and then checks that this session has authenticated.

    • If this is not the case, a NotAuthenticatedFault is returned to the client.
    • If the client is already authenticated, the service then runs the command that the client wanted to run.
  5. The client gets the returned data from the given command. The session-ID is included in the SOAP-headers.

  6. When the client is done, it should call deauthenticate() , together with its session-ID in the SOAP-headers. The service now logs out the user and deletes the session, this last step is not totally necessary, since there is a session timeout that will deauthenticate the user, but for graceful operation it is advised to call it.

Authorization

For now any authenticated user is authorized to call exposed methods.

You may skip the rest of Authorization section.

To support access control, we сan simply make direct use of BofhdAuth, which is the class that is used in bofhd for access control. It might be too complex for this service, but it is a wellknown and well tested class, and it was quite easy to make use of in the service.

How it works is that methods call BofhdAuth's methods, for instance can_moderate_group(operator_id, group_id). If access is granted, it returns true. If not, it raises a PermissionDenied exception, which propagates to the client as a NotAuthorizedFault.

The GroupService class

Service class skeleton as recommended in Transition from soaplib to rpclib:

class GroupService(ServiceBase):
    """The main Service"""
    __in_header__ = SessionHeader
    __out_header__ = SessionHeader

    @rpc(Mandatory.String, _returns=Iterable(String))
    def get_members(ctx, group_name):
        return ctx.udc.group.search_members_flat(group_name)

def _on_method_call(ctx):
    """Event for validating session ID, which should be in SOAP header."""
    assert ctx.in_object is not None
    if ctx.in_header.session_id != ctx.udc['session'].uid
        raise AuthenticationError()

# Every call to methods in GroupService must first go through
# _on_method_call, which validates the session ID. This makes the
# authentication part.
GroupService.event_manager.add_listener('method_call', _on_method_call)

For each public method of the service, that is, those decorated with rpc, we make use of following events:

  • method_call - occurs right before the service method is executed.
  • method_return_object - occurs right after the service method is executed.
  • method_exception_object - when exception occurs in the service method, but before the exception is serialized and sent back to the client.

Functions registered for respective events will control soapheader contents and as a result only authenticated users will gain access.

Cerebrum tier

The Group service needs a Cerebrum class for communicating with the Cerebrum API and retrieving data from Cerebrum. A class similar to Individuation will implement the needed functionality. Note that the GroupService class needs to explicitly call the methods in the Cerebrum tier.

class CerebrumGroup:
   def __init__(self):
        <initialize database connections>
   def close(self):
        <close db connections>
   def search_members_flat(self, groupname):
        <look for group members>
   return [iterable of group members]
Functions

The service will in the future be extended with more functions, but for now we have the function seach_members_flat that lists all group members and flattens the member if it is a group itself. Example of such a function implemented in bofhd is

*group list_expaned <operator> <groupname>:*
 limited by BOFHD_MAX_MATCHES, superuser can see all members
 return list of group members after expansion
 each element has:
   member_id
   member_type
   member_name
   group_name

Communication scheme

Client calls authenticate method, gets session_id back from server in the soap header and sends the session_id in the soap header in subsequent calls. To stop communication client calls remote method deauthenticate which completes the interaction in this session . If deauthenticate is not called, the session with expire after timeout. We control session timeout by a single attribute

SoapListener.SessionCacher.sesionTimeout = 600

When the session has expired, the user will simply get a NotAuthenticatedFault.

Remark on rpclib's Events

Events support calling more than one callback. This makes it possible to split up event functionality in different methods, e.g. one for generic session handling and authentication, and one that handles the specific GroupService functionality. This requires that the different callbacks are added to the event manager. For example:

#SoapListener.py
BasicSoapServer.event_manager.add_listener('method_call', _on_method_call)
#SoapGroupServer.py
GroupService.event_manager.add_listener('method_call', SoapListener.on_method_call_session)

Now two handlers are registered for ' method_call' event and right before the service method is executed _on_method_call and then SoapListener.method_call_session will run.

Deployment

This service uses rpclib and this package must be installed in UIO production environment prior to deployment. A deployment request for rpclib was created earlier, refer to RT ticket #980351.

Steps to perform

Communication with a client must be encrypted, so provide a certificate in the same way as for the Individuation service. Note that checking of a client certificate is disabled, meaning that the server can accept all connections. The service uses username password authentication.

  1. Decide what port to open for the service. Individuation uses port 8998, AFAIK, so port 8997 could be a good candidate. Ask nett-drift to open the port for the UiO net. IP blocking may  apply later. Save it in cisconf/groupservice.py as PORT = 8997

  2. Deploy source from git Cerebrum-repo: >

    • Cerebrum/Errors.py
    • Cerebrum/modules/cis/auth.py
    • Cerebrum/modules/cis/SoapListener.py
    • Cerebrum/modules/cis/faults.py
    • Cerebrum/modules/cis/GroupInfo.py
    • servers/cis/SoapGroupServer.py
    • Cerebrum/modules/cis/Utils.py
  3. deploy source from git cerebrum_config repo: relative to /etc/uio/:
    • cisconf/
    • cereconf.py TODO: check if this one is actually used
    • base.py
    • groupservice.py
  4. GroupService job_runner jobs in cerebrum_config/etc/uio/scheduled_jobs.py

    Daemons gets a new action:

    class Daemons(Jobs):
          ...
          cis_groupservice = Action(
             call=AssertRunning("/local/bin/keep-running",
                          params=["-m", "cerebrum-uio-logs@usit.uio.no",
                                        pj(sbin, "SoapGroupServer.py")]),
             when=When(freq=15*60),
             max_duration=None)
    

    MailLog gets a new action

    class MailLog(Jobs):
         ...
         maillog_cis_groupinfo = Action(
             call=System(pj(bin, 'maillog.py'),
                        params=['-f', pj(logdir, 'cis_groupinfo.log'),
                              '-m', 'cerebrum-uio-logs@usit.uio.no',
                             '-u', pj(etc, 'maillog-exceptions.txt'),
                                '--simple-group-jobs']),
             when = When(freq=60*60))
    

    Add the jobs to scheduled_jobs.py

  5. Check security

  6. Smoke test: There is a simple smoke test client in cerebrum_sites/hacks/uio/

    Run python cis_groupservice_test_client.py --help for more details.

Test scenario

  1. Authenticate with your username and password

  2. Use service The test will show number of members in the searched group Any "Server raised fault" message indicates an error at this stage

  3. Deauthenticate

  4. Try to use the service without authentication The test should output:

    -- get_members --
    No handlers could be found for logger "suds.client"
    Server raised fault: 'Not authenticated'
    

Authors: igorer, jokim

Publisert 27. mai 2015 08:36