Public Member Functions
def	__init__
def	process
def	put
Public Attributes
	cm
	free
	handles
	queue
	request_error
	request_init
	request_respond
Private Member Functions
def	_request_error
def	_request_init
def	_request_respond

Detailed Description

Manager of multiple concurrent or overlapping HTTP requests.

This is a utility class acting as a pump of several overlapping
HTTP requests against any number of HTTP or HTTPS servers. It
uses a configurable number of simultaneous connections, ten by
default. The actual connection layer is handled using curl, and
the client classes need to aware of this to a limited degree.

The client supplies optional callback methods for initialising,
responding and handling errors on connections. At the very least
the request response callback should be defined.

This class is not designed for multi-threaded use. It employs
overlapping requests, but in a single thread. Only one thread
at a time should be calling `process()`; several threads may
call `.put()` provided the caller uses a mutex so that only one
thread calls into the method at a time.

Definition at line 4 of file HTTP.py.

Constructor & Destructor Documentation

def HTTP::RequestManager::__init__	(	self,
		num_connections = `10`,
		ssl_opts = `None`,
		user_agent = `None`,
		request_headers = `None`,
		request_init = `None`,
		request_respond = `None`,
		request_error = `None`,
		handle_init = `None`
	)

Initialise the request manager. The arguments are:

:arg num_connections: maximum number of simultaneous connections.
:arg ssl_opts: optional SSLOptions (Monitoring.Core.X509) for SSL
X509 parametre values, e.g. for X509 client authentication.
:arg user_agent: sets user agent identification string if defined.
:arg request_headers: if defined, specifies list of additional HTTP
request headers to be added to each request.
:arg request_init: optional callback to initialise requests; the
default assumes each task is a URL to access and sets the `URL`
property on the curl object to the task value.
:arg request_respond: callback for handling responses; at the very
minimum this should be defined as the default one does nothing.
:arg request_error: callback for handling connection errors; the
default one raises a RuntimeException.
:arg handle_init: callback for customising connection handles at
creation time; the callback will be invoked for each connection
object as it's created and queued to the idle connection list.

Definition at line 23 of file HTTP.py.

00027                                                         :
00028     """Initialise the request manager. The arguments are:
00029 
00030 :arg num_connections: maximum number of simultaneous connections.
00031 :arg ssl_opts: optional SSLOptions (Monitoring.Core.X509) for SSL
00032 X509 parametre values, e.g. for X509 client authentication.
00033 :arg user_agent: sets user agent identification string if defined.
00034 :arg request_headers: if defined, specifies list of additional HTTP
00035 request headers to be added to each request.
00036 :arg request_init: optional callback to initialise requests; the
00037 default assumes each task is a URL to access and sets the `URL`
00038 property on the curl object to the task value.
00039 :arg request_respond: callback for handling responses; at the very
00040 minimum this should be defined as the default one does nothing.
00041 :arg request_error: callback for handling connection errors; the
00042 default one raises a RuntimeException.
00043 :arg handle_init: callback for customising connection handles at
00044 creation time; the callback will be invoked for each connection
00045 object as it's created and queued to the idle connection list."""
00046     self.request_respond = request_respond or self._request_respond
00047     self.request_error = request_error or self._request_error
00048     self.request_init = request_init or self._request_init
00049     self.cm = CurlMulti()
00050     self.handles = [Curl() for i in xrange(0, num_connections)]
00051     self.free = [c for c in self.handles]
00052     self.queue = []
00053 
00054     for c in self.handles:
00055       c.buffer = None
00056       c.setopt(NOSIGNAL, 1)
00057       c.setopt(TIMEOUT, 300)
00058       c.setopt(CONNECTTIMEOUT, 30)
00059       c.setopt(FOLLOWLOCATION, 1)
00060       c.setopt(MAXREDIRS, 5)
00061       if user_agent:
00062         c.setopt(USERAGENT, user_agent)
00063       if ssl_opts:
00064         c.setopt(CAPATH, ssl_opts.ca_path)
00065         c.setopt(SSLCERT, ssl_opts.cert_file)
00066         c.setopt(SSLKEY, ssl_opts.key_file)
00067         if ssl_opts.key_pass:
00068           c.setopt(SSLKEYPASSWD, ssl_opts.key_pass)
00069       if request_headers:
00070         c.setopt(HTTPHEADER, request_headers)
00071       if handle_init:
00072         handle_init(c)

Member Function Documentation

def HTTP::RequestManager::_request_error	(	self,
		c,
		task,
		errmsg,
		errno
	)	`[private]`

Default request error callback.

Definition at line 77 of file HTTP.py.

00078                                                   :
00079     """Default request error callback."""
00080     raise RuntimeError((task, errmsg, errno))

def HTTP::RequestManager::_request_init	(	self,
		c,
		url
	)	`[private]`

Default request initialisation callback.

Definition at line 73 of file HTTP.py.

00074                                  :
00075     """Default request initialisation callback."""
00076     c.setopt(URL, url)

def HTTP::RequestManager::_request_respond	(	self,
		args
	)	`[private]`

Default request response callback.

Definition at line 81 of file HTTP.py.

00082                                    :
00083     """Default request response callback."""
00084     pass

def HTTP::RequestManager::process ( self )

Process pending requests until none are left.

This method processes all requests queued with `.put()` until they
have been fully processed. It calls the ``request_respond`` callback
for all successfully completed requests, and ``request_error`` for
all failed ones.

Any new requests added by callbacks by invoking ``put()`` are also
processed before returning.

Definition at line 90 of file HTTP.py.

00091                    :
00092     """Process pending requests until none are left.
00093 
00094 This method processes all requests queued with `.put()` until they
00095 have been fully processed. It calls the ``request_respond`` callback
00096 for all successfully completed requests, and ``request_error`` for
00097 all failed ones.
00098 
00099 Any new requests added by callbacks by invoking ``put()`` are also
00100 processed before returning."""
00101     npending = 0
00102     while self.queue or npending:
00103       while self.queue and self.free:
00104         c = self.free.pop()
00105         c.task = self.queue.pop(0)
00106         c.buffer = b = StringIO()
00107         c.setopt(WRITEFUNCTION, b.write)
00108         self.request_init(c, *c.task)
00109         self.cm.add_handle(c)
00110         npending += 1
00111 
00112       while True:
00113         ret, nhandles = self.cm.perform()
00114         if ret != E_CALL_MULTI_PERFORM:
00115           break
00116 
00117       while True:
00118         numq, ok, err = self.cm.info_read()
00119 
00120         for c in ok:
00121           assert npending > 0
00122           self.cm.remove_handle(c)
00123           self.request_respond(c)
00124           c.buffer = None
00125           self.free.append(c)
00126           npending -= 1
00127 
00128         for c, errno, errmsg in err:
00129           assert npending > 0
00130           self.cm.remove_handle(c)
00131           self.free.append(c)
00132           npending -= 1
00133           self.request_error(c, c.task, errmsg, errno)
00134 
00135         if numq == 0:
00136           break
00137 
00138       self.cm.select(1.)
00139

def HTTP::RequestManager::put	(	self,
		task
	)

Add a new task. The task object should be a tuple and is
passed to ``request_init`` callback passed to the constructor.

Definition at line 85 of file HTTP.py.

00086                      :
00087     """Add a new task. The task object should be a tuple and is
00088 passed to ``request_init`` callback passed to the constructor."""
00089     self.queue.append(task)