CMS 3D CMS Logo

 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes | Private Member Functions
HTTP.RequestManager Class Reference

Public Member Functions

def __init__
 
def process
 
def put
 

Public Attributes

 cm
 
 free
 
 handles
 
 queue
 
 request_error
 
 request_init
 
 request_respond
 

Private Member Functions

def _request_error
 
def _request_init
 
def _request_respond
 

Detailed Description

Manager of multiple concurrent or overlapping HTTP requests.

This is a utility class acting as a pump of several overlapping
HTTP requests against any number of HTTP or HTTPS servers. It
uses a configurable number of simultaneous connections, ten by
default. The actual connection layer is handled using curl, and
the client classes need to aware of this to a limited degree.

The client supplies optional callback methods for initialising,
responding and handling errors on connections. At the very least
the request response callback should be defined.

This class is not designed for multi-threaded use. It employs
overlapping requests, but in a single thread. Only one thread
at a time should be calling `process()`; several threads may
call `.put()` provided the caller uses a mutex so that only one
thread calls into the method at a time.

Definition at line 4 of file HTTP.py.

Constructor & Destructor Documentation

def HTTP.RequestManager.__init__ (   self,
  num_connections = 10,
  ssl_opts = None,
  user_agent = None,
  request_headers = None,
  request_init = None,
  request_respond = None,
  request_error = None,
  handle_init = None 
)
Initialise the request manager. The arguments are:

:arg num_connections: maximum number of simultaneous connections.
:arg ssl_opts: optional SSLOptions (Monitoring.Core.X509) for SSL
X509 parametre values, e.g. for X509 client authentication.
:arg user_agent: sets user agent identification string if defined.
:arg request_headers: if defined, specifies list of additional HTTP
request headers to be added to each request.
:arg request_init: optional callback to initialise requests; the
default assumes each task is a URL to access and sets the `URL`
property on the curl object to the task value.
:arg request_respond: callback for handling responses; at the very
minimum this should be defined as the default one does nothing.
:arg request_error: callback for handling connection errors; the
default one raises a RuntimeException.
:arg handle_init: callback for customising connection handles at
creation time; the callback will be invoked for each connection
object as it's created and queued to the idle connection list.

Definition at line 26 of file HTTP.py.

26 
27  request_error = None, handle_init = None):
28  """Initialise the request manager. The arguments are:
29 
30 :arg num_connections: maximum number of simultaneous connections.
31 :arg ssl_opts: optional SSLOptions (Monitoring.Core.X509) for SSL
32 X509 parametre values, e.g. for X509 client authentication.
33 :arg user_agent: sets user agent identification string if defined.
34 :arg request_headers: if defined, specifies list of additional HTTP
35 request headers to be added to each request.
36 :arg request_init: optional callback to initialise requests; the
37 default assumes each task is a URL to access and sets the `URL`
38 property on the curl object to the task value.
39 :arg request_respond: callback for handling responses; at the very
40 minimum this should be defined as the default one does nothing.
41 :arg request_error: callback for handling connection errors; the
42 default one raises a RuntimeException.
43 :arg handle_init: callback for customising connection handles at
44 creation time; the callback will be invoked for each connection
45 object as it's created and queued to the idle connection list."""
46  self.request_respond = request_respond or self._request_respond
47  self.request_error = request_error or self._request_error
48  self.request_init = request_init or self._request_init
49  self.cm = CurlMulti()
50  self.handles = [Curl() for i in xrange(0, num_connections)]
51  self.free = [c for c in self.handles]
52  self.queue = []
53 
54  for c in self.handles:
55  c.buffer = None
56  c.setopt(NOSIGNAL, 1)
57  c.setopt(TIMEOUT, 300)
58  c.setopt(CONNECTTIMEOUT, 30)
59  c.setopt(FOLLOWLOCATION, 1)
60  c.setopt(MAXREDIRS, 5)
61  if user_agent:
62  c.setopt(USERAGENT, user_agent)
63  if ssl_opts:
64  c.setopt(CAPATH, ssl_opts.ca_path)
65  c.setopt(SSLCERT, ssl_opts.cert_file)
66  c.setopt(SSLKEY, ssl_opts.key_file)
67  if ssl_opts.key_pass:
68  c.setopt(SSLKEYPASSWD, ssl_opts.key_pass)
69  if request_headers:
70  c.setopt(HTTPHEADER, request_headers)
71  if handle_init:
72  handle_init(c)
def _request_error
Definition: HTTP.py:77
def _request_respond
Definition: HTTP.py:81
def _request_init
Definition: HTTP.py:73

Member Function Documentation

def HTTP.RequestManager._request_error (   self,
  c,
  task,
  errmsg,
  errno 
)
private
Default request error callback.

Definition at line 77 of file HTTP.py.

77 
78  def _request_error(self, c, task, errmsg, errno):
79  """Default request error callback."""
80  raise RuntimeError((task, errmsg, errno))
def _request_error
Definition: HTTP.py:77
def HTTP.RequestManager._request_init (   self,
  c,
  url 
)
private
Default request initialisation callback.

Definition at line 73 of file HTTP.py.

73 
74  def _request_init(self, c, url):
75  """Default request initialisation callback."""
76  c.setopt(URL, url)
def _request_init
Definition: HTTP.py:73
def HTTP.RequestManager._request_respond (   self,
  args 
)
private
Default request response callback.

Definition at line 81 of file HTTP.py.

81 
82  def _request_respond(self, *args):
83  """Default request response callback."""
84  pass
def _request_respond
Definition: HTTP.py:81
def HTTP.RequestManager.process (   self)
Process pending requests until none are left.

This method processes all requests queued with `.put()` until they
have been fully processed. It calls the ``request_respond`` callback
for all successfully completed requests, and ``request_error`` for
all failed ones.

Any new requests added by callbacks by invoking ``put()`` are also
processed before returning.

Definition at line 90 of file HTTP.py.

References HTTP.RequestManager.free, HTTP.RequestManager.queue, HTTP.RequestManager.request_error, HTTP.RequestManager.request_init, and HTTP.RequestManager.request_respond.

Referenced by ConfigBuilder.ConfigBuilder.addExtraStream(), ConfigBuilder.ConfigBuilder.completeInputCommand(), ConfigBuilder.ConfigBuilder.doNotInlineEventContent(), Vispa.Plugins.ConfigEditor.ConfigDataAccessor.ConfigDataAccessor.dumpPython(), ConfigBuilder.ConfigBuilder.PrintAllModules.leave(), Vispa.Plugins.ConfigEditor.ConfigDataAccessor.ConfigDataAccessor.open(), Vispa.Plugins.ConfigEditor.ConfigDataAccessor.ConfigDataAccessor.outputEventContent(), ConfigBuilder.ConfigBuilder.prepare_HLT(), ConfigBuilder.ConfigBuilder.prepare_LHE(), ConfigBuilder.ConfigBuilder.prepare_PATFILTER(), ConfigBuilder.ConfigBuilder.prepare_VALIDATION(), ConfigBuilder.ConfigBuilder.renameHLTprocessInSequence(), ConfigBuilder.ConfigBuilder.renameInputTagsInSequence(), ConfigBuilder.ConfigBuilder.scheduleSequence(), Vispa.Plugins.ConfigEditor.ConfigDataAccessor.ConfigDataAccessor.setProcess(), and Vispa.Plugins.ConfigEditor.ConfigDataAccessor.ConfigDataAccessor.setProperty().

90 
91  def process(self):
92  """Process pending requests until none are left.
93 
94 This method processes all requests queued with `.put()` until they
95 have been fully processed. It calls the ``request_respond`` callback
96 for all successfully completed requests, and ``request_error`` for
97 all failed ones.
98 
99 Any new requests added by callbacks by invoking ``put()`` are also
100 processed before returning."""
101  npending = 0
102  while self.queue or npending:
103  while self.queue and self.free:
104  c = self.free.pop()
105  c.task = self.queue.pop(0)
106  c.buffer = b = StringIO()
107  c.setopt(WRITEFUNCTION, b.write)
108  self.request_init(c, *c.task)
109  self.cm.add_handle(c)
110  npending += 1
111 
112  while True:
113  ret, nhandles = self.cm.perform()
114  if ret != E_CALL_MULTI_PERFORM:
115  break
116 
117  while True:
118  numq, ok, err = self.cm.info_read()
119 
120  for c in ok:
121  assert npending > 0
122  self.cm.remove_handle(c)
123  self.request_respond(c)
124  c.buffer = None
125  self.free.append(c)
126  npending -= 1
127 
128  for c, errno, errmsg in err:
129  assert npending > 0
130  self.cm.remove_handle(c)
131  self.free.append(c)
132  npending -= 1
133  self.request_error(c, c.task, errmsg, errno)
134 
135  if numq == 0:
136  break
137 
138  self.cm.select(1.)
139 
def HTTP.RequestManager.put (   self,
  task 
)
Add a new task. The task object should be a tuple and is
passed to ``request_init`` callback passed to the constructor.

Definition at line 85 of file HTTP.py.

85 
86  def put(self, task):
87  """Add a new task. The task object should be a tuple and is
88 passed to ``request_init`` callback passed to the constructor."""
89  self.queue.append(task)

Member Data Documentation

HTTP.RequestManager.cm

Definition at line 48 of file HTTP.py.

HTTP.RequestManager.free

Definition at line 50 of file HTTP.py.

Referenced by HTTP.RequestManager.process().

HTTP.RequestManager.handles

Definition at line 49 of file HTTP.py.

Referenced by SimpleJetAnalyzer.SimpleJetAnalyzer.declareHandles(), ZEleEleAnalyzer.ZEleEleAnalyzer.declareHandles(), ZMuMuAnalyzer.ZMuMuAnalyzer.declareHandles(), TriggerAnalyzer.TriggerAnalyzer.declareHandles(), MetAnalyzer.MetAnalyzer.declareHandles(), objects.TauAnalyzer.TauAnalyzer.declareHandles(), core.TriggerBitFilter.TriggerBitFilter.declareHandles(), core.TriggerMatchAnalyzer.TriggerMatchAnalyzer.declareHandles(), core.TriggerBitAnalyzer.TriggerBitAnalyzer.declareHandles(), RazorAnalyzer.RazorAnalyzer.declareHandles(), objects.METAnalyzer.METAnalyzer.declareHandles(), AlphaTAnalyzer.AlphaTAnalyzer.declareHandles(), objects.PhotonAnalyzer.PhotonAnalyzer.declareHandles(), core.AutoFillTreeProducer.AutoFillTreeProducer.declareHandles(), MT2Analyzer.MT2Analyzer.declareHandles(), JetAnalyzer.JetAnalyzer.declareHandles(), objects.VertexAnalyzer.VertexAnalyzer.declareHandles(), objects.IsoTrackAnalyzer.IsoTrackAnalyzer.declareHandles(), core.PileUpAnalyzer.PileUpAnalyzer.declareHandles(), objects.LeptonAnalyzer.LeptonAnalyzer.declareHandles(), objects.JetAnalyzer.JetAnalyzer.declareHandles(), core.AutoFillTreeProducer.AutoFillTreeProducer.fillTree(), objects.LeptonAnalyzer.LeptonAnalyzer.makeAllElectrons(), objects.LeptonAnalyzer.LeptonAnalyzer.makeAllMuons(), objects.IsoTrackAnalyzer.IsoTrackAnalyzer.makeIsoTrack(), objects.LeptonAnalyzer.LeptonAnalyzer.makeLeptons(), objects.PhotonAnalyzer.PhotonAnalyzer.makePhotons(), objects.TauAnalyzer.TauAnalyzer.makeTaus(), objects.METAnalyzer.METAnalyzer.makeTkMETs(), SimpleJetAnalyzer.SimpleJetAnalyzer.process(), MetAnalyzer.MetAnalyzer.process(), core.TriggerBitFilter.TriggerBitFilter.process(), core.TriggerMatchAnalyzer.TriggerMatchAnalyzer.process(), TriggerAnalyzer.TriggerAnalyzer.process(), JetAnalyzer.JetAnalyzer.process(), core.TriggerBitAnalyzer.TriggerBitAnalyzer.process(), DiLeptonAnalyzer.DiLeptonAnalyzer.process(), objects.VertexAnalyzer.VertexAnalyzer.process(), core.PileUpAnalyzer.PileUpAnalyzer.process(), objects.JetAnalyzer.JetAnalyzer.process(), and objects.PhotonAnalyzer.PhotonAnalyzer.randomCone().

HTTP.RequestManager.queue

Definition at line 51 of file HTTP.py.

Referenced by HTTP.RequestManager.process().

HTTP.RequestManager.request_error

Definition at line 46 of file HTTP.py.

Referenced by HTTP.RequestManager.process().

HTTP.RequestManager.request_init

Definition at line 47 of file HTTP.py.

Referenced by HTTP.RequestManager.process().

HTTP.RequestManager.request_respond

Definition at line 45 of file HTTP.py.

Referenced by HTTP.RequestManager.process().