CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes | Static Public Attributes
upload_popcon.HTTP Class Reference
Inheritance diagram for upload_popcon.HTTP:

Public Member Functions

def __init__
 
def addCERNSSOCookies
 
def discardCookies
 
def getCookies
 
def query
 
def setBaseUrl
 
def setProxy
 
def setRetries
 
def setTimeout
 

Public Attributes

 baseUrl
 
 curl
 
 retries
 

Static Public Attributes

tuple retryCodes = frozenset([502, 503])
 

Detailed Description

Class used for querying URLs using the HTTP protocol.

Definition at line 225 of file upload_popcon.py.

Constructor & Destructor Documentation

def upload_popcon.HTTP.__init__ (   self)

Definition at line 232 of file upload_popcon.py.

References upload_popcon.HTTP.setBaseUrl(), and upload_popcon.HTTP.setRetries().

232 
233  def __init__(self):
234  self.setBaseUrl()
235  self.setRetries()
236 
237  self.curl = pycurl.Curl()
238  self.curl.setopt(self.curl.COOKIEFILE, '')
239  self.curl.setopt(self.curl.SSL_VERIFYPEER, 0)
240  self.curl.setopt(self.curl.SSL_VERIFYHOST, 0)
241 
upload_popcon.HTTP.setRetries
def setRetries
Definition: upload_popcon.py:278
upload_popcon.HTTP.setBaseUrl
def setBaseUrl
Definition: upload_popcon.py:256
upload_popcon.HTTP.curl
curl
Definition: upload_popcon.py:236
upload_popcon.HTTP.__init__
def __init__
Definition: upload_popcon.py:232

Member Function Documentation

def upload_popcon.HTTP.addCERNSSOCookies (   self,
  url,
  secureTarget = True,
  secure = True 
) 
Adds the required CERN SSO cookies for a URL using Kerberos.

After calling this, you can use query() for your SSO-protected URLs.

This method will use your Kerberos ticket to sign in automatically
in CERN SSO (i.e. no password required).

If you do not have a ticket yet, use kinit.

If you wish to make secure SSL connections to the CERN SSO
(i.e. verify peers/hosts), you may need to install the CERN-CA-certs package.
Use secure == False to skip this (i.e. this is the same as curl
-k/--insecure). Not recommended: tell users to install them or use lxplus6.

The same way, if you have a self-signed certificate in your target URL
you can use secureTarget == False as well. Note that this option
is provided in order to be able to use a secure SSL connection to CERN SSO,
even if the connection to your target URL is not secure. Note that
you will probably need the CERN-CA-certs package after you get a certificate
signed by the CERN CA (https://cern.ch/ca), even if you did not need it
for the CERN SSO.

Note that this method *does* a query to the given URL if successful.

Note that you may need different cookies for different URLs/applications.

Note that this method may raise also CERNSSOError exceptions.

Definition at line 367 of file upload_popcon.py.

References upload_popcon._getCERNSSOCookies(), and upload_popcon.HTTP.baseUrl.

367 
368  def addCERNSSOCookies(self, url, secureTarget = True, secure = True):
369  '''Adds the required CERN SSO cookies for a URL using Kerberos.
370 
371  After calling this, you can use query() for your SSO-protected URLs.
372 
373  This method will use your Kerberos ticket to sign in automatically
374  in CERN SSO (i.e. no password required).
375 
376  If you do not have a ticket yet, use kinit.
377 
378  If you wish to make secure SSL connections to the CERN SSO
379  (i.e. verify peers/hosts), you may need to install the CERN-CA-certs package.
380  Use secure == False to skip this (i.e. this is the same as curl
381  -k/--insecure). Not recommended: tell users to install them or use lxplus6.
382 
383  The same way, if you have a self-signed certificate in your target URL
384  you can use secureTarget == False as well. Note that this option
385  is provided in order to be able to use a secure SSL connection to CERN SSO,
386  even if the connection to your target URL is not secure. Note that
387  you will probably need the CERN-CA-certs package after you get a certificate
388  signed by the CERN CA (https://cern.ch/ca), even if you did not need it
389  for the CERN SSO.
390 
391  Note that this method *does* a query to the given URL if successful.
392 
393  Note that you may need different cookies for different URLs/applications.
394 
395  Note that this method may raise also CERNSSOError exceptions.
396  '''
397 
398  for cookie in _getCERNSSOCookies(self.baseUrl + url, secureTarget, secure):
399  self.curl.setopt(self.curl.COOKIELIST, cookie)
400 
401 # common/http.py end
402 
upload_popcon.HTTP.baseUrl
baseUrl
Definition: upload_popcon.py:261
upload_popcon._getCERNSSOCookies
def _getCERNSSOCookies
Definition: upload_popcon.py:53
upload_popcon.HTTP.addCERNSSOCookies
def addCERNSSOCookies
Definition: upload_popcon.py:367
def upload_popcon.HTTP.discardCookies (   self) 
Discards cookies.

Definition at line 249 of file upload_popcon.py.

Referenced by upload_popcon.HTTP.query(), and uploadConditions.HTTP.query().

249 
250  def discardCookies(self):
251  '''Discards cookies.
252  '''
253 
254  self.curl.setopt(self.curl.COOKIELIST, 'ALL')
255 
upload_popcon.HTTP.discardCookies
def discardCookies
Definition: upload_popcon.py:249
def upload_popcon.HTTP.getCookies (   self) 
Returns the list of cookies.

Definition at line 242 of file upload_popcon.py.

242 
243  def getCookies(self):
244  '''Returns the list of cookies.
245  '''
246 
247  return self.curl.getinfo(self.curl.INFO_COOKIELIST)
248 
upload_popcon.HTTP.getCookies
def getCookies
Definition: upload_popcon.py:242
def upload_popcon.HTTP.query (   self,
  url,
  data = None,
  files = None,
  keepCookies = True 
) 
Queries a URL, optionally with some data (dictionary).

If no data is specified, a GET request will be used.
If some data is specified, a POST request will be used.

If files is specified, it must be a dictionary like data but
the values are filenames.

By default, cookies are kept in-between requests.

A HTTPError exception is raised if the response's HTTP code is not 200.

Definition at line 295 of file upload_popcon.py.

References upload_popcon.HTTP.baseUrl, upload_popcon.HTTP.discardCookies(), list(), upload_popcon.HTTP.retries, and upload_popcon.HTTP.retryCodes.

Referenced by production_tasks.BaseDataset.run(), and edmIntegrityCheck.IntegrityCheck.test().

295 
296  def query(self, url, data = None, files = None, keepCookies = True):
297  '''Queries a URL, optionally with some data (dictionary).
298 
299  If no data is specified, a GET request will be used.
300  If some data is specified, a POST request will be used.
301 
302  If files is specified, it must be a dictionary like data but
303  the values are filenames.
304 
305  By default, cookies are kept in-between requests.
306 
307  A HTTPError exception is raised if the response's HTTP code is not 200.
308  '''
309 
310  if not keepCookies:
311  self.discardCookies()
312 
313  url = self.baseUrl + url
314 
315  # make sure the logs are safe ... at least somewhat :)
316  data4log = copy.copy(data)
317  if data4log:
318  if 'password' in data4log.keys():
319  data4log['password'] = '*'
320 
321  retries = [0] + list(self.retries)
322 
323  while True:
324  logging.debug('Querying %s with data %s and files %s (retries left: %s, current sleep: %s)...', url, data4log, files, len(retries), retries[0])
325 
326  time.sleep(retries.pop(0))
327 
328  try:
329  self.curl.setopt(self.curl.URL, url)
330  self.curl.setopt(self.curl.HTTPGET, 1)
331 
332  if data is not None or files is not None:
333  # If there is data or files to send, use a POST request
334 
335  finalData = {}
336 
337  if data is not None:
338  finalData.update(data)
339 
340  if files is not None:
341  for (key, fileName) in files.items():
342  finalData[key] = (self.curl.FORM_FILE, fileName)
343 
344  self.curl.setopt(self.curl.HTTPPOST, finalData.items())
345 
346  response = cStringIO.StringIO()
347  self.curl.setopt(self.curl.WRITEFUNCTION, response.write)
348  self.curl.perform()
349 
350  code = self.curl.getinfo(self.curl.RESPONSE_CODE)
351 
352  if code in self.retryCodes and len(retries) > 0:
353  logging.debug('Retrying since we got the %s error code...', code)
354  continue
355 
356  if code != 200:
357  raise HTTPError(code, response.getvalue())
358 
359  return response.getvalue()
360 
361  except pycurl.error as e:
362  if len(retries) == 0:
363  raise e
364 
365  logging.debug('Retrying since we got the %s pycurl exception...', str(e))
366 
upload_popcon.HTTP.discardCookies
def discardCookies
Definition: upload_popcon.py:249
upload_popcon.HTTP.baseUrl
baseUrl
Definition: upload_popcon.py:261
upload_popcon.HTTPError
Definition: upload_popcon.py:207
upload_popcon.HTTP.retries
retries
Definition: upload_popcon.py:292
upload_popcon.HTTP.query
def query
Definition: upload_popcon.py:295
upload_popcon.HTTP.retryCodes
tuple retryCodes
Definition: upload_popcon.py:229
list
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*","!HLTx*"if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL.It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of"!*"before the partial wildcard feature was incorporated).Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run
def upload_popcon.HTTP.setBaseUrl (   self,
  baseUrl = '' 
) 
Allows to set a base URL which will be prefixed to all the URLs
that will be queried later.

Definition at line 256 of file upload_popcon.py.

Referenced by uploadConditions.HTTP.__init__(), and upload_popcon.HTTP.__init__().

256 
257  def setBaseUrl(self, baseUrl = ''):
258  '''Allows to set a base URL which will be prefixed to all the URLs
259  that will be queried later.
260  '''
261 
262  self.baseUrl = baseUrl
263 
upload_popcon.HTTP.setBaseUrl
def setBaseUrl
Definition: upload_popcon.py:256
upload_popcon.HTTP.baseUrl
baseUrl
Definition: upload_popcon.py:261
def upload_popcon.HTTP.setProxy (   self,
  proxy = '' 
) 
Allows to set a proxy.

Definition at line 264 of file upload_popcon.py.

264 
265  def setProxy(self, proxy = ''):
266  '''Allows to set a proxy.
267  '''
268 
269  self.curl.setopt(self.curl.PROXY, proxy)
270 
upload_popcon.HTTP.setProxy
def setProxy
Definition: upload_popcon.py:264
def upload_popcon.HTTP.setRetries (   self,
  retries = () 
) 
Allows to set retries.

The retries are a sequence of the seconds to wait per retry.

The retries are done on:
    * PyCurl errors (includes network problems, e.g. not being able
      to connect to the host).
    * 502 Bad Gateway (for the moment, to avoid temporary
      Apache-CherryPy issues).
    * 503 Service Temporarily Unavailable (for when we update
      the frontends).

Definition at line 278 of file upload_popcon.py.

Referenced by uploadConditions.HTTP.__init__(), and upload_popcon.HTTP.__init__().

278 
279  def setRetries(self, retries = ()):
280  '''Allows to set retries.
281 
282  The retries are a sequence of the seconds to wait per retry.
283 
284  The retries are done on:
285  * PyCurl errors (includes network problems, e.g. not being able
286  to connect to the host).
287  * 502 Bad Gateway (for the moment, to avoid temporary
288  Apache-CherryPy issues).
289  * 503 Service Temporarily Unavailable (for when we update
290  the frontends).
291  '''
292 
293  self.retries = retries
294 
upload_popcon.HTTP.setRetries
def setRetries
Definition: upload_popcon.py:278
upload_popcon.HTTP.retries
retries
Definition: upload_popcon.py:292
def upload_popcon.HTTP.setTimeout (   self,
  timeout = 0 
) 
Allows to set a timeout.

Definition at line 271 of file upload_popcon.py.

271 
272  def setTimeout(self, timeout = 0):
273  '''Allows to set a timeout.
274  '''
275 
276  self.curl.setopt(self.curl.TIMEOUT, timeout)
277 
upload_popcon.HTTP.setTimeout
def setTimeout
Definition: upload_popcon.py:271

Member Data Documentation

upload_popcon.HTTP.baseUrl

Definition at line 261 of file upload_popcon.py.

Referenced by upload_popcon.HTTP.addCERNSSOCookies(), uploadConditions.HTTP.getToken(), upload_popcon.HTTP.query(), uploadConditions.HTTP.query(), and uploadConditions.HTTP.setBaseUrl().

upload_popcon.HTTP.curl

Definition at line 236 of file upload_popcon.py.

upload_popcon.HTTP.retries

Definition at line 292 of file upload_popcon.py.

Referenced by upload_popcon.HTTP.query(), and uploadConditions.HTTP.query().

tuple upload_popcon.HTTP.retryCodes = frozenset([502, 503])
static

Definition at line 229 of file upload_popcon.py.

Referenced by upload_popcon.HTTP.query(), and uploadConditions.HTTP.query().

Generated for CMSSW Reference Manual by doxygen 1.8.5