CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
List of all members | Public Member Functions | Public Attributes | Static Public Attributes
uploadConditions.HTTP Class Reference
Inheritance diagram for uploadConditions.HTTP:

Public Member Functions

def __init__
 
def discardCookies
 
def getCookies
 
def getToken
 
def query
 
def setBaseUrl
 
def setProxy
 
def setRetries
 
def setTimeout
 

Public Attributes

 baseUrl
 
 curl
 
 retries
 
 token
 

Static Public Attributes

tuple retryCodes = frozenset([502, 503])
 

Detailed Description

Class used for querying URLs using the HTTP protocol.

Definition at line 220 of file uploadConditions.py.

Constructor & Destructor Documentation

def uploadConditions.HTTP.__init__ (   self)

Definition at line 226 of file uploadConditions.py.

References upload_popcon.HTTP.setBaseUrl(), uploadConditions.HTTP.setBaseUrl(), upload_popcon.HTTP.setRetries(), and uploadConditions.HTTP.setRetries().

226 
227  def __init__(self):
228  self.setBaseUrl()
229  self.setRetries()
230 
231  self.curl = pycurl.Curl()
232  self.curl.setopt(self.curl.COOKIEFILE, '') # in memory
233 
234  #-toDo: make sure we have the right options set here to use ssl
235  #-review(2015-09-25): check and see - action: AP
236  # self.curl.setopt(self.curl.SSL_VERIFYPEER, 1)
237  self.curl.setopt(self.curl.SSL_VERIFYPEER, 0)
238  self.curl.setopt(self.curl.SSL_VERIFYHOST, 2)
239 
240  self.baseUrl = None
241 
242  self.token = None

Member Function Documentation

def uploadConditions.HTTP.discardCookies (   self) 
Discards cookies.

Definition at line 248 of file uploadConditions.py.

Referenced by uploadConditions.HTTP.query().

248 
249  def discardCookies(self):
250  '''Discards cookies.
251  '''
252  self.curl.setopt(self.curl.COOKIELIST, 'ALL')
253 
def uploadConditions.HTTP.getCookies (   self) 
Returns the list of cookies.

Definition at line 243 of file uploadConditions.py.

243 
244  def getCookies(self):
245  '''Returns the list of cookies.
246  '''
247  return self.curl.getinfo(self.curl.INFO_COOKIELIST)
def uploadConditions.HTTP.getToken (   self,
  username,
  password 
)

Definition at line 288 of file uploadConditions.py.

References upload_popcon.HTTP.baseUrl, uploadConditions.HTTP.baseUrl, edm.decode(), str, upload_popcon.HTTP.token, uploadConditions.HTTP.token, and upload_popcon.ConditionsUploader.token.

288 
289  def getToken(self, username, password):
290 
291  url = self.baseUrl + 'token'
292 
293  self.curl.setopt(pycurl.URL, url)
294  self.curl.setopt(pycurl.VERBOSE, 0)
295 
296  #-toDo: check if/why these are needed ...
297  #-ap: hmm ...
298  # self.curl.setopt(pycurl.DNS_CACHE_TIMEOUT, 0)
299  # self.curl.setopt(pycurl.IPRESOLVE, pycurl.IPRESOLVE_V4)
300  #-end hmmm ...
301  #-review(2015-09-25): check and see - action: AP
302 
303  self.curl.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
304  # self.curl.setopt( self.curl.POST, {})
305  self.curl.setopt(self.curl.HTTPGET, 0)
306 
307  response = io.BytesIO()
308  self.curl.setopt(pycurl.WRITEFUNCTION, response.write)
309  self.curl.setopt(pycurl.USERPWD, '%s:%s' % (username, password) )
310  logging.debug('going to connect to server at: %s' % url )
311 
312  self.curl.perform()
313  code = self.curl.getinfo(pycurl.RESPONSE_CODE)
314  logging.debug('got: %s ', str(code))
315  if code in ( 502,503,504 ):
316  logging.debug('Trying again after %d seconds...', waitForRetry)
317  time.sleep( waitForRetry )
318  response = io.StringIO()
319  self.curl.setopt(pycurl.WRITEFUNCTION, response.write)
320  self.curl.setopt(pycurl.USERPWD, '%s:%s' % (username, password) )
321  self.curl.perform()
322  code = self.curl.getinfo(pycurl.RESPONSE_CODE)
323  resp = response.getvalue().decode('UTF-8')
324  errorMsg = None
325  if code==500 and not resp.find("INVALID_CREDENTIALS")==-1:
326  logging.error("Invalid credentials provided.")
327  return None
328  if code==403 and not resp.find("Unauthorized access")==-1:
329  logging.error("Unauthorized access. Please check the membership of group 'cms-cond-dropbox'")
330  return None
331  if code==200:
332  try:
333  self.token = json.loads( resp )['token']
334  except Exception as e:
335  errorMsg = 'Error while decoding returned json string'
336  logging.debug('http::getToken> error while decoding json: %s ', str(resp) )
337  logging.debug("error getting token: %s", str(e))
338  resp = None
339  else:
340  errorMsg = 'HTTP Error code %s ' %code
341  logging.debug('got: %s ', str(code))
342  logging.debug('http::getToken> got error from server: %s ', str(resp) )
343  resp = None
344  if resp is None:
345  raise Exception(errorMsg)
346 
347  logging.debug('token: %s', self.token)
348  logging.debug('returning: %s', response.getvalue().decode('UTF-8'))
349 
350  return response.getvalue()
edm::decode
bool decode(bool &, std::string const &)
Definition: types.cc:71
str
#define str(s)
Definition: TestProcessor.cc:53
def uploadConditions.HTTP.query (   self,
  url,
  data = None,
  files = None,
  keepCookies = True 
) 
Queries a URL, optionally with some data (dictionary).

If no data is specified, a GET request will be used.
If some data is specified, a POST request will be used.

If files is specified, it must be a dictionary like data but
the values are filenames.

By default, cookies are kept in-between requests.

A HTTPError exception is raised if the response's HTTP code is not 200.

Definition at line 351 of file uploadConditions.py.

References upload_popcon.HTTP.baseUrl, uploadConditions.HTTP.baseUrl, upload_popcon.HTTP.discardCookies(), uploadConditions.HTTP.discardCookies(), upload_popcon.HTTP.retries, uploadConditions.HTTP.retries, upload_popcon.HTTP.retryCodes, uploadConditions.HTTP.retryCodes, str, upload_popcon.HTTP.token, uploadConditions.HTTP.token, and upload_popcon.ConditionsUploader.token.

Referenced by production_tasks.BaseDataset.run(), and edmIntegrityCheck.IntegrityCheck.test().

351 
352  def query(self, url, data = None, files = None, keepCookies = True):
353  '''Queries a URL, optionally with some data (dictionary).
354 
355  If no data is specified, a GET request will be used.
356  If some data is specified, a POST request will be used.
357 
358  If files is specified, it must be a dictionary like data but
359  the values are filenames.
360 
361  By default, cookies are kept in-between requests.
362 
363  A HTTPError exception is raised if the response's HTTP code is not 200.
364  '''
365 
366  if not keepCookies:
367  self.discardCookies()
368 
369  url = self.baseUrl + url
370 
371  # make sure the logs are safe ... at least somewhat :)
372  data4log = copy.copy(data)
373  if data4log:
374  if 'password' in data4log.keys():
375  data4log['password'] = '*'
376 
377  retries = [0] + list(self.retries)
378 
379  while True:
380  logging.debug('Querying %s with data %s and files %s (retries left: %s, current sleep: %s)...', url, data4log, files, len(retries), retries[0])
381 
382  time.sleep(retries.pop(0))
383 
384  try:
385  self.curl.setopt(self.curl.URL, url)
386  self.curl.setopt(self.curl.HTTPGET, 1)
387 
388  # from now on we use the token we got from the login
389  self.curl.setopt(pycurl.USERPWD, '%s:""' % ( str(self.token), ) )
390  self.curl.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
391 
392  if data is not None or files is not None:
393  # If there is data or files to send, use a POST request
394 
395  finalData = {}
396 
397  if data is not None:
398  finalData.update(data)
399 
400  if files is not None:
401  for (key, fileName) in files.items():
402  finalData[key] = (self.curl.FORM_FILE, fileName)
403  self.curl.setopt( self.curl.HTTPPOST, list(finalData.items()) )
404 
405  self.curl.setopt(pycurl.VERBOSE, 0)
406 
407  response = io.BytesIO()
408  self.curl.setopt(self.curl.WRITEFUNCTION, response.write)
409  self.curl.perform()
410 
411  code = self.curl.getinfo(self.curl.RESPONSE_CODE)
412 
413  if code in self.retryCodes and len(retries) > 0:
414  logging.debug('Retrying since we got the %s error code...', code)
415  continue
416 
417  if code != 200:
418  raise HTTPError(code, response.getvalue())
419 
420  return response.getvalue()
421 
422  except pycurl.error as e:
423  if len(retries) == 0:
424  raise e
425  logging.debug('Retrying since we got the %s pycurl exception...', str(e))
426 
427 # common/http.py end
str
#define str(s)
Definition: TestProcessor.cc:53
def uploadConditions.HTTP.setBaseUrl (   self,
  baseUrl = '' 
) 
Allows to set a base URL which will be prefixed to all the URLs
that will be queried later.

Definition at line 254 of file uploadConditions.py.

References upload_popcon.HTTP.baseUrl, and uploadConditions.HTTP.baseUrl.

Referenced by uploadConditions.HTTP.__init__().

254 
255  def setBaseUrl(self, baseUrl = ''):
256  '''Allows to set a base URL which will be prefixed to all the URLs
257  that will be queried later.
258  '''
259  self.baseUrl = baseUrl
260 
def uploadConditions.HTTP.setProxy (   self,
  proxy = '' 
) 
Allows to set a proxy.

Definition at line 261 of file uploadConditions.py.

261 
262  def setProxy(self, proxy = ''):
263  '''Allows to set a proxy.
264  '''
265  self.curl.setopt(self.curl.PROXY, proxy)
266 
def uploadConditions.HTTP.setRetries (   self,
  retries = () 
) 
Allows to set retries.

The retries are a sequence of the seconds to wait per retry.

The retries are done on:
    * PyCurl errors (includes network problems, e.g. not being able
      to connect to the host).
    * 502 Bad Gateway (for the moment, to avoid temporary
      Apache-CherryPy issues).
    * 503 Service Temporarily Unavailable (for when we update
      the frontends).

Definition at line 273 of file uploadConditions.py.

Referenced by uploadConditions.HTTP.__init__().

273 
274  def setRetries(self, retries = ()):
275  '''Allows to set retries.
276 
277  The retries are a sequence of the seconds to wait per retry.
278 
279  The retries are done on:
280  * PyCurl errors (includes network problems, e.g. not being able
281  to connect to the host).
282  * 502 Bad Gateway (for the moment, to avoid temporary
283  Apache-CherryPy issues).
284  * 503 Service Temporarily Unavailable (for when we update
285  the frontends).
286  '''
287  self.retries = retries
def uploadConditions.HTTP.setTimeout (   self,
  timeout = 0 
) 
Allows to set a timeout.

Definition at line 267 of file uploadConditions.py.

267 
268  def setTimeout(self, timeout = 0):
269  '''Allows to set a timeout.
270  '''
271  self.curl.setopt(self.curl.TIMEOUT, timeout)
272 

Member Data Documentation

uploadConditions.HTTP.baseUrl

Definition at line 239 of file uploadConditions.py.

Referenced by uploadConditions.HTTP.getToken(), uploadConditions.HTTP.query(), and uploadConditions.HTTP.setBaseUrl().

uploadConditions.HTTP.curl

Definition at line 230 of file uploadConditions.py.

uploadConditions.HTTP.retries

Definition at line 286 of file uploadConditions.py.

Referenced by uploadConditions.HTTP.query().

tuple uploadConditions.HTTP.retryCodes = frozenset([502, 503])
static

Definition at line 224 of file uploadConditions.py.

Referenced by uploadConditions.HTTP.query().

uploadConditions.HTTP.token

Definition at line 241 of file uploadConditions.py.

Referenced by uploadConditions.HTTP.getToken(), uploadConditions.HTTP.query(), uploadConditions.ConditionsUploader.setHost(), uploadConditions.ConditionsUploader.signIn(), and uploadConditions.ConditionsUploader.signOut().

Generated for CMSSW Reference Manual by doxygen 1.8.5