CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes | Private Member Functions
uploadConditions.ConditionsUploader Class Reference
Inheritance diagram for uploadConditions.ConditionsUploader:

Public Member Functions

def __init__
 
def setHost
 
def signIn
 
def signInAgain
 
def signOut
 
def uploadFile
 

Public Attributes

 hostname
 
 http
 
 password
 
 token
 
 urlTemplate
 
 userName
 

Private Member Functions

def _checkForUpdates
 

Detailed Description

Upload conditions to the CMS conditions uploader service.

Definition at line 417 of file uploadConditions.py.

Constructor & Destructor Documentation

def uploadConditions.ConditionsUploader.__init__ (   self,
  hostname = defaultHostname,
  urlTemplate = defaultUrlTemplate 
)

Definition at line 421 of file uploadConditions.py.

421 
422  def __init__(self, hostname = defaultHostname, urlTemplate = defaultUrlTemplate):
423  self.hostname = hostname
424  self.urlTemplate = urlTemplate
425  self.userName = None
426  self.http = None
427  self.password = None

Member Function Documentation

def uploadConditions.ConditionsUploader._checkForUpdates (   self)
private 
Updates this script, if a new version is found.

Definition at line 467 of file uploadConditions.py.

References upload_popcon.ConditionsUploader.hostname, uploadConditions.ConditionsUploader.hostname, upload_popcon.ConditionsUploader.signOut(), and uploadConditions.ConditionsUploader.signOut().

467 
468  def _checkForUpdates(self):
469  '''Updates this script, if a new version is found.
470  '''
471 
472  logging.debug('%s: Checking if a newer version of this script is available ...', self.hostname)
473  version = int(self.http.query('getUploadScriptVersion'))
474 
475  if version <= __version__:
476  logging.debug('%s: Script is up-to-date.', self.hostname)
477  return
478 
479  logging.info('%s: Updating to a newer version (%s) than the current one (%s): downloading ...', self.hostname, version, __version__)
480 
481  uploadScript = self.http.query('getUploadScript')
482 
483  self.signOut()
484 
485  logging.info('%s: ... saving the new version ...', self.hostname)
486  with open(sys.argv[0], 'wb') as f:
487  f.write(uploadScript)
488 
489  logging.info('%s: ... executing the new version...', self.hostname)
490  os.execl(sys.executable, *([sys.executable] + sys.argv))
491 
def uploadConditions.ConditionsUploader.setHost (   self,
  hostname 
)

Definition at line 428 of file uploadConditions.py.

References upload_popcon.ConditionsUploader.hostname, and uploadConditions.ConditionsUploader.hostname.

428 
429  def setHost( self, hostname ):
430  self.hostname = hostname
def uploadConditions.ConditionsUploader.signIn (   self,
  username,
  password 
) 
init the server.

Definition at line 431 of file uploadConditions.py.

References upload_popcon.ConditionsUploader.hostname, uploadConditions.ConditionsUploader.hostname, upload_popcon.ConditionsUploader.http, uploadConditions.ConditionsUploader.http, upload_popcon.ConditionsUploader.urlTemplate, and uploadConditions.ConditionsUploader.urlTemplate.

Referenced by uploadConditions.ConditionsUploader.signInAgain(), and cmstc.TagCollector.signInInteractive().

431 
432  def signIn(self, username, password):
433  ''' init the server.
434  '''
435  self.http = HTTP()
436  self.http.setBaseUrl(self.urlTemplate % self.hostname)
437  '''Signs in the server.
438  '''
439 
440  logging.info('%s: Signing in user %s ...', self.hostname, username)
441  try:
442  self.token = self.http.getToken(username, password)
443  except Exception as e:
444  logging.error("Caught exception when trying to get token for user %s from %s: %s" % (username, self.hostname, str(e)) )
445  return False
446 
447  if not self.token:
448  logging.error("could not get token for user %s from %s" % (username, self.hostname) )
449  return False
450 
451  logging.debug( "got: '%s'", str(self.token) )
452  self.userName = username
453  self.password = password
454  return True
def uploadConditions.ConditionsUploader.signInAgain (   self)

Definition at line 455 of file uploadConditions.py.

References cond::auth::ServiceCredentials.password, cond::CredentialData.password, upload_popcon.ConditionsUploader.password, uploadConditions.ConditionsUploader.password, upload_popcon.ConditionsUploader.signIn(), uploadConditions.ConditionsUploader.signIn(), cond::auth::ServiceCredentials.userName, cond::CredentialData.userName, upload_popcon.ConditionsUploader.userName, and uploadConditions.ConditionsUploader.userName.

455 
456  def signInAgain(self):
457  return self.signIn( self.userName, self.password )
def uploadConditions.ConditionsUploader.signOut (   self) 
Signs out the server.

Definition at line 458 of file uploadConditions.py.

References upload_popcon.ConditionsUploader.hostname, uploadConditions.ConditionsUploader.hostname, cond::TagInfo_t.token, upload_popcon.HTTP.token, uploadConditions.HTTP.token, upload_popcon.ConditionsUploader.token, and uploadConditions.ConditionsUploader.token.

Referenced by uploadConditions.ConditionsUploader._checkForUpdates().

458 
459  def signOut(self):
460  '''Signs out the server.
461  '''
462 
463  logging.info('%s: Signing out...', self.hostname)
464  # self.http.query('logout')
465  self.token = None
466 
def uploadConditions.ConditionsUploader.uploadFile (   self,
  filename,
  backend = defaultBackend,
  temporaryFile = defaultTemporaryFile 
) 
Uploads a file to the dropBox.

The filename can be without extension, with .db or with .txt extension.
It will be stripped and then both .db and .txt files are used.

Definition at line 492 of file uploadConditions.py.

References uploadConditions.addToTarFile(), upload_popcon.ConditionsUploader.hostname, uploadConditions.ConditionsUploader.hostname, mps_monitormerge.items, cond::auth::ServiceCredentials.userName, cond::CredentialData.userName, upload_popcon.ConditionsUploader.userName, and uploadConditions.ConditionsUploader.userName.

492 
493  def uploadFile(self, filename, backend = defaultBackend, temporaryFile = defaultTemporaryFile):
494  '''Uploads a file to the dropBox.
495 
496  The filename can be without extension, with .db or with .txt extension.
497  It will be stripped and then both .db and .txt files are used.
498  '''
499 
500  basepath = filename.rsplit('.db', 1)[0].rsplit('.txt', 1)[0]
501  basename = os.path.basename(basepath)
502 
503  logging.debug('%s: %s: Creating tar file for upload ...', self.hostname, basename)
504 
505  try:
506  tarFile = tarfile.open(temporaryFile, 'w:bz2')
507 
508  with open('%s.db' % basepath, 'rb') as data:
509  addToTarFile(tarFile, data, 'data.db')
510  except Exception as e:
511  msg = 'Error when creating tar file. \n'
512  msg += 'Please check that you have write access to the directory you are running,\n'
513  msg += 'and that you have enough space on this disk (df -h .)\n'
514  logging.error(msg)
515  raise Exception(msg)
516 
517  with tempfile.NamedTemporaryFile() as metadata:
518  with open('%s.txt' % basepath, 'rb') as originalMetadata:
519  json.dump(json.load(originalMetadata), metadata, sort_keys = True, indent = 4)
520 
521  metadata.seek(0)
522  addToTarFile(tarFile, metadata, 'metadata.txt')
523 
524  tarFile.close()
525 
526  logging.debug('%s: %s: Calculating hash...', self.hostname, basename)
527 
528  fileHash = hashlib.sha1()
529  with open(temporaryFile, 'rb') as f:
530  while True:
531  data = f.read(4 * 1024 * 1024)
532  if not data:
533  break
534  fileHash.update(data)
535 
536  fileHash = fileHash.hexdigest()
537  fileInfo = os.stat(temporaryFile)
538  fileSize = fileInfo.st_size
539 
540  logging.debug('%s: %s: Hash: %s', self.hostname, basename, fileHash)
541 
542  logging.info('%s: %s: Uploading file (%s, size %s) to the %s backend...', self.hostname, basename, fileHash, fileSize, backend)
543  os.rename(temporaryFile, fileHash)
544  try:
545  ret = self.http.query('uploadFile',
546  {
547  'backend': backend,
548  'fileName': basename,
549  'userName': self.userName,
550  },
551  files = {
552  'uploadedFile': fileHash,
553  }
554  )
555  except Exception as e:
556  logging.error('Error from uploading: %s' % str(e))
557  ret = json.dumps( { "status": -1, "upload" : { 'itemStatus' : { basename : {'status':'failed', 'info':str(e)}}}, "error" : str(e)} )
558 
559  os.unlink(fileHash)
560 
561  statusInfo = json.loads(ret)['upload']
562  logging.debug( 'upload returned: %s', statusInfo )
563 
564  okTags = []
565  skippedTags = []
566  failedTags = []
567  for tag, info in statusInfo['itemStatus'].items():
568  logging.debug('checking tag %s, info %s', tag, str(json.dumps(info, indent=4,sort_keys=True)) )
569  if 'ok' in info['status'].lower() :
570  okTags.append( tag )
571  logging.info('tag %s successfully uploaded', tag)
572  if 'skip' in info['status'].lower() :
573  skippedTags.append( tag )
574  logging.warning('found tag %s to be skipped. reason: \n ... \t%s ', tag, info['info'])
575  if 'fail' in info['status'].lower() :
576  failedTags.append( tag )
577  logging.error('found tag %s failed to upload. reason: \n ... \t%s ', tag, info['info'])
578 
579  if len(okTags) > 0: logging.info ("tags sucessfully uploaded: %s ", str(okTags) )
580  if len(skippedTags) > 0: logging.warning("tags SKIPped to upload : %s ", str(skippedTags) )
581  if len(failedTags) > 0: logging.error ("tags FAILed to upload : %s ", str(failedTags) )
582 
583  fileLogURL = 'https://%s/logs/dropBox/getFileLog?fileHash=%s'
584  logging.info('file log at: %s', fileLogURL % (self.hostname,fileHash))
585 
586  return len(okTags)>0
mps_monitormerge.items
list items
Definition: mps_monitormerge.py:27

Member Data Documentation

uploadConditions.ConditionsUploader.hostname

Definition at line 422 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader._checkForUpdates(), uploadConditions.ConditionsUploader.setHost(), uploadConditions.ConditionsUploader.signIn(), uploadConditions.ConditionsUploader.signOut(), and uploadConditions.ConditionsUploader.uploadFile().

uploadConditions.ConditionsUploader.http

Definition at line 425 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader.signIn().

uploadConditions.ConditionsUploader.password

Definition at line 426 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader.signInAgain().

uploadConditions.ConditionsUploader.token

Definition at line 441 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader.signOut().

uploadConditions.ConditionsUploader.urlTemplate

Definition at line 423 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader.signIn().

uploadConditions.ConditionsUploader.userName

Definition at line 424 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader.signInAgain(), and uploadConditions.ConditionsUploader.uploadFile().

Generated for CMSSW Reference Manual by doxygen 1.8.5