Classes
class	ConditionsUploader

class	HTTP

class	HTTPError

Functions
def	addToTarFile (tarFile, fileobj, arcname)

def	get_version (url)

def	getCredentials (options)

def	getInput (default, prompt='')

def	getInputChoose (optionsList, default, prompt='')

def	getInputRepeat (prompt='')

def	getInputWorkflow (prompt='')

def	main ()

def	parse_arguments ()

def	re_upload (options)

def	run_upload (parameters)

def	runWizard (basename, dataFilename, metadataFilename)

def	testTier0Upload ()

def	upload (options, arguments)

def	uploadAllFiles (options, arguments)

def	uploadTier0Files (filenames, username, password, cookieFileName=None)

Variables
string	__author__ = 'Andreas Pfeiffer'

string	__copyright__ = 'Copyright 2015, CERN CMS'

list	__credits__ = ['Giacomo Govi', 'Salvatore Di Guida', 'Miguel Ojeda', 'Andreas Pfeiffer']

string	__email__ = 'giacomo.govi@cern.ch'

string	__license__ = 'Unknown'

string	__maintainer__ = 'Giacomo Govi'

int	__version__ = 1

string	authPathEnvVar = 'COND_AUTH_PATH'

string	CERN_SSO_CURL_CAPATH = '/etc/pki/tls/certs'

string	defaultBackend = 'online'

string	defaultDevHostname = 'cms-conddb-dev.cern.ch'

string	defaultHostname = 'cms-conddb-prod.cern.ch'

string	defaultNetrcHost = 'ConditionUploader'

string	defaultTemporaryFile = 'upload.tar.bz2'

string	defaultUrlTemplate = 'https://%s/cmsDbUpload/'

string	defaultWorkflow = 'offline'

string	devLogDbSrv = 'cms_orcoff_prep'

def	final_service_url = upload_metadata["server"]

string	horizontal_rule = "="*60

string	logDbSchema = 'CMS_COND_DROPBOX'

string	prodLogDbSrv = 'cms_orcoff_prod'

def	response = get_version(final_service_url)

def	server_version = response.json()

def	upload_metadata = parse_arguments()

dictionary	upload_metadata_argument = {}

int	waitForRetry = 15

Detailed Description

Primary Author:
Joshua Dawes - CERN, CMS - The University of Manchester

Debugging, Integration and Maintenance:
Andres Cardenas - CERN, CMS - Universidad San Francisco

Upload script wrapper - controls the automatic update system.

Note: the name of the file follows a different convention to the others because it should be the same as the current upload script name.

Takes user arguments and passes them to the main upload module CondDBFW.uploads, once the correct version exists.

1. Ask the server corresponding to the database we're uploading to which version of CondDBFW it has (query the /conddbfw_version/ url).
2. Decide which directory that we can write to - either the current local directory, or /tmp/random_string/.
3. Pull the commit returned from the server into the directory from step 2.
4. Invoke the CondDBFW.uploads module with the arguments given to this script.

Script that uploads to the new CMS conditions uploader.
Adapted to the new infrastructure from v6 of the upload.py script for the DropBox from Miguel Ojeda.

Function Documentation

◆ addToTarFile()

def uploadConditions.addToTarFile	(	tarFile,
		fileobj,
		arcname
	)

Definition at line 428 of file uploadConditions.py.

Referenced by uploadConditions.ConditionsUploader.uploadFile().

 def addToTarFile(tarFile, fileobj, arcname):
     tarInfo = tarFile.gettarinfo(fileobj = fileobj, arcname = arcname)
     tarInfo.mode = 0o400
     tarInfo.uid = tarInfo.gid = tarInfo.mtime = 0
     tarInfo.uname = tarInfo.gname = 'root'
     tarFile.addfile(tarInfo, fileobj)
 

◆ get_version()

def uploadConditions.get_version ( url )

Definition at line 435 of file uploadConditions.py.

Referenced by ValidationMatrix_v2.ReleaseComparison.compare().

 def get_version(url):
     return requests.get(url + "script_version/", verify=False)

◆ getCredentials()

def uploadConditions.getCredentials ( options )

Definition at line 617 of file uploadConditions.py.

References getInput().

Referenced by uploadAllFiles().

 def getCredentials( options ):
 
     username = None
     password = None
     netrcPath = None
     if authPathEnvVar in os.environ:
         authPath = os.environ[authPathEnvVar]
         netrcPath = os.path.join(authPath,'.netrc')
     if options.authPath is not None:
         netrcPath = os.path.join( options.authPath,'.netrc' )
     try:
         # Try to find the netrc entry
         (username, account, password) = netrc.netrc( netrcPath ).authenticators(options.netrcHost)
     except Exception:
         # netrc entry not found, ask for the username and password
         logging.info(
             'netrc entry "%s" not found: if you wish not to have to retype your password, you can add an entry in your .netrc file. However, beware of the risks of having your password stored as plaintext. Instead.',
             options.netrcHost)
 
         # Try to get a default username
         defaultUsername = getpass.getuser()
         if defaultUsername is None:
             defaultUsername = '(not found)'
 
         username = getInput(defaultUsername, '\nUsername [%s]: ' % defaultUsername)
         password = getpass.getpass('Password: ')
 
     return username, password
 
 

◆ getInput()

def uploadConditions.getInput	(	default,
		prompt = `''`
	)

Like raw_input() but with a default and automatic strip().

Like input() but with a default and automatic strip().

Definition at line 61 of file uploadConditions.py.

Referenced by getCredentials(), getInputChoose(), getInputWorkflow(), parse_arguments(), runWizard(), and uploadAllFiles().

 def getInput(default, prompt = ''):
     '''Like raw_input() but with a default and automatic strip().
     '''
 
     answer = raw_input(prompt)
     if answer:
         return answer.strip()
 
     return default.strip()
 
 

◆ getInputChoose()

def uploadConditions.getInputChoose	(	optionsList,
		default,
		prompt = `''`
	)

Makes the user choose from a list of options.

Definition at line 85 of file uploadConditions.py.

References getInput(), createfilelist.int, and print().

Referenced by runWizard().

 def getInputChoose(optionsList, default, prompt = ''):
     '''Makes the user choose from a list of options.
     '''
 
     while True:
         index = getInput(default, prompt)
 
         try:
             return optionsList[int(index)]
         except ValueError:
             print('Please specify an index of the list (i.e. integer).')
         except IndexError:
             print('The index you provided is not in the given list.')
 
 

◆ getInputRepeat()

def uploadConditions.getInputRepeat ( prompt = '' )

Like raw_input() but repeats if nothing is provided and automatic strip().

Like input() but repeats if nothing is provided and automatic strip().

Definition at line 100 of file uploadConditions.py.

References print().

Referenced by runWizard().

 def getInputRepeat(prompt = ''):
     '''Like raw_input() but repeats if nothing is provided and automatic strip().
     '''
 
     while True:
         answer = raw_input(prompt)
         if answer:
             return answer.strip()
 
         print('You need to provide a value.')
 

◆ getInputWorkflow()

def uploadConditions.getInputWorkflow ( prompt = '' )

Like getInput() but tailored to get target workflows (synchronization options).

Definition at line 72 of file uploadConditions.py.

References getInput(), and print().

 def getInputWorkflow(prompt = ''):
     '''Like getInput() but tailored to get target workflows (synchronization options).
     '''
 
     while True:
         workflow = getInput(defaultWorkflow, prompt)
 
         if workflow in frozenset(['offline', 'hlt', 'express', 'prompt', 'pcl']):
             return workflow
 
         print('Please specify one of the allowed workflows. See above for the explanation on each of them.')
 
 

◆ main()

def uploadConditions.main ( )

Entry point.

Definition at line 886 of file uploadConditions.py.

References print(), re_upload(), and upload().

 def main():
     '''Entry point.
     '''
 
     parser = optparse.OptionParser(usage =
         'Usage: %prog [options] <file> [<file> ...]\n'
     )
 
     parser.add_option('-d', '--debug',
         dest = 'debug',
         action="store_true",
         default = False,
         help = 'Switch on printing debug information. Default: %default',
     )
 
     parser.add_option('-b', '--backend',
         dest = 'backend',
         default = defaultBackend,
         help = 'dropBox\'s backend to upload to. Default: %default',
     )
 
     parser.add_option('-H', '--hostname',
         dest = 'hostname',
         default = defaultHostname,
         help = 'dropBox\'s hostname. Default: %default',
     )
 
     parser.add_option('-u', '--urlTemplate',
         dest = 'urlTemplate',
         default = defaultUrlTemplate,
         help = 'dropBox\'s URL template. Default: %default',
     )
 
     parser.add_option('-f', '--temporaryFile',
         dest = 'temporaryFile',
         default = defaultTemporaryFile,
         help = 'Temporary file that will be used to store the first tar file. Note that it then will be moved to a file with the hash of the file as its name, so there will be two temporary files created in fact. Default: %default',
     )
 
     parser.add_option('-n', '--netrcHost',
         dest = 'netrcHost',
         default = defaultNetrcHost,
         help = 'The netrc host (machine) from where the username and password will be read. Default: %default',
     )
 
     parser.add_option('-a', '--authPath',
         dest = 'authPath',
         default = None,
         help = 'The path of the .netrc file for the authentication. Default: $HOME',
     )
 
     parser.add_option('-r', '--reUpload',
         dest = 'reUpload',
         default = None,
         help = 'The hash of the file to upload again.',
     )
 
     (options, arguments) = parser.parse_args()
 
     logLevel = logging.INFO
     if options.debug:
         logLevel = logging.DEBUG
     logging.basicConfig(
         format = '[%(asctime)s] %(levelname)s: %(message)s',
         level = logLevel,
     )
 
     if len(arguments) < 1:
         if options.reUpload is None:
             parser.print_help()
             return -2
         else:
             return re_upload(options)
     if options.reUpload is not None:
         print("ERROR: options -r can't be specified on a new file upload.")
         return -2
 
     return upload(options, arguments)
 

◆ parse_arguments()

def uploadConditions.parse_arguments ( )

Definition at line 232 of file uploadConditions.py.

References beamvalidation.exit(), getInput(), join(), print(), RooUtil::StringUtil.rsplit(), runWizard(), and str.

 def parse_arguments():
     # read in command line arguments, and build metadata dictionary from them
     parser = argparse.ArgumentParser(prog="cmsDbUpload client", description="CMS Conditions Upload Script in CondDBFW.")
 
     parser.add_argument("--sourceDB", type=str, help="DB to find Tags, IOVs + Payloads in.", required=False)
 
     # metadata arguments
     parser.add_argument("--inputTag", type=str,\
                         help="Tag to take IOVs + Payloads from in --sourceDB.", required=False)
     parser.add_argument("--destinationTag", type=str,\
                         help="Tag to copy IOVs + Payloads to in --destDB.", required=False)
     parser.add_argument("--destinationDatabase", type=str,\
                         help="Database to copy IOVs + Payloads to.", required=False)
     parser.add_argument("--since", type=int,\
                         help="Since to take IOVs from.", required=False)
     parser.add_argument("--userText", type=str,\
                         help="Description of --destTag (can be empty).")
 
     # non-metadata arguments
     parser.add_argument("--metadataFile", "-m", type=str, help="Metadata file to take metadata from.", required=False)
 
     parser.add_argument("--debug", required=False, action="store_true")
     parser.add_argument("--verbose", required=False, action="store_true")
     parser.add_argument("--testing", required=False, action="store_true")
     parser.add_argument("--fcsr-filter", type=str, help="Synchronization to take FCSR from for local filtering of IOVs.", required=False)
 
     parser.add_argument("--netrc", required=False)
 
     parser.add_argument("--hashToUse", required=False)
 
     parser.add_argument("--server", required=False)
 
     parser.add_argument("--review-options", required=False, action="store_true")
 
     parser.add_argument("--replay-file", required=False)
 
     command_line_data = parser.parse_args()
 
     if command_line_data.replay_file:
         dictionary = json.loads("".join(open(command_line_data.replay_file, "r").readlines()))      command_line_data.tier0_response = dictionary["tier0_response"]
 
     # default is the production server, which can point to either database anyway
     server_alias_to_url = {
         "prep" : "https://cms-conddb-dev.cern.ch/cmsDbCondUpload/",
         "dev" : "https://cms-conddb-dev.cern.ch/cmsDbCondUpload/",
         "prod" : "https://cms-conddb.cern.ch/cmsDbCondUpload/"
     }
 
     # if prep, prod or None were given, convert to URLs in dictionary server_alias_to_url
     # if not, assume a URL has been given and use this instead
     if command_line_data.server in server_alias_to_url.keys():
         command_line_data.server = server_alias_to_url[command_line_data.server]
 
     # resolve destination databases
     database_alias_to_connection = {
         "prep": "oracle://cms_orcoff_prep/CMS_CONDITIONS",
         "dev": "oracle://cms_orcoff_prep/CMS_CONDITIONS",
         "prod": "oracle://cms_orcon_adg/CMS_CONDITIONS"
     }
     
     if command_line_data.destinationDatabase in database_alias_to_connection.keys():
         command_line_data.destinationDatabase = database_alias_to_connection[command_line_data.destinationDatabase]
 
 
     # use netrc to get username and password
     try:
         netrc_file = command_line_data.netrc
         netrc_authenticators = netrc.netrc(netrc_file).authenticators("ConditionUploader")
         if netrc_authenticators == None:
             print("Your netrc file must contain the key 'ConditionUploader'.")
             manual_input = raw_input("Do you want to try to type your credentials? ")
             if manual_input == "y":
                 # ask for username and password
                 username = raw_input("Username: ")
                 password = getpass.getpass("Password: ")
             else:
                 exit()
         else:
             print("Read your credentials from ~/.netrc.  If you want to use a different file, supply its name with the --netrc argument.")
             username = netrc_authenticators[0]
             password = netrc_authenticators[2]
     except:
         print("Couldn't obtain your credentials (either from netrc or manual input).")
         exit()
 
     command_line_data.username = username
     command_line_data.password = password
     # this will be used as the final destinationTags value by all input methods
     # apart from the metadata file
     command_line_data.destinationTags = {command_line_data.destinationTag:{}}
 
     """
     Construct metadata_dictionary:
     Currently, this is 3 cases:
 
     1) An IOV is being appended to an existing Tag with an existing Payload.
     In this case, we just take all data from the command line.
 
     2) No metadata file is given, so we assume that ALL upload metadata is coming from the command line.
 
     3) A metadata file is given, hence we parse the file, and then iterate through command line arguments
     since these override the options set in the metadata file.
 
     """
 
     # Hash to use, entirely from command line
     if command_line_data.hashToUse != None:
         command_line_data.userText = ""
         metadata_dictionary = command_line_data.__dict__
     elif command_line_data.metadataFile == None:
         if command_line_data.sourceDB != None and (command_line_data.inputTag == None or command_line_data.destinationTag == None or command_line_data.destinationDatabase == None):
             basepath = command_line_data.sourceDB.rsplit('.db', 1)[0].rsplit('.txt', 1)[0]
             basename = os.path.basename(basepath)
             dataFilename = '%s.db' % basepath
             metadataFilename = '%s.txt' % basepath
             # Data file
             try:
                 with open(dataFilename, 'rb') as dataFile:
                     pass
             except IOError as e:
                 errMsg = 'Impossible to open SQLite data file %s' %dataFilename
                 print( errMsg )
                 ret['status'] = -3
                 ret['error'] = errMsg
                 return ret
 
             # Metadata file
 
             try:
                 with open(metadataFilename, 'rb') as metadataFile:
                     pass
             except IOError as e:
                 if e.errno != errno.ENOENT:
                     errMsg = 'Impossible to open file %s (for other reason than not existing)' %metadataFilename
                     ret = {}
                     ret['status'] = -4
                     ret['error'] = errMsg
                     exit (ret)
 
                 if getInput('y', '\nIt looks like the metadata file %s does not exist and not enough parameters were received in the command line. Do you want me to create it and help you fill it?\nAnswer [y]: ' % metadataFilename).lower() != 'y':
                     errMsg = 'Metadata file %s does not exist' %metadataFilename
                     ret = {}
                     ret['status'] = -5
                     ret['error'] = errMsg
                     exit(ret)
                 # Wizard
                 runWizard(basename, dataFilename, metadataFilename)
             command_line_data.metadataFile = metadataFilename
         else:
             command_line_data.userText = command_line_data.userText\
                                         if command_line_data.userText != None\
                                         else str(raw_input("Tag's description [can be empty]:"))
             metadata_dictionary = command_line_data.__dict__
 
     if command_line_data.metadataFile != None:
         metadata_dictionary = json.loads("".join(open(os.path.abspath(command_line_data.metadataFile), "r").readlines()))
         metadata_dictionary["username"] = username
         metadata_dictionary["password"] = password
         metadata_dictionary["userText"] = metadata_dictionary.get("userText")\
                                             if metadata_dictionary.get("userText") != None\
                                             else str(raw_input("Tag's description [can be empty]:"))
 
         # go through command line options and, if they are set, overwrite entries
         for (option_name, option_value) in command_line_data.__dict__.items():
             # if the metadata_dictionary sets this, overwrite it
             if option_name != "destinationTags":
                 if option_value != None or (option_value == None and not(option_name in metadata_dictionary.keys())):
                     # if option_value has a value, override the metadata file entry
                     # or if option_value is None but the metadata file doesn't give a value,
                     # set the entry to None as well
                     metadata_dictionary[option_name] = option_value
             else:
                 if option_value != {None:{}}:
                     metadata_dictionary["destinationTags"] = {option_value:{}}
                 elif option_value == {None:{}} and not("destinationTags" in metadata_dictionary.keys()):
                     metadata_dictionary["destinationTags"] = {None:{}}
 
     if command_line_data.review_options:
         defaults = {
             "since" : "Since of first IOV",
             "userText" : "Populated by upload process",
             "netrc" : "None given",
             "fcsr_filter" : "Don't apply",
             "hashToUse" : "Using local SQLite file instead"
         }
         print("Configuration to use for the upload:")
         for key in metadata_dictionary:
             if not(key) in ["username", "password", "destinationTag"]:
                 value_to_print = metadata_dictionary[key] if metadata_dictionary[key] != None else defaults[key]
                 print("\t%s : %s" % (key, value_to_print))
 
         if raw_input("\nDo you want to continue? [y/n] ") != "y":
             exit()
 
     if metadata_dictionary["server"] == None:
         if metadata_dictionary["destinationDatabase"] == "oracle://cms_orcoff_prep/CMS_CONDITIONS":
             metadata_dictionary["server"] = server_alias_to_url["prep"]
         else:
             metadata_dictionary["server"] = server_alias_to_url["prod"]
 
     return metadata_dictionary
 

◆ re_upload()

def uploadConditions.re_upload ( options )

Definition at line 802 of file uploadConditions.py.

References edm.decode(), SiPixelLorentzAngle_cfi.read, str, and upload().

Referenced by main().

 def re_upload( options ):
     netrcPath = None
     logDbSrv = prodLogDbSrv
     if options.hostname == defaultDevHostname:
         logDbSrv = devLogDbSrv
     if options.authPath is not None:
         netrcPath = os.path.join( options.authPath,'.netrc' )
     try:
         netrcKey = '%s/%s' %(logDbSrv,logDbSchema)
         # Try to find the netrc entry
         (username, account, password) = netrc.netrc( netrcPath ).authenticators( netrcKey )
     except IOError as e:
         logging.error('Cannot access netrc file.')
         return 1
     except Exception as e:
         logging.error('Netrc file is invalid: %s' %str(e))
         return 1
     conStr = '%s/%s@%s' %(username,password,logDbSrv)
     con = cx_Oracle.connect( conStr )
     cur = con.cursor()
     fh = options.reUpload
     cur.execute('SELECT FILECONTENT, STATE FROM FILES WHERE FILEHASH = :HASH',{'HASH':fh})
     res = cur.fetchall()
     found = False
     fdata = None
     for r in res:
         found = True
         logging.info("Found file %s in state '%s;" %(fh,r[1]))
         fdata = r[0].read().decode('bz2')
     con.close()
     if not found:
         logging.error("No file uploaded found with hash %s" %fh)
         return 1
     # writing as a tar file and open it ( is there a why to open it in memory?)
     fname = '%s.tar' %fh
     with open(fname, "wb" ) as f:
         f.write(fdata)
     rname = 'reupload_%s' %fh
     with tarfile.open(fname) as tar:
         tar.extractall()
     os.remove(fname)
     dfile = 'data.db'
     mdfile = 'metadata.txt'
     if os.path.exists(dfile):
         os.utime(dfile,None)
         os.chmod(dfile,0o755)
         os.rename(dfile,'%s.db' %rname)
     else:
         logging.error('Tar file does not contain the data file')
         return 1
     if os.path.exists(mdfile):
         os.utime(mdfile,None)
         os.chmod(mdfile,0o755)
         mdata = None
         with open(mdfile) as md:
             mdata = json.load(md)
         datelabel = datetime.now().strftime("%y-%m-%d %H:%M:%S")
         if mdata is None:
             logging.error('Metadata file is empty.')
             return 1
         logging.debug('Preparing new metadata file...')
         mdata['userText'] = 'reupload %s : %s' %(datelabel,mdata['userText'])
         with open( '%s.txt' %rname, 'wb') as jf:
             jf.write( json.dumps( mdata, sort_keys=True, indent = 2 ) )
             jf.write('\n')
         os.remove(mdfile)
     else:
         logging.error('Tar file does not contain the metadata file')
         return 1
     logging.info('Files %s prepared for the upload.' %rname)
     arguments = [rname]
     return upload(options, arguments)
 

◆ run_upload()

def uploadConditions.run_upload ( parameters )

Imports CondDBFW.uploads and runs the upload with the upload metadata obtained.

Definition at line 47 of file uploadConditions.py.

References beamvalidation.exit().

 def run_upload(**parameters):
     """
     Imports CondDBFW.uploads and runs the upload with the upload metadata obtained.
     """
     try:
         import CondCore.Utilities.CondDBFW.uploads as uploads
     except Exception as e:
         traceback.print_exc()
         exit("CondDBFW or one of its dependencies could not be imported.\n"\
             + "If the CondDBFW directory exists, you are likely not in a CMSSW environment.")
     # we have CondDBFW, so just call the module with the parameters given in the command line
     uploader = uploads.uploader(**parameters)
     result = uploader.upload()
 

◆ runWizard()

def uploadConditions.runWizard	(	basename,
		dataFilename,
		metadataFilename
	)

Definition at line 111 of file uploadConditions.py.

References getInput(), getInputChoose(), getInputRepeat(), createfilelist.int, print(), and reco.zip().

Referenced by parse_arguments(), and uploadAllFiles().

 def runWizard(basename, dataFilename, metadataFilename):
     while True:
         print('''\nWizard for metadata for %s
 
 I will ask you some questions to fill the metadata file. For some of the questions there are defaults between square brackets (i.e. []), leave empty (i.e. hit Enter) to use them.''' % basename)
 
         # Try to get the available inputTags
         try:
             dataConnection = sqlite3.connect(dataFilename)
             dataCursor = dataConnection.cursor()
             dataCursor.execute('select name from sqlite_master where type == "table"')
             tables = set(zip(*dataCursor.fetchall())[0])
 
             # only conddb V2 supported...
             if 'TAG' in tables:
                 dataCursor.execute('select NAME from TAG')
             # In any other case, do not try to get the inputTags
             else:
                 raise Exception()
 
             inputTags = dataCursor.fetchall()
             if len(inputTags) == 0:
                 raise Exception()
             inputTags = list(zip(*inputTags))[0]
 
         except Exception:
             inputTags = []
 
         if len(inputTags) == 0:
             print('\nI could not find any input tag in your data file, but you can still specify one manually.')
 
             inputTag = getInputRepeat(
                 '\nWhich is the input tag (i.e. the tag to be read from the SQLite data file)?\ne.g. BeamSpotObject_ByRun\ninputTag: ')
 
         else:
             print('\nI found the following input tags in your SQLite data file:')
             for (index, inputTag) in enumerate(inputTags):
                 print('   %s) %s' % (index, inputTag))
 
             inputTag = getInputChoose(inputTags, '0',
                                       '\nWhich is the input tag (i.e. the tag to be read from the SQLite data file)?\ne.g. 0 (you select the first in the list)\ninputTag [0]: ')
 
         databases = {
             'oraprod': 'oracle://cms_orcon_prod/CMS_CONDITIONS',
             'prod': 'oracle://cms_orcon_prod/CMS_CONDITIONS',
             'oradev': 'oracle://cms_orcoff_prep/CMS_CONDITIONS',
             'prep': 'oracle://cms_orcoff_prep/CMS_CONDITIONS',
         }
 
         destinationDatabase = ''
         ntry = 0
         print('\nWhich is the destination database where the tags should be exported?')
         print('\n%s) %s' % ('oraprod', databases['oraprod']))
         print('\n%s) %s' % ('oradev', databases['oradev']))
             
         while ( destinationDatabase not in databases.values() ): 
             if ntry==0:
                 inputMessage = \
                 '\nPossible choices: oraprod or oradev \ndestinationDatabase: '
             elif ntry==1:
                 inputMessage = \
                 '\nPlease choose one of the two valid destinations: oraprod or oradev \ndestinationDatabase: '
             else:
                 raise Exception('No valid destination chosen. Bailing out...')
             
             databaseInput = getInputRepeat(inputMessage).lower()
             if databaseInput in databases.keys():
                 destinationDatabase = databases[databaseInput]
             ntry += 1
 
         while True:
             since = getInput('',
                              '\nWhich is the given since? (if not specified, the one from the SQLite data file will be taken -- note that even if specified, still this may not be the final since, depending on the synchronization options you select later: if the synchronization target is not offline, and the since you give is smaller than the next possible one (i.e. you give a run number earlier than the one which will be started/processed next in prompt/hlt/express), the DropBox will move the since ahead to go to the first safe run instead of the value you gave)\ne.g. 1234\nsince []: ')
             if not since:
                 since = None
                 break
             else:
                 try:
                     since = int(since)
                     break
                 except ValueError:
                     print('The since value has to be an integer or empty (null).')
 
         userText = getInput('',
                             '\nWrite any comments/text you may want to describe your request\ne.g. Muon alignment scenario for...\nuserText []: ')
 
         destinationTags = {}
         while True:
             destinationTag = getInput('',
                                       '\nWhich is the next destination tag to be added (leave empty to stop)?\ne.g. BeamSpotObjects_PCL_byRun_v0_offline\ndestinationTag []: ')
             if not destinationTag:
                 if len(destinationTags) == 0:
                     print('There must be at least one destination tag.')
                     continue
                 break
 
             if destinationTag in destinationTags:
                 print(
                     'You already added this destination tag. Overwriting the previous one with this new one.')
 
             destinationTags[destinationTag] = {
             }
 
         metadata = {
             'destinationDatabase': destinationDatabase,
             'destinationTags': destinationTags,
             'inputTag': inputTag,
             'since': since,
             'userText': userText,
         }
 
         metadata = json.dumps(metadata, sort_keys=True, indent=4)
         print('\nThis is the generated metadata:\n%s' % metadata)
 
         if getInput('n',
                     '\nIs it fine (i.e. save in %s and *upload* the conditions if this is the latest file)?\nAnswer [n]: ' % metadataFilename).lower() == 'y':
             break
     print('Saving generated metadata in %s...', metadataFilename)
     with open(metadataFilename, 'wb') as metadataFile:
         metadataFile.write(metadata)
 

◆ testTier0Upload()

def uploadConditions.testTier0Upload ( )

Definition at line 965 of file uploadConditions.py.

References uploadTier0Files().

 def testTier0Upload():
 
     global defaultNetrcHost
 
     (username, account, password) = netrc.netrc().authenticators(defaultNetrcHost)
 
     filenames = ['testFiles/localSqlite-top2']
 
     uploadTier0Files(filenames, username, password, cookieFileName = None)

◆ upload()

def uploadConditions.upload	(	options,
		arguments
	)

Definition at line 875 of file uploadConditions.py.

References print(), and uploadAllFiles().

Referenced by main(), and re_upload().

 def upload(options, arguments):
     results = uploadAllFiles(options, arguments)
 
     if 'status' not in results:
         print('Unexpected error.')
         return -1
     ret = results['status']
     print(results)
     print("upload ended with code: %s" %ret)
     return ret    
 

◆ uploadAllFiles()

def uploadConditions.uploadAllFiles	(	options,
		arguments
	)

Definition at line 647 of file uploadConditions.py.

References getCredentials(), getInput(), RooUtil::StringUtil.rsplit(), runWizard(), and str.

Referenced by upload().

 def uploadAllFiles(options, arguments):
     
     ret = {}
     ret['status'] = 0
 
     # Check that we can read the data and metadata files
     # If the metadata file does not exist, start the wizard
     for filename in arguments:
         basepath = filename.rsplit('.db', 1)[0].rsplit('.txt', 1)[0]
         basename = os.path.basename(basepath)
         dataFilename = '%s.db' % basepath
         metadataFilename = '%s.txt' % basepath
 
         logging.info('Checking %s...', basename)
 
         # Data file
         try:
             with open(dataFilename, 'rb') as dataFile:
                 pass
         except IOError as e:
             errMsg = 'Impossible to open SQLite data file %s' %dataFilename
             logging.error( errMsg )
             ret['status'] = -3
             ret['error'] = errMsg
             return ret
 
         # Check the data file
         empty = True
         try:
             dbcon = sqlite3.connect( dataFilename )
             dbcur = dbcon.cursor()
             dbcur.execute('SELECT * FROM IOV')
             rows = dbcur.fetchall()
             for r in rows:
                 empty = False
             dbcon.close()
             if empty:
                 errMsg = 'The input SQLite data file %s contains no data.' %dataFilename
                 logging.error( errMsg )
                 ret['status'] = -4
                 ret['error'] = errMsg
                 return ret
         except Exception as e:
             errMsg = 'Check on input SQLite data file %s failed: %s' %(dataFilename,str(e))
             logging.error( errMsg )
             ret['status'] = -5
             ret['error'] = errMsg
             return ret
 
         # Metadata file
         try:
             with open(metadataFilename, 'rb') as metadataFile:
                 pass
         except IOError as e:
             if e.errno != errno.ENOENT:
                 errMsg = 'Impossible to open file %s (for other reason than not existing)' %metadataFilename
                 logging.error( errMsg )
                 ret['status'] = -4
                 ret['error'] = errMsg
                 return ret
 
             if getInput('y', '\nIt looks like the metadata file %s does not exist. Do you want me to create it and help you fill it?\nAnswer [y]: ' % metadataFilename).lower() != 'y':
                 errMsg = 'Metadata file %s does not exist' %metadataFilename
                 logging.error( errMsg )
                 ret['status'] = -5
                 ret['error'] = errMsg
                 return ret
             # Wizard
             runWizard(basename, dataFilename, metadataFilename)
 
     # Upload files
     try:
         dropBox = ConditionsUploader(options.hostname, options.urlTemplate)
 
         # Authentication
         username, password = getCredentials(options)
 
         results = {}
         for filename in arguments:
             backend = options.backend
             basepath = filename.rsplit('.db', 1)[0].rsplit('.txt', 1)[0]
             metadataFilename = '%s.txt' % basepath
             with open(metadataFilename, 'rb') as metadataFile:
                 metadata = json.load( metadataFile )
             # When dest db = prep the hostname has to be set to dev.
             forceHost = False
             destDb = metadata['destinationDatabase']
             if destDb.startswith('oracle://cms_orcon_prod') or destDb.startswith('oracle://cms_orcoff_prep'):
                 hostName = defaultHostname
                 if destDb.startswith('oracle://cms_orcoff_prep'):
                      hostName = defaultDevHostname
                 dropBox.setHost( hostName )
                 authRet = dropBox.signIn( username, password )
                 if not authRet==0:
                     msg = "Error trying to connect to the server. Aborting."
                     if authRet==-2:
                         msg = "Error while signin in. Aborting."
                     logging.error(msg)
                     return { 'status' : authRet, 'error' : msg }
                 results[filename] = dropBox.uploadFile(filename, options.backend, options.temporaryFile)
             else:
                 results[filename] = False
                 logging.error("DestinationDatabase %s is not valid. Skipping the upload." %destDb)
             if not results[filename]:
                 if ret['status']<0:
                     ret['status'] = 0
                 ret['status'] += 1
         ret['files'] = results
         logging.debug("all files processed, logging out now.")
 
         dropBox.signOut()
 
     except HTTPError as e:
         logging.error('got HTTP error: %s', str(e))
         return { 'status' : -1, 'error' : str(e) }
 
     return ret
 

◆ uploadTier0Files()

def uploadConditions.uploadTier0Files	(	filenames,
		username,
		password,
		cookieFileName = `None`
	)

Uploads a bunch of files coming from Tier0.
This has the following requirements:
    * Username/Password based authentication.
    * Uses the online backend.
    * Ignores errors related to the upload/content (e.g. duplicated file).

Definition at line 765 of file uploadConditions.py.

Referenced by testTier0Upload().

 def uploadTier0Files(filenames, username, password, cookieFileName = None):
     '''Uploads a bunch of files coming from Tier0.
     This has the following requirements:
         * Username/Password based authentication.
         * Uses the online backend.
         * Ignores errors related to the upload/content (e.g. duplicated file).
     '''
 
     dropBox = ConditionsUploader()
 
     dropBox.signIn(username, password)
 
     for filename in filenames:
         try:
             result = dropBox.uploadFile(filename, backend = 'test')
         except HTTPError as e:
             if e.code == 400:
                 # 400 Bad Request: This is an exception related to the upload
                 # being wrong for some reason (e.g. duplicated file).
                 # Since for Tier0 this is not an issue, continue
                 logging.error('HTTP Exception 400 Bad Request: Upload-related, skipping. Message: %s', e)
                 continue
 
             # In any other case, re-raise.
             raise
 
         #-toDo: add a flag to say if we should retry or not. So far, all retries are done server-side (Tier-0),
         #       if we flag as failed any retry would not help and would result in the same error (e.g.
         #       when a file with an identical hash is uploaded again)
         #-review(2015-09-25): get feedback from tests at Tier-0 (action: AP)
 
         if not result: # dropbox reported an error when uploading, do not retry.
             logging.error('Error from dropbox, upload-related, skipping.')
             continue
 
     dropBox.signOut()
 

Variable Documentation

◆ author

string uploadConditions.__author__ = 'Andreas Pfeiffer'

private

Definition at line 7 of file uploadConditions.py.

◆ copyright

private

Definition at line 8 of file uploadConditions.py.

◆ credits

list uploadConditions.__credits__ = ['Giacomo Govi', 'Salvatore Di Guida', 'Miguel Ojeda', 'Andreas Pfeiffer']

private

Definition at line 9 of file uploadConditions.py.

◆ email

string uploadConditions.__email__ = 'giacomo.govi@cern.ch'

private

Definition at line 12 of file uploadConditions.py.

◆ license

string uploadConditions.__license__ = 'Unknown'

private

Definition at line 10 of file uploadConditions.py.

◆ maintainer

string uploadConditions.__maintainer__ = 'Giacomo Govi'

private

Definition at line 11 of file uploadConditions.py.

◆ version

int uploadConditions.__version__ = 1

private

Definition at line 22 of file uploadConditions.py.

◆ authPathEnvVar

string uploadConditions.authPathEnvVar = 'COND_AUTH_PATH'

Definition at line 40 of file uploadConditions.py.

◆ CERN_SSO_CURL_CAPATH

string uploadConditions.CERN_SSO_CURL_CAPATH = '/etc/pki/tls/certs'

Definition at line 218 of file uploadConditions.py.

◆ defaultBackend

string uploadConditions.defaultBackend = 'online'

Definition at line 30 of file uploadConditions.py.

◆ defaultDevHostname

string uploadConditions.defaultDevHostname = 'cms-conddb-dev.cern.ch'

Definition at line 32 of file uploadConditions.py.

◆ defaultHostname

string uploadConditions.defaultHostname = 'cms-conddb-prod.cern.ch'

Definition at line 31 of file uploadConditions.py.

◆ defaultNetrcHost

string uploadConditions.defaultNetrcHost = 'ConditionUploader'

Definition at line 35 of file uploadConditions.py.

◆ defaultTemporaryFile

string uploadConditions.defaultTemporaryFile = 'upload.tar.bz2'

Definition at line 34 of file uploadConditions.py.

◆ defaultUrlTemplate

string uploadConditions.defaultUrlTemplate = 'https://%s/cmsDbUpload/'

Definition at line 33 of file uploadConditions.py.

◆ defaultWorkflow

string uploadConditions.defaultWorkflow = 'offline'

Definition at line 36 of file uploadConditions.py.

◆ devLogDbSrv

string uploadConditions.devLogDbSrv = 'cms_orcoff_prep'

Definition at line 38 of file uploadConditions.py.

◆ final_service_url

def uploadConditions.final_service_url = upload_metadata["server"]

Definition at line 444 of file uploadConditions.py.

◆ horizontal_rule

string uploadConditions.horizontal_rule = "="*60

Definition at line 45 of file uploadConditions.py.

◆ logDbSchema

string uploadConditions.logDbSchema = 'CMS_COND_DROPBOX'

Definition at line 39 of file uploadConditions.py.

◆ prodLogDbSrv

string uploadConditions.prodLogDbSrv = 'cms_orcoff_prod'

Definition at line 37 of file uploadConditions.py.

◆ response

def uploadConditions.response = get_version(final_service_url)

Definition at line 446 of file uploadConditions.py.

Referenced by JetTester.analyze(), JetTester_HeavyIons.analyze(), evf::EvFDaqDirector.contactFileBroker(), HBHEDarkening.degradation(), HLTTauCertifier.dqmEndJob(), PFJetAnalyzerDQM.fillJetResponse(), PFJetFilter.filter(), GBRForest.GetGradBoostClassifier(), HCALResponse.getHCALEnergyResponse(), GBRForest.GetResponse(), GBRForestD.GetResponse(), XrdAdaptor::Source.getXrootdSiteFromURL(), XrdAdaptor::ClientRequest.HandleResponse(), SendMonitoringInfoHandler.HandleResponse(), QueryAttrHandler.HandleResponse(), L1TPhase2CorrelatorOffline.medianResponseCorrResolution(), PhotonMVAEstimator.mvaValue(), ElectronMVAEstimatorRun2.mvaValue(), cms::MuonTCMETValueMapProducer.produce(), MVAValueMapProducer< ParticleType >.produce(), GBRForest.SetInitialResponse(), and GBRForestD.SetInitialResponse().

◆ server_version

def uploadConditions.server_version = response.json()

Definition at line 447 of file uploadConditions.py.

Referenced by evf::EvFDaqDirector.contactFileBroker().

◆ upload_metadata

def uploadConditions.upload_metadata = parse_arguments()

Definition at line 441 of file uploadConditions.py.

◆ upload_metadata_argument

dictionary uploadConditions.upload_metadata_argument = {}

Definition at line 473 of file uploadConditions.py.

◆ waitForRetry

int uploadConditions.waitForRetry = 15

Definition at line 41 of file uploadConditions.py.

Classes

Functions

Variables

Detailed Description

Function Documentation

◆ addToTarFile()

◆ get_version()

◆ getCredentials()

◆ getInput()

◆ getInputChoose()

◆ getInputRepeat()

◆ getInputWorkflow()

◆ main()

◆ parse_arguments()

◆ re_upload()

◆ run_upload()

◆ runWizard()

◆ testTier0Upload()

◆ upload()

◆ uploadAllFiles()

◆ uploadTier0Files()

Variable Documentation

◆ __author__

◆ __copyright__

◆ __credits__

◆ __email__

◆ __license__

◆ __maintainer__

◆ __version__

◆ authPathEnvVar

◆ CERN_SSO_CURL_CAPATH

◆ defaultBackend

◆ defaultDevHostname

◆ defaultHostname

◆ defaultNetrcHost

◆ defaultTemporaryFile

◆ defaultUrlTemplate

◆ defaultWorkflow

◆ devLogDbSrv

◆ final_service_url

◆ horizontal_rule

◆ logDbSchema

◆ prodLogDbSrv

◆ response

◆ server_version

◆ upload_metadata

◆ upload_metadata_argument

◆ waitForRetry

◆ author

◆ copyright

◆ credits

◆ email

◆ license

◆ maintainer

◆ version