tier0.Tier0Handler Class Reference

Inheritance diagram for tier0.Tier0Handler:

Public Member Functions
def	__init__ (self, uri, timeOut, maxTime, retries, retryPeriod, proxy, debug)
def	getFirstSafeRun (self)
def	getGlobalTag (self, config)
def	setDebug (self)
def	setProxy (self, proxy)
def	unsetDebug (self)

Private Member Functions
def	_curlQueryTier0
def	_getCerts (self)
def	_queryTier0DataSvc (self, url)

Private Attributes
	_debug
	_maxTime
	_proxy
	_retries
	_retryPeriod
	_timeOut
	_uri

Detailed Description

Definition at line 71 of file tier0.py.

Constructor & Destructor Documentation

__init__()

def tier0.Tier0Handler.__init__	(		self,
			uri,
			timeOut,
			maxTime,
			retries,
			retryPeriod,
			proxy,
			debug
	)

Parameters:
uri: Tier0DataSvc URI;
timeOut: time out for connection of Tier0DataSvc HTTPS calls [seconds];
maxTime: maximum time for Tier0DataSvc HTTPS calls (including data transfer) [seconds];
retries: maximum retries for Tier0DataSvc HTTPS calls;
retryPeriod: sleep time between two Tier0DataSvc HTTPS calls [seconds];
proxy: HTTP proxy for accessing Tier0DataSvc HTTPS calls;
debug: if set to True, enables debug information.

Definition at line 73 of file tier0.py.

    def __init__( self, uri, timeOut, maxTime, retries, retryPeriod, proxy, debug ):
        """
        Parameters:
        uri: Tier0DataSvc URI;
        timeOut: time out for connection of Tier0DataSvc HTTPS calls [seconds];
        maxTime: maximum time for Tier0DataSvc HTTPS calls (including data transfer) [seconds];
        retries: maximum retries for Tier0DataSvc HTTPS calls;
        retryPeriod: sleep time between two Tier0DataSvc HTTPS calls [seconds];
        proxy: HTTP proxy for accessing Tier0DataSvc HTTPS calls;
        debug: if set to True, enables debug information.
        """
        self._uri = uri
        self._timeOut = timeOut
        self._maxTime = maxTime
        self._retries = retries
        self._retryPeriod = retryPeriod
        self._proxy = proxy
        self._debug = debug

Member Function Documentation

_curlQueryTier0()

def tier0.Tier0Handler._curlQueryTier0 (   self,   url  )

private

Definition at line 116 of file tier0.py.

References EcalMatacqAnalyzer._debug, EcalABAnalyzer._debug, tier0.Tier0Handler._debug, uploads.uploader._debug, EcalLaserAnalyzer2._debug, EcalLaserAnalyzer._debug, tier0.Tier0Handler._getCerts(), and tier0.Tier0Handler._proxy.

Referenced by tier0.Tier0Handler._queryTier0DataSvc().

    def _curlQueryTier0( self, url:str, force_debug:bool = False, force_cert:bool = False):
        userAgent = "User-Agent: ConditionWebServices/1.0 python/%d.%d.%d PycURL/%s" \
            % ( sys.version_info[ :3 ] + ( pycurl.version_info()[ 1 ], ) )
        debug = "-v" if self._debug or force_debug else "-s -S"

        proxy = f"--proxy {self._proxy}" if self._proxy else ""
        certs = self._getCerts() if not self._proxy or force_cert else ""
        
        cmd = f'/usr/bin/curl -k -L --user-agent "{userAgent}" {proxy}'\
              f' --connect-timeout {self._timeOut} --max-time {self._maxTime} --retry {self._retries}'\
              f' {debug} {url} {certs}'

        # time the curl to understand if re-tries have been carried out
        start = time.time()
        process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        (stdoutdata, stderrdata) =  process.communicate()
        end = time.time()
        return process.returncode, stdoutdata, stderrdata, end-start

_getCerts()

def tier0.Tier0Handler._getCerts (   self,   str  )

private

Definition at line 101 of file tier0.py.

Referenced by tier0.Tier0Handler._curlQueryTier0().

    def _getCerts( self ) -> str:
        cert_path = os.getenv('X509_USER_CERT', '')
        key_path = os.getenv('X509_USER_KEY', '')
        
        certs = ""
        if cert_path:
            certs += f' --cert {cert_path}'
        else:
            logging.warning("No certificate provided for Tier0 access, use X509_USER_CERT and"
                            " optionally X509_USER_KEY env variables to specify the path to the cert"
                            " (and the key unless included in the cert file)")
        if key_path:
            certs += f' --key {key_path}'
        return certs

_queryTier0DataSvc()

def tier0.Tier0Handler._queryTier0DataSvc (   self,   url  )

private

Queries Tier0DataSvc.
url: Tier0DataSvc URL.
@returns: dictionary, from whence the required information must be retrieved according to the API call.
Raises if connection error, bad response, or timeout after retries occur.

Definition at line 135 of file tier0.py.

References tier0.Tier0Handler._curlQueryTier0(), tier0.Tier0Handler._proxy, tier0.Tier0Handler._retryPeriod, join(), python.rootplot.root2matplotlib.replace(), and str.

Referenced by FrontierCondition_GT_autoExpress_cfi.Tier0Handler.getFirstSafeRun(), tier0.Tier0Handler.getFirstSafeRun(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.getGlobalTag(), and tier0.Tier0Handler.getGlobalTag().

    def _queryTier0DataSvc( self, url ):
        """
        Queries Tier0DataSvc.
        url: Tier0DataSvc URL.
        @returns: dictionary, from whence the required information must be retrieved according to the API call.
        Raises if connection error, bad response, or timeout after retries occur.
        """

        retcode, stdoutdata, stderrdata, query_time = self._curlQueryTier0(url)

        if retcode != 0 or stderrdata:

            # if the first curl has failed, logg its stderror and prepare and independent retry
            msg = "looks like curl returned an error: retcode=%s and took %s seconds" % (retcode, query_time,)
            msg += ' msg = "'+str(stderrdata)+'"'
            logging.error(msg)
            if self._proxy:
                logging.info("before assumed proxy provides authentication, now trying with both proxy and certificate")
                
            time.sleep(self._retryPeriod)
            retcode, stdoutdata, stderrdata, query_time = self._curlQueryTier0(url, force_debug=True, force_cert=True)
            if retcode != 0:
                msg = "looks like curl returned an error for the second time: retcode=%s" % (retcode,)
                msg += ' msg = "'+str(stderrdata)+'"'
                logging.error(msg)
                raise Tier0Error(msg)
            else:
                msg = "curl returned ok upon the second try"
                logging.info(msg)
        resp = json.loads( ''.join(stdoutdata.decode()).replace( "'", '"').replace(' None', ' "None"') )
        return resp

getFirstSafeRun()

def tier0.Tier0Handler.getFirstSafeRun

(

self

)

Queries Tier0DataSvc to get the first condition safe run.
Parameters:
@returns: integer, the run number.
Raises if connection error, bad response, timeout after retries occur, or if the run number is not available.

Definition at line 168 of file tier0.py.

References tier0.Tier0Handler._proxy, tier0.Tier0Handler._queryTier0DataSvc(), tier0.Tier0Handler._uri, createfilelist.int, and str.

    def getFirstSafeRun( self ):
        """
        Queries Tier0DataSvc to get the first condition safe run.
        Parameters:
        @returns: integer, the run number.
        Raises if connection error, bad response, timeout after retries occur, or if the run number is not available.
        """
        firstConditionSafeRunAPI = "firstconditionsaferun"
        safeRunDict = self._queryTier0DataSvc( os.path.join( self._uri, firstConditionSafeRunAPI ) )
        if safeRunDict is None:
            errStr = """First condition safe run is not available in Tier0DataSvc from URL \"%s\"""" \
                %( os.path.join( self._uri, firstConditionSafeRunAPI ), )
            if self._proxy:
                errStr += """ using proxy \"%s\".""" %( str( self._proxy ), )
            raise Tier0Error( errStr )
        return int(safeRunDict['result'][0])

getGlobalTag()

def tier0.Tier0Handler.getGlobalTag	(		self,
			config
	)

Queries Tier0DataSvc to get the most recent Global Tag for a given workflow.
Parameters:
config: Tier0DataSvc API call for the workflow to be looked for;
@returns: a string with the Global Tag name.
Raises if connection error, bad response, timeout after retries occur, or if no Global Tags are available.

Definition at line 185 of file tier0.py.

References tier0.Tier0Handler._proxy, tier0.Tier0Handler._queryTier0DataSvc(), tier0.Tier0Handler._uri, str, and tier0.unique().

    def getGlobalTag( self, config ):
        """
        Queries Tier0DataSvc to get the most recent Global Tag for a given workflow.
        Parameters:
        config: Tier0DataSvc API call for the workflow to be looked for;
        @returns: a string with the Global Tag name.
        Raises if connection error, bad response, timeout after retries occur, or if no Global Tags are available.
        """
        data = self._queryTier0DataSvc( os.path.join( self._uri, config ) )
        gtnames = sorted(unique( [ str( di['global_tag'] ) for di in data['result'] if di['global_tag'] is not None ] ))
        try:
            recentGT = gtnames[-1]
            return recentGT
        except IndexError:
            errStr = """No Global Tags for \"%s\" are available in Tier0DataSvc from URL \"%s\"""" \
                %( config, os.path.join( self._uri, config ) )
            if self._proxy:
                errStr += """ using proxy \"%s\".""" %( str( self._proxy ), )
        raise Tier0Error( errStr )

setDebug()

def tier0.Tier0Handler.setDebug

(

self

)

Definition at line 92 of file tier0.py.

References EcalMatacqAnalyzer._debug, EcalABAnalyzer._debug, tier0.Tier0Handler._debug, uploads.uploader._debug, EcalLaserAnalyzer2._debug, and EcalLaserAnalyzer._debug.

    def setDebug( self ):
        self._debug = True

setProxy()

def tier0.Tier0Handler.setProxy	(		self,
			proxy
	)

Definition at line 98 of file tier0.py.

References tier0.Tier0Handler._proxy.

    def setProxy( self, proxy ):
        self._proxy = proxy

unsetDebug()

def tier0.Tier0Handler.unsetDebug

(

self

)

Definition at line 95 of file tier0.py.

References EcalMatacqAnalyzer._debug, EcalABAnalyzer._debug, tier0.Tier0Handler._debug, uploads.uploader._debug, EcalLaserAnalyzer2._debug, and EcalLaserAnalyzer._debug.

    def unsetDebug( self ):
        self._debug = False

Member Data Documentation

_debug

tier0.Tier0Handler._debug

private

Definition at line 90 of file tier0.py.

Referenced by tier0.Tier0Handler._curlQueryTier0(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler._queryTier0DataSvc(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.setDebug(), tier0.Tier0Handler.setDebug(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.unsetDebug(), and tier0.Tier0Handler.unsetDebug().

_maxTime

tier0.Tier0Handler._maxTime

private

Definition at line 86 of file tier0.py.

_proxy

tier0.Tier0Handler._proxy

private

Definition at line 89 of file tier0.py.

Referenced by tier0.Tier0Handler._curlQueryTier0(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler._queryTier0DataSvc(), tier0.Tier0Handler._queryTier0DataSvc(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.getFirstSafeRun(), tier0.Tier0Handler.getFirstSafeRun(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.getGlobalTag(), tier0.Tier0Handler.getGlobalTag(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.setProxy(), and tier0.Tier0Handler.setProxy().

_retries

tier0.Tier0Handler._retries

private

Definition at line 87 of file tier0.py.

Referenced by FrontierCondition_GT_autoExpress_cfi.Tier0Handler._queryTier0DataSvc().

_retryPeriod

tier0.Tier0Handler._retryPeriod

private

Definition at line 88 of file tier0.py.

Referenced by tier0.Tier0Handler._queryTier0DataSvc().

_timeOut

tier0.Tier0Handler._timeOut

private

Definition at line 85 of file tier0.py.

Referenced by FrontierCondition_GT_autoExpress_cfi.Tier0Handler._queryTier0DataSvc().

_uri

tier0.Tier0Handler._uri

private

Definition at line 84 of file tier0.py.

Referenced by FrontierCondition_GT_autoExpress_cfi.Tier0Handler.getFirstSafeRun(), tier0.Tier0Handler.getFirstSafeRun(), FrontierCondition_GT_autoExpress_cfi.Tier0Handler.getGlobalTag(), and tier0.Tier0Handler.getGlobalTag().