------------------------------------------------------ Automatic pick-up of RelVal input files ------------------------------------------------------ More...

Inherits FWCore::GuiBrowsers::ConfigToolBase::ConfigToolBase.

Public Member Functions
def	__call__
def	__init__
def	apply
def	getDefaultParameters
def	messageEmptyList
Private Attributes
	_comment
	_parameters
Static Private Attributes
tuple	_defaultParameters = dicttypes.SortedKeysDict()
string	_label = 'pickRelValInputFiles'

Detailed Description

------------------------------------------------------ Automatic pick-up of RelVal input files ------------------------------------------------------

Picks up RelVal input files automatically and
  returns a vector of strings with the paths to be used in [PoolSource].fileNames
PickRelValInputFiles( cmsswVersion, relVal, dataTier, condition, globalTag, maxVersions, skipFiles, numberOfFiles, debug )
- cmsswVersion : CMSSW release to pick up the RelVal files from
                 optional; default: the current release (determined automatically from environment)
- formerVersion: use the last before the last valid CMSSW release to pick up the RelVal files from
                 applies also, if 'cmsswVersion' is set explicitly
                 optional; default: False
- relVal       : RelVal sample to be used
                 optional; default: 'RelValTTbar'
- dataTier     : data tier to be used
                 optional; default: 'GEN-SIM-RECO'
- condition    : identifier of GlobalTag as defined in Configurations/PyReleaseValidation/python/autoCond.py
                 possibly overwritten, if 'globalTag' is set explicitly
                 optional; default: 'startup'
- globalTag    : name of GlobalTag as it is used in the data path of the RelVals
                 optional; default: determined automatically as defined by 'condition' in Configurations/PyReleaseValidation/python/autoCond.py
  !!!            Determination is done for the release one runs in, not for the release the RelVals have been produced in.
  !!!            Example of deviation: data RelVals (CMSSW_4_1_X) might not only have the pure name of the GlobalTag 'GR_R_311_V2' in the full path,
                 but also an extension identifying the data: 'GR_R_311_V2_RelVal_wzMu2010B'
- maxVersions  : max. versioning number of RelVal to check
                 optional; default: 9
- skipFiles    : number of files to skip for a found RelVal sample
                 optional; default: 0
- numberOfFiles: number of files to pick up
                 setting it to 0, returns all found ('skipFiles' remains active though)
                 optional; default: 1
- debug        : switch to enable enhanced messages in 'stdout'
                 optional; default: False

Definition at line 764 of file cmsswVersionTools.py.

Constructor & Destructor Documentation

def cmsswVersionTools::PickRelValInputFiles::__init__ ( self )

Definition at line 802 of file cmsswVersionTools.py.

00803                         :
00804         ConfigToolBase.__init__( self )
00805         self.addParameter( self._defaultParameters, 'cmsswVersion' , os.getenv( "CMSSW_VERSION" )                                        , 'auto from environment' )
00806         self.addParameter( self._defaultParameters, 'formerVersion', False                                                               , '' )
00807         self.addParameter( self._defaultParameters, 'relVal'       , 'RelValTTbar'                                                       , '' )
00808         self.addParameter( self._defaultParameters, 'dataTier'     , 'GEN-SIM-RECO'                                                      , '' )
00809         self.addParameter( self._defaultParameters, 'condition'    , 'startup'                                                           , '' )
00810         self.addParameter( self._defaultParameters, 'globalTag'    , autoCond[ self.getDefaultParameters()[ 'condition' ].value ][ : -5 ], 'auto from \'condition\'' )
00811         self.addParameter( self._defaultParameters, 'maxVersions'  , 9                                                                   , '' )
00812         self.addParameter( self._defaultParameters, 'skipFiles'    , 0                                                                   , '' )
00813         self.addParameter( self._defaultParameters, 'numberOfFiles', 1                                                                   , '' )
00814         self.addParameter( self._defaultParameters, 'debug'        , False                                                               , '' )
00815         self._parameters = copy.deepcopy( self._defaultParameters )
00816         self._comment = ""

Member Function Documentation

def cmsswVersionTools::PickRelValInputFiles::__call__	(	self,
		cmsswVersion = `None`,
		formerVersion = `None`,
		relVal = `None`,
		dataTier = `None`,
		condition = `None`,
		globalTag = `None`,
		maxVersions = `None`,
		skipFiles = `None`,
		numberOfFiles = `None`,
		debug = `None`
	)

Definition at line 817 of file cmsswVersionTools.py.

00829                  :
00830         if cmsswVersion is None:
00831             cmsswVersion = self.getDefaultParameters()[ 'cmsswVersion' ].value
00832         if formerVersion is None:
00833             formerVersion = self.getDefaultParameters()[ 'formerVersion' ].value
00834         if relVal is None:
00835             relVal = self.getDefaultParameters()[ 'relVal' ].value
00836         if dataTier is None:
00837             dataTier = self.getDefaultParameters()[ 'dataTier' ].value
00838         if condition is None:
00839             condition = self.getDefaultParameters()[ 'condition' ].value
00840         if globalTag is None:
00841             globalTag = autoCond[ condition ][ : -5 ] # auto from 'condition'
00842         if maxVersions is None:
00843             maxVersions = self.getDefaultParameters()[ 'maxVersions' ].value
00844         if skipFiles is None:
00845             skipFiles = self.getDefaultParameters()[ 'skipFiles' ].value
00846         if numberOfFiles is None:
00847             numberOfFiles = self.getDefaultParameters()[ 'numberOfFiles' ].value
00848         if debug is None:
00849             debug = self.getDefaultParameters()[ 'debug' ].value
00850         self.setParameter( 'cmsswVersion' , cmsswVersion )
00851         self.setParameter( 'formerVersion', formerVersion )
00852         self.setParameter( 'relVal'       , relVal )
00853         self.setParameter( 'dataTier'     , dataTier )
00854         self.setParameter( 'condition'    , condition )
00855         self.setParameter( 'globalTag'    , globalTag )
00856         self.setParameter( 'maxVersions'  , maxVersions )
00857         self.setParameter( 'skipFiles'    , skipFiles )
00858         self.setParameter( 'numberOfFiles', numberOfFiles )
00859         self.setParameter( 'debug'        , debug )
00860         return self.apply()

def cmsswVersionTools::PickRelValInputFiles::apply ( self )

Definition at line 865 of file cmsswVersionTools.py.

00866                      :
00867         cmsswVersion  = self._parameters[ 'cmsswVersion'  ].value
00868         formerVersion = self._parameters[ 'formerVersion' ].value
00869         relVal        = self._parameters[ 'relVal'        ].value
00870         dataTier      = self._parameters[ 'dataTier'      ].value
00871         condition     = self._parameters[ 'condition'     ].value # only used for GT determination in initialization, if GT not explicitly given
00872         globalTag     = self._parameters[ 'globalTag'     ].value
00873         maxVersions   = self._parameters[ 'maxVersions'   ].value
00874         skipFiles     = self._parameters[ 'skipFiles'     ].value
00875         numberOfFiles = self._parameters[ 'numberOfFiles' ].value
00876         debug         = self._parameters[ 'debug'         ].value
00877 
00878         filePaths = []
00879 
00880         patchId = '_patch'
00881         ibId    = '_X_'
00882         if patchId in cmsswVersion:
00883             cmsswVersion = cmsswVersion.split( patchId )[ 0 ]
00884         elif ibId in cmsswVersion or formerVersion:
00885             outputTuple = Popen( [ 'scram', 'l -c CMSSW' ], stdout = PIPE, stderr = PIPE ).communicate()
00886             if len( outputTuple[ 1 ] ) != 0:
00887                 print '%s INFO : SCRAM error'%( self._label )
00888                 if debug:
00889                     print '    from trying to determine last valid releases before \'%s\''%( cmsswVersion )
00890                     print
00891                     print outputTuple[ 1 ]
00892                     print
00893                     self.messageEmptyList()
00894                 return filePaths
00895             versions = { 'last'      :''
00896                        , 'lastToLast':''
00897                        }
00898             for line in outputTuple[ 0 ].splitlines():
00899                 version = line.split()[ 1 ]
00900                 if cmsswVersion.split( ibId )[ 0 ] in version or cmsswVersion.rpartition( '_' )[ 0 ] in version:
00901                     if not ( patchId in version or ibId in version ):
00902                         versions[ 'lastToLast' ] = versions[ 'last' ]
00903                         versions[ 'last' ]       = version
00904                         if version == cmsswVersion:
00905                             break
00906             if formerVersion:
00907                 cmsswVersion = versions[ 'lastToLast' ]
00908             else:
00909                 cmsswVersion = versions[ 'last' ]
00910 
00911         if debug:
00912             print '%s DEBUG: Called with...'%( self._label )
00913             for key in self._parameters.keys():
00914                print '    %s:\t'%( key ),
00915                print self._parameters[ key ].value,
00916                if self._parameters[ key ].value is self.getDefaultParameters()[ key ].value:
00917                    print ' (default)'
00918                else:
00919                    print
00920                if key == 'cmsswVersion' and cmsswVersion != self._parameters[ key ].value:
00921                    if formerVersion:
00922                        print '    ==> modified to last to last valid release %s (s. \'formerVersion\' parameter)'%( cmsswVersion )
00923                    else:
00924                        print '    ==> modified to last valid release %s'%( cmsswVersion )
00925 
00926         command   = ''
00927         storage   = ''
00928         domain    = socket.getfqdn().split( '.' )
00929         if len( domain ) == 0:
00930             print '%s INFO : Cannot determine domain of this computer'%( self._label )
00931             if debug:
00932                 self.messageEmptyList()
00933             return filePaths
00934         elif len( domain ) == 1:
00935             print '%s INFO : Running on local host \'%s\' without direct access to RelVal files'%( self._label, domain[ 0 ] )
00936             if debug:
00937                 self.messageEmptyList()
00938             return filePaths
00939         if domain[ -2 ] == 'cern' and domain[ -1 ] == 'ch':
00940             command = 'nsls'
00941             storage = '/castor/cern.ch/cms'
00942         elif domain[ -2 ] == 'fnal' and domain[ -1 ] == 'gov':
00943             command = 'ls'
00944             storage = '/pnfs/cms/WAX/11'
00945         else:
00946             print '%s INFO : Running on site \'%s.%s\' without direct access to RelVal files'%( self._label, domain[ -2 ], domain[ -1 ] )
00947             if debug:
00948                 self.messageEmptyList()
00949             return filePaths
00950         if debug:
00951             print '%s DEBUG: Running at site \'%s.%s\''%( self._label, domain[ -2 ], domain[ -1 ] )
00952             print '    using command   \'%s\''%( command )
00953             print '    on storage path %s'%( storage )
00954         rfdirPath    = '/store/relval/%s/%s/%s/%s-v'%( cmsswVersion, relVal, dataTier, globalTag )
00955         argument     = '%s%s'%( storage, rfdirPath )
00956         validVersion = 0
00957 
00958         for version in range( maxVersions, 0, -1 ):
00959             filePaths = []
00960             fileCount = 0
00961             if debug:
00962                 print '%s DEBUG: Checking directory \'%s%i\''%( self._label, argument, version )
00963             directories = Popen( [ command, '%s%i'%( argument, version ) ], stdout = PIPE, stderr = PIPE ).communicate()[0]
00964             for directory in directories.splitlines():
00965                 files = Popen( [ command, '%s%i/%s'%( argument, version, directory ) ], stdout = PIPE, stderr = PIPE ).communicate()[0]
00966                 for file in files.splitlines():
00967                     if len( file ) > 0:
00968                         if debug:
00969                             print '%s DEBUG: File \'%s\' found'%( self._label, file )
00970                         fileCount += 1
00971                         validVersion = version
00972                     if fileCount > skipFiles:
00973                         filePath = '%s%i/%s/%s'%( rfdirPath, version, directory, file )
00974                         filePaths.append( filePath )
00975                     if numberOfFiles != 0 and len( filePaths ) >= numberOfFiles:
00976                         break
00977                 if debug:
00978                     print '%s DEBUG: %i file(s) found'%( self._label, fileCount )
00979                 if numberOfFiles != 0 and len( filePaths ) >= numberOfFiles:
00980                     break
00981             if numberOfFiles != 0:
00982               if len( filePaths ) >= numberOfFiles:
00983                 break
00984             elif validVersion > 0:
00985               break
00986 
00987         if validVersion == 0:
00988             print '%s INFO : No RelVal file(s) found at all in \'%s*\''%( self._label, argument )
00989             if debug:
00990                 self.messageEmptyList()
00991         elif len( filePaths ) == 0:
00992             print '%s INFO : No RelVal file(s) picked up in \'%s%i\''%( self._label, argument, validVersion )
00993             if debug:
00994                 self.messageEmptyList()
00995         elif len( filePaths ) < numberOfFiles:
00996             print '%s INFO : Only %i RelVal files picked up in \'%s%i\''%( self._label, len( filePaths ), argument, validVersion )
00997 
00998         if debug:
00999             print '%s DEBUG: returning %i file(s)\n%s'%( self._label, len( filePaths ), filePaths )
01000         return filePaths

def cmsswVersionTools::PickRelValInputFiles::getDefaultParameters ( self )

Definition at line 799 of file cmsswVersionTools.py.

00800                                     :
00801         return self._defaultParameters

def cmsswVersionTools::PickRelValInputFiles::messageEmptyList ( self )

Definition at line 861 of file cmsswVersionTools.py.

00862                                 :
00863         print '%s DEBUG: Empty file list returned'%( self._label )
00864         print '    This might be overwritten by providing input files explicitly to the source module in the main configuration file.'

Member Data Documentation

cmsswVersionTools::PickRelValInputFiles::_comment [private]

Definition at line 802 of file cmsswVersionTools.py.

tuple cmsswVersionTools::PickRelValInputFiles::_defaultParameters = dicttypes.SortedKeysDict() [static, private]

Definition at line 797 of file cmsswVersionTools.py.

string cmsswVersionTools::PickRelValInputFiles::_label = 'pickRelValInputFiles' [static, private]

Definition at line 796 of file cmsswVersionTools.py.

cmsswVersionTools::PickRelValInputFiles::_parameters [private]

Definition at line 802 of file cmsswVersionTools.py.

cmsswVersionTools::PickRelValInputFiles Class Reference

Public Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation