genericValidation.GenericValidationData Class Reference

Inheritance diagram for genericValidation.GenericValidationData:

Public Member Functions
def	__init__ (self, valName, alignment, config)
def	Bookkeeping (self)
def	cfgName (self)
def	cfgTemplate (self)
def	createConfiguration (self, path)
def	createCrabCfg (self, path, crabCfgBaseName)
def	createScript (self, path, template=configTemplates.scriptTemplate, downloadFiles=[], repMap=None, repMaps=None)
def	DefinePath (self)
def	FileOutputTemplate (self)
def	filesToCompare (self)
def	getRepMap (self, alignment=None)
def	LoadBasicModules (self)
def	ProcessName (self)
def	TrackSelectionRefitting (self)
def	ValidationTemplate (self)
Public Member Functions inherited from genericValidation.GenericValidation
def	__init__ (self, valName, alignment, config)
def	createConfiguration (self, fileContents, path, schedule=None, repMap=None, repMaps=None)
def	createCrabCfg (self, fileContents, path)
def	createFiles (self, fileContents, path, repMap=None, repMaps=None)
def	createScript (self, fileContents, path, downloadFiles=[], repMap=None, repMaps=None)
def	filesToCompare (self)
def	getCompareStrings (self, requestId=None, plain=False)
def	getRepMap (self, alignment=None)
Public Member Functions inherited from genericValidation.ValidationMetaClass
def	__new__ (cls, clsname, bases, dct)

Public Attributes
	crabWorkingDir
	dataset
Public Attributes inherited from genericValidation.GenericValidation
	alignmentToValidate
	AutoAlternates
	cmssw
	cmsswreleasebase
	config
	configFiles
	crabConfigFiles
	general
	jobid
	jobmode
	name
	needsproxy
	NJobs
	packages
	randomWorkdirPart
	scramarch
	scriptFiles

Static Public Attributes
dictionary	defaults
dictionary	mandatories = {"dataset", "maxevents"}
bool	needParentFiles = False
dictionary	optionals = {"magneticfield"}
Static Public Attributes inherited from genericValidation.GenericValidation
string	defaultReferenceName = "DEFAULT"
dictionary	defaults
	mandatories = set()
dictionary	needpackages = {"Alignment/OfflineValidation"}
dictionary	optionals = {"jobmode"}
Static Public Attributes inherited from genericValidation.ValidationMetaClass
list	dicts = ["defaults"]
list	sets = ["mandatories", "optionals", "needpackages"]

Detailed Description

Subclass of `GenericValidation` which is the base for validations using
datasets.

Definition at line 250 of file genericValidation.py.

Constructor & Destructor Documentation

__init__()

def genericValidation.GenericValidationData.__init__	(		self,
			valName,
			alignment,
			config
	)

This method adds additional items to the `self.general` dictionary
which are only needed for validations using datasets.

Arguments:
- `valName`: String which identifies individual validation instances
- `alignment`: `Alignment` instance to validate
- `config`: `BetterConfigParser` instance which includes the
    configuration of the validations

Definition at line 270 of file genericValidation.py.

References genericValidation.GenericValidation.cmssw, genericValidation.GenericValidation.cmsswreleasebase, nano_mu_digi_cff.float, genericValidation.GenericValidation.general, createfilelist.int, genericValidation.GenericValidation.jobmode, genericValidation.GenericValidation.NJobs, print(), and submitPVValidationJobs.split().

    def __init__(self, valName, alignment, config):
        """
        This method adds additional items to the `self.general` dictionary
        which are only needed for validations using datasets.
        
        Arguments:
        - `valName`: String which identifies individual validation instances
        - `alignment`: `Alignment` instance to validate
        - `config`: `BetterConfigParser` instance which includes the
                    configuration of the validations
        """

        super(GenericValidationData, self).__init__(valName, alignment, config)

        # if maxevents is not specified, cannot calculate number of events for
        # each parallel job, and therefore running only a single job
        if int( self.general["maxevents"] ) < 0 and self.NJobs > 1:
            msg = ("Maximum number of events (maxevents) not specified: "
                   "cannot use parallel jobs.")
            raise AllInOneError(msg)
        if int( self.general["maxevents"] ) / self.NJobs != float( self.general["maxevents"] ) / self.NJobs:
            msg = ("maxevents has to be divisible by parallelJobs")
            raise AllInOneError(msg)

        tryPredefinedFirst = (not self.jobmode.split( ',' )[0] == "crab" and self.general["JSON"]    == ""
                              and self.general["firstRun"] == ""         and self.general["lastRun"] == ""
                              and self.general["begin"]    == ""         and self.general["end"]     == "")

        if self.general["dataset"] not in globalDictionaries.usedDatasets:
            globalDictionaries.usedDatasets[self.general["dataset"]] = {}

        if self.cmssw not in globalDictionaries.usedDatasets[self.general["dataset"]]:
            if globalDictionaries.usedDatasets[self.general["dataset"]] != {}:
                print(("Warning: you use the same dataset '%s' in multiple cmssw releases.\n"
                       "This is allowed, but make sure it's not a mistake") % self.general["dataset"])
            globalDictionaries.usedDatasets[self.general["dataset"]][self.cmssw] = {False: None, True: None}

        Bfield = self.general.get("magneticfield", None)
        if globalDictionaries.usedDatasets[self.general["dataset"]][self.cmssw][tryPredefinedFirst] is None:
            dataset = Dataset(
                self.general["dataset"], tryPredefinedFirst = tryPredefinedFirst,
                cmssw = self.cmssw, cmsswrelease = self.cmsswreleasebase, magneticfield = Bfield,
                dasinstance = self.general["dasinstance"])
            globalDictionaries.usedDatasets[self.general["dataset"]][self.cmssw][tryPredefinedFirst] = dataset
            if tryPredefinedFirst and not dataset.predefined():                              #No point finding the data twice in that case
                globalDictionaries.usedDatasets[self.general["dataset"]][self.cmssw][False] = dataset

        self.dataset = globalDictionaries.usedDatasets[self.general["dataset"]][self.cmssw][tryPredefinedFirst]
        self.general["magneticField"] = self.dataset.magneticField()
        self.general["defaultMagneticField"] = "MagneticField"
        if self.general["magneticField"] == "unknown":
            print("Could not get the magnetic field for this dataset.")
            print("Using the default: ", self.general["defaultMagneticField"])
            self.general["magneticField"] = '.oO[defaultMagneticField]Oo.'
        
        if not self.jobmode.split( ',' )[0] == "crab":
            try:
                self.general["datasetDefinition"] = self.dataset.datasetSnippet(
                    jsonPath = self.general["JSON"],
                    firstRun = self.general["firstRun"],
                    lastRun = self.general["lastRun"],
                    begin = self.general["begin"],
                    end = self.general["end"],
                    parent = self.needParentFiles )
            except AllInOneError as e:
                msg = "In section [%s:%s]: "%(self.valType, self.name)
                msg += str(e)
                raise AllInOneError(msg)
        else:
            if self.dataset.predefined():
                msg = ("For jobmode 'crab' you cannot use predefined datasets "
                       "(in your case: '%s')."%( self.dataset.name() ))
                raise AllInOneError( msg )
            try:
                theUpdate = config.getResultingSection(self.valType+":"+self.name,
                                                       demandPars = ["parallelJobs"])
            except AllInOneError as e:
                msg = str(e)[:-1]+" when using 'jobmode: crab'."
                raise AllInOneError(msg)
            self.general.update(theUpdate)
            if self.general["begin"] or self.general["end"]:
                ( self.general["begin"],
                  self.general["end"],
                  self.general["firstRun"],
                  self.general["lastRun"] ) = self.dataset.convertTimeToRun(
                    firstRun = self.general["firstRun"],
                    lastRun = self.general["lastRun"],
                    begin = self.general["begin"],
                    end = self.general["end"],
                    shortTuple = False)
                if self.general["begin"] == None:
                    self.general["begin"] = ""
                if self.general["end"] == None:
                    self.general["end"] = ""
                self.general["firstRun"] = str( self.general["firstRun"] )
                self.general["lastRun"] = str( self.general["lastRun"] )
            if ( not self.general["firstRun"] ) and \
                   ( self.general["end"] or self.general["lastRun"] ):
                self.general["firstRun"] = str(
                    self.dataset.runList()[0]["run_number"])
            if ( not self.general["lastRun"] ) and \
                   ( self.general["begin"] or self.general["firstRun"] ):
                self.general["lastRun"] = str(
                    self.dataset.runList()[-1]["run_number"])
            if self.general["firstRun"] and self.general["lastRun"]:
                if int(self.general["firstRun"]) > int(self.general["lastRun"]):
                    msg = ( "The lower time/runrange limit ('begin'/'firstRun') "
                            "chosen is greater than the upper time/runrange limit "
                            "('end'/'lastRun').")
                    raise AllInOneError( msg )
                self.general["runRange"] = (self.general["firstRun"]
                                            + '-' + self.general["lastRun"])
            try:
                self.general["datasetDefinition"] = self.dataset.datasetSnippet(
                    jsonPath = self.general["JSON"],
                    firstRun = self.general["firstRun"],
                    lastRun = self.general["lastRun"],
                    begin = self.general["begin"],
                    end = self.general["end"],
                    crab = True )
            except AllInOneError as e:
                msg = "In section [%s:%s]: "%(self.valType, self.name)
                msg += str( e )
                raise AllInOneError( msg )

        self.general["usepixelqualityflag"] = pythonboolstring(self.general["usepixelqualityflag"], "usepixelqualityflag")

Member Function Documentation

Bookkeeping()

def genericValidation.GenericValidationData.Bookkeeping

(

self

)

Definition at line 494 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def Bookkeeping(self):
        return configTemplates.Bookkeeping

cfgName()

def genericValidation.GenericValidationData.cfgName

(

self

)

Definition at line 423 of file genericValidation.py.

References genericValidation.GenericValidation.alignmentToValidate, primaryVertexResolution.PrimaryVertexResolution.configBaseName, primaryVertexValidation.PrimaryVertexValidation.configBaseName, offlineValidation.OfflineValidation.configBaseName, zMuMuValidation.ZMuMuValidation.configBaseName, trackSplittingValidation.TrackSplittingValidation.configBaseName, monteCarloValidation.MonteCarloValidation.configBaseName, overlapValidation.OverlapValidation.configBaseName, offlineValidation.OfflineValidationDQM.configBaseName, AlignableObjectId::entry.name, alignment.Alignment.name, genericValidation.GenericValidation.name, TrackerSectorStruct.name, MuonGeometrySanityCheckPoint.name, classes.MonitorData.name, classes.OutputData.name, geometry.Structure.name, and plotscripts.SawTeethFunction.name.

Referenced by genericValidation.GenericValidationData.createConfiguration().

    def cfgName(self):
        return "%s.%s.%s_cfg.py"%( self.configBaseName, self.name,
                                   self.alignmentToValidate.name )

cfgTemplate()

def genericValidation.GenericValidationData.cfgTemplate

(

self

)

Definition at line 431 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.createConfiguration().

    def cfgTemplate(self):
        return configTemplates.cfgTemplate

createConfiguration()

def genericValidation.GenericValidationData.createConfiguration	(		self,
			path
	)

Definition at line 442 of file genericValidation.py.

References genericValidation.GenericValidationData.cfgName(), Mpslibclass.jobdatabase.cfgTemplate, genericValidation.GenericValidationData.cfgTemplate(), genericValidation.GenericValidation.getRepMap(), and alignment.Alignment.getRepMap().

    def createConfiguration(self, path ):
        repMap = self.getRepMap()
        cfgs = {self.cfgName: self.cfgTemplate}
        super(GenericValidationData, self).createConfiguration(cfgs, path, repMap=repMap)

createCrabCfg()

def genericValidation.GenericValidationData.createCrabCfg	(		self,
			path,
			crabCfgBaseName
	)

Method which creates a `crab.cfg` for a validation on datasets.

Arguments:
- `path`: Path at which the file will be stored.
- `crabCfgBaseName`: String which depends on the actual type of
             validation calling this method.

Definition at line 461 of file genericValidation.py.

References genericValidation.GenericValidation.alignmentToValidate, genericValidation.GenericValidation.getRepMap(), alignment.Alignment.getRepMap(), AlignableObjectId::entry.name, alignment.Alignment.name, genericValidation.GenericValidation.name, TrackerSectorStruct.name, MuonGeometrySanityCheckPoint.name, classes.MonitorData.name, classes.OutputData.name, geometry.Structure.name, and plotscripts.SawTeethFunction.name.

    def createCrabCfg(self, path, crabCfgBaseName):
        """
        Method which creates a `crab.cfg` for a validation on datasets.
        
        Arguments:
        - `path`: Path at which the file will be stored.
        - `crabCfgBaseName`: String which depends on the actual type of
                             validation calling this method.
        """
        crabCfgName = "crab.%s.%s.%s.cfg"%( crabCfgBaseName, self.name,
                                            self.alignmentToValidate.name )
        repMap = self.getRepMap()
        repMap["script"] = "dummy_script.sh"
        # repMap["crabOutputDir"] = os.path.basename( path )
        repMap["crabWorkingDir"] = crabCfgName.split( '.cfg' )[0]
        self.crabWorkingDir = repMap["crabWorkingDir"]
        repMap["numberOfJobs"] = self.general["parallelJobs"]
        repMap["cfgFile"] = self.configFiles[0]
        repMap["queue"] = self.jobmode.split( ',' )[1].split( '-q' )[1]
        if self.dataset.dataType() == "mc":
            repMap["McOrData"] = "events = .oO[nEvents]Oo."
        elif self.dataset.dataType() == "data":
            repMap["McOrData"] = "lumis = -1"
            if self.jobmode.split( ',' )[0] == "crab":
                print ("For jobmode 'crab' the parameter 'maxevents' will be "
                       "ignored and all events will be processed.")
        else:
            raise AllInOneError("Unknown data type!  Can't run in crab mode")
        crabCfg = {crabCfgName: replaceByMap( configTemplates.crabCfgTemplate,
                                              repMap ) }
        return super(GenericValidationData, self).createCrabCfg( crabCfg, path )

createScript()

def genericValidation.GenericValidationData.createScript	(		self,
			path,
			template = configTemplates.scriptTemplate,
			downloadFiles = [],
			repMap = None,
			repMaps = None
	)

Definition at line 447 of file genericValidation.py.

References helperFunctions.addIndex(), genericValidation.GenericValidation.alignmentToValidate, genericValidation.GenericValidation.configFiles, genericValidation.GenericValidation.getRepMap(), alignment.Alignment.getRepMap(), AlignableObjectId::entry.name, alignment.Alignment.name, genericValidation.GenericValidation.name, TrackerSectorStruct.name, MuonGeometrySanityCheckPoint.name, classes.MonitorData.name, classes.OutputData.name, geometry.Structure.name, plotscripts.SawTeethFunction.name, genericValidation.GenericValidation.NJobs, primaryVertexResolution.PrimaryVertexResolution.scriptBaseName, primaryVertexValidation.PrimaryVertexValidation.scriptBaseName, offlineValidation.OfflineValidation.scriptBaseName, trackSplittingValidation.TrackSplittingValidation.scriptBaseName, zMuMuValidation.ZMuMuValidation.scriptBaseName, monteCarloValidation.MonteCarloValidation.scriptBaseName, and overlapValidation.OverlapValidation.scriptBaseName.

    def createScript(self, path, template = configTemplates.scriptTemplate, downloadFiles=[], repMap = None, repMaps = None):
        scriptName = "%s.%s.%s.sh"%(self.scriptBaseName, self.name,
                                    self.alignmentToValidate.name )
        if repMap is None and repMaps is None:
            repMap = self.getRepMap()
            repMap["CommandLine"]=""
            for cfg in self.configFiles:
                repMap["CommandLine"]+= repMap["CommandLineTemplate"]%{"cfgFile":addIndex(cfg, self.NJobs, ".oO[nIndex]Oo."),
                                                      "postProcess":""
                                                     }
        scripts = {scriptName: template}
        return super(GenericValidationData, self).createScript(scripts, path, downloadFiles = downloadFiles,
                                                               repMap = repMap, repMaps = repMaps)

DefinePath()

def genericValidation.GenericValidationData.DefinePath

(

self

)

Definition at line 506 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def DefinePath(self):
        pass

FileOutputTemplate()

def genericValidation.GenericValidationData.FileOutputTemplate

(

self

)

Definition at line 503 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def FileOutputTemplate(self):
        return configTemplates.FileOutputTemplate

filesToCompare()

def genericValidation.GenericValidationData.filesToCompare

(

self

)

Definition at line 439 of file genericValidation.py.

References genericValidation.GenericValidation.defaultReferenceName, genericValidation.GenericValidation.getRepMap(), and alignment.Alignment.getRepMap().

    def filesToCompare(self):
        return {self.defaultReferenceName: self.getRepMap()["finalResultFile"]}

getRepMap()

def genericValidation.GenericValidationData.getRepMap	(		self,
			alignment = None
	)

Definition at line 397 of file genericValidation.py.

References helperFunctions.addIndex(), genericValidation.GenericValidationData.Bookkeeping(), genericValidation.GenericValidationData.DefinePath(), genericValidation.GenericValidationData.FileOutputTemplate(), genericValidation.GenericValidationData.LoadBasicModules(), AlignableObjectId::entry.name, alignment.Alignment.name, genericValidation.GenericValidation.name, TrackerSectorStruct.name, MuonGeometrySanityCheckPoint.name, classes.MonitorData.name, classes.OutputData.name, geometry.Structure.name, plotscripts.SawTeethFunction.name, genericValidation.GenericValidation.NJobs, primaryVertexResolution.PrimaryVertexResolution.outputBaseName, primaryVertexValidation.PrimaryVertexValidation.outputBaseName, offlineValidation.OfflineValidation.outputBaseName, zMuMuValidation.ZMuMuValidation.outputBaseName, trackSplittingValidation.TrackSplittingValidation.outputBaseName, monteCarloValidation.MonteCarloValidation.outputBaseName, overlapValidation.OverlapValidation.outputBaseName, genericValidation.GenericValidationData.ProcessName(), helperFunctions.replaceByMap(), primaryVertexResolution.PrimaryVertexResolution.resultBaseName, primaryVertexValidation.PrimaryVertexValidation.resultBaseName, offlineValidation.OfflineValidation.resultBaseName, trackSplittingValidation.TrackSplittingValidation.resultBaseName, zMuMuValidation.ZMuMuValidation.resultBaseName, monteCarloValidation.MonteCarloValidation.resultBaseName, overlapValidation.OverlapValidation.resultBaseName, genericValidation.GenericValidationData.TrackSelectionRefitting(), and genericValidation.GenericValidationData.ValidationTemplate().

Referenced by plottingOptions.PlottingOptionsTrackSplitting.validsubdets().

    def getRepMap(self, alignment = None):
        result = super(GenericValidationData, self).getRepMap(alignment)
        outputfile = os.path.expandvars(replaceByMap(
                           "%s_%s_.oO[name]Oo..root" % (self.outputBaseName, self.name)
                                 , result))
        resultfile = os.path.expandvars(replaceByMap(("/store/group/alca_trackeralign/AlignmentValidation/.oO[eosdir]Oo./" +
                           "%s_%s_.oO[name]Oo..root" % (self.resultBaseName, self.name))
                                 , result))
        result.update({
                "resultFile": ".oO[resultFiles[.oO[nIndex]Oo.]]Oo.",
                "resultFiles": addIndex(resultfile, self.NJobs),
                "finalResultFile": resultfile,
                "outputFile": ".oO[outputFiles[.oO[nIndex]Oo.]]Oo.",
                "outputFiles": addIndex(outputfile, self.NJobs),
                "finalOutputFile": outputfile,
                "ProcessName": self.ProcessName,
                "Bookkeeping": self.Bookkeeping,
                "LoadBasicModules": self.LoadBasicModules,
                "TrackSelectionRefitting": self.TrackSelectionRefitting,
                "ValidationConfig": self.ValidationTemplate,
                "FileOutputTemplate": self.FileOutputTemplate,
                "DefinePath": self.DefinePath,
                })
        return result

LoadBasicModules()

def genericValidation.GenericValidationData.LoadBasicModules

(

self

)

Definition at line 497 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def LoadBasicModules(self):
        return configTemplates.LoadBasicModules

ProcessName()

def genericValidation.GenericValidationData.ProcessName

(

self

)

Definition at line 427 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def ProcessName(self):
        pass

TrackSelectionRefitting()

def genericValidation.GenericValidationData.TrackSelectionRefitting

(

self

)

Definition at line 500 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def TrackSelectionRefitting(self):
        pass

ValidationTemplate()

def genericValidation.GenericValidationData.ValidationTemplate

(

self

)

Definition at line 435 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData.getRepMap().

    def ValidationTemplate(self):
        pass

Member Data Documentation

crabWorkingDir

genericValidation.GenericValidationData.crabWorkingDir

Definition at line 476 of file genericValidation.py.

dataset

genericValidation.GenericValidationData.dataset

Definition at line 317 of file genericValidation.py.

Referenced by genericValidation.GenericValidationData_CTSR.cosmics0T(), edmIntegrityCheck.IntegrityCheck.query(), production_tasks.BaseDataset.query(), production_tasks.CheckDatasetExists.run(), production_tasks.BaseDataset.run(), production_tasks.GenerateMask.run(), production_tasks.CreateJobDirectory.run(), production_tasks.SourceCFG.run(), production_tasks.FullCFG.run(), production_tasks.WriteToDatasets.run(), production_tasks.RunCMSBatch.run(), production_tasks.MonitorJobs.run(), production_tasks.CleanJobFiles.run(), and edmIntegrityCheck.IntegrityCheck.structured().

defaults

dictionary genericValidation.GenericValidationData.defaults

static

Initial value:

=  {
                "runRange": "",
                "firstRun": "",
                "lastRun": "",
                "begin": "",
                "end": "",
                "JSON": "",
                "dasinstance": "prod/global",
                "ttrhbuilder":"WithAngleAndTemplate",
                "usepixelqualityflag": "True",
               }

Definition at line 257 of file genericValidation.py.

Referenced by tree.Tree.reset(), and tree.Tree.var().

mandatories

dictionary genericValidation.GenericValidationData.mandatories = {"dataset", "maxevents"}

static

Definition at line 256 of file genericValidation.py.

needParentFiles

bool genericValidation.GenericValidationData.needParentFiles = False

static

Definition at line 255 of file genericValidation.py.

optionals

dictionary genericValidation.GenericValidationData.optionals = {"magneticfield"}

static

Definition at line 268 of file genericValidation.py.