dataset.BaseDataset Class Reference

Inheritance diagram for dataset.BaseDataset:

Public Member Functions
def	__init__ (self, name, user, pattern='.*root', run_range=None, dbsInstance=None)
	def init(self, name, user, pattern='. More...
def	buildListOfBadFiles (self)
def	buildListOfFiles (self, pattern)
def	extractFileSizes (self)
def	getPrimaryDatasetEntries (self)
def	listOfFiles (self)
def	listOfGoodFiles (self)
def	listOfGoodFilesWithPrescale (self, prescale)
def	printFiles (self, abspath=True, info=True)
def	printInfo (self)

Public Attributes
	bad_files
	dbsInstance
	MM. More...
	files
	filesAndSizes
	good_files
	name
	pattern
	primaryDatasetEntries
	MM. More...
	report
	run_range
	user

Detailed Description

Definition at line 21 of file dataset.py.

Constructor & Destructor Documentation

def dataset.BaseDataset.__init__	(		self,
			name,
			user,
			pattern = '.*root',
			run_range = None,
			dbsInstance = None
	)

def init(self, name, user, pattern='.

*root', run_range=None):

Definition at line 24 of file dataset.py.

    def __init__(self, name, user, pattern='.*root', run_range=None, dbsInstance=None):
        self.name = name
        self.user = user
        self.pattern = pattern
        self.run_range = run_range
        ### MM
        self.dbsInstance = dbsInstance
        ### MM
        self.primaryDatasetEntries = -1
        self.report = None
        self.buildListOfFiles( self.pattern )
        self.extractFileSizes()
        self.buildListOfBadFiles()
        self.primaryDatasetEntries = self.getPrimaryDatasetEntries()
     

Member Function Documentation

def dataset.BaseDataset.buildListOfBadFiles

(

self

)

Definition at line 47 of file dataset.py.

    def buildListOfBadFiles(self):
        self.good_files = []
        self.bad_files = {}

def dataset.BaseDataset.buildListOfFiles	(		self,
			pattern
	)

Definition at line 39 of file dataset.py.

Referenced by dataset.BaseDataset.printFiles().

    def buildListOfFiles( self, pattern ):
        self.files = []

def dataset.BaseDataset.extractFileSizes

(

self

)

Get the file size for each file, 
from the eos ls -l command.

Definition at line 42 of file dataset.py.

    def extractFileSizes(self):
        '''Get the file size for each file, 
        from the eos ls -l command.'''
        self.filesAndSizes = {}

def dataset.BaseDataset.getPrimaryDatasetEntries

(

self

)

Definition at line 55 of file dataset.py.

References dataset.BaseDataset.primaryDatasetEntries.

    def getPrimaryDatasetEntries(self):
        return self.primaryDatasetEntries

def dataset.BaseDataset.listOfFiles

(

self

)

Returns all files, even the bad ones.

Definition at line 81 of file dataset.py.

References readConfig.fileINI.files, dataset.BaseDataset.files, chain.Chain.files, MatrixReader.MatrixReader.files, MatrixUtil.InputInfo.files, and JsonOutputProducer::JsonConfigurationBlock.files.

    def listOfFiles(self):
        '''Returns all files, even the bad ones.'''
        return self.files

def dataset.BaseDataset.listOfGoodFiles

(

self

)

Returns all files flagged as good in the integrity 
check text output, or not present in this file, are 
considered as good.

Definition at line 85 of file dataset.py.

References dataset.BaseDataset.bad_files, readConfig.fileINI.files, dataset.BaseDataset.files, chain.Chain.files, MatrixReader.MatrixReader.files, MatrixUtil.InputInfo.files, JsonOutputProducer::JsonConfigurationBlock.files, and dataset.BaseDataset.good_files.

Referenced by dataset.BaseDataset.listOfGoodFilesWithPrescale().

    def listOfGoodFiles(self):
        '''Returns all files flagged as good in the integrity 
        check text output, or not present in this file, are 
        considered as good.'''
        self.good_files = []
        for file in self.files:            
            if file not in self.bad_files:
                self.good_files.append( file )
        return self.good_files

def dataset.BaseDataset.listOfGoodFilesWithPrescale	(		self,
			prescale
	)

Takes the list of good files and selects a random sample 
from them according to the prescale factor. 
E.g. a prescale of 10 will select 1 in 10 files.

Definition at line 95 of file dataset.py.

References dataset.BaseDataset.good_files, dataset.int, and dataset.BaseDataset.listOfGoodFiles().

    def listOfGoodFilesWithPrescale(self, prescale):
        """Takes the list of good files and selects a random sample 
        from them according to the prescale factor. 
        E.g. a prescale of 10 will select 1 in 10 files."""

        good_files = self.listOfGoodFiles()
        if prescale < 2:
            return self.good_files
        
        #the number of files to select from the dataset
        num_files = int( (len(good_files)/(1.0*prescale)) + 0.5)
        if num_files < 1:
            num_files = 1
        if num_files > len(good_files):
            num_files = len(good_files)
        
        #pick unique good files randomly
        import random
        subset = set()
        while len(subset) < num_files:
            #pick a random file from the list
            choice = random.choice(good_files)
            slen = len(subset)
            #add to the set
            subset.add(choice)
            #if this was a unique file remove so we don't get 
            #very slow corner cases where prescale is small
            if len(subset) > slen:
                good_files.remove(choice)
        assert len(subset)==num_files,'The number of files does not match'

        return [f for f in subset]

def dataset.BaseDataset.printFiles	(		self,
			abspath = True,
			info = True
	)

Definition at line 58 of file dataset.py.

References dataset.BaseDataset.bad_files, dataset.BaseDataset.buildListOfFiles(), readConfig.fileINI.files, dataset.BaseDataset.files, chain.Chain.files, MatrixReader.MatrixReader.files, MatrixUtil.InputInfo.files, JsonOutputProducer::JsonConfigurationBlock.files, dataset.BaseDataset.good_files, dataset.BaseDataset.pattern, CSCALCT.pattern, presentation.PageLayout.pattern, l1t::EMTFRoad.pattern, L1TMuon::TriggerPrimitive::CSCData.pattern, CSCCorrelatedLCTDigi.pattern, cscdqm::MOFilterItem.pattern, l1t::EMTFHit.pattern, dataset.BaseDataset.primaryDatasetEntries, and edm.print().

    def printFiles(self, abspath=True, info=True):
        # import pdb; pdb.set_trace()
        if self.files == None:
            self.buildListOfFiles(self.pattern)
        for file in self.files:
            status = 'OK'
            if file in self.bad_files:
                status = self.bad_files[file]
            elif file not in self.good_files:
                status = 'UNKNOWN'
            fileNameToPrint = file
            if abspath == False:
                fileNameToPrint = os.path.basename(file)
            if info:
                size=self.filesAndSizes.get(file,'UNKNOWN').rjust(10)
                # if size is not None:
                #     size = size.rjust(10)
                print(status.ljust(10), size, \
                      '\t', fileNameToPrint)
            else:
                print(fileNameToPrint)
        print('PrimaryDatasetEntries: %d' % self.primaryDatasetEntries)
                

def dataset.BaseDataset.printInfo

(

self

)

Definition at line 51 of file dataset.py.

References ElectronMVAID.ElectronMVAID.name, counter.Counter.name, average.Average.name, histograms.Histograms.name, AlignableObjectId::entry.name, TmModule.name, cond::persistency::TAG::NAME.name, cond::persistency::GLOBAL_TAG::NAME.name, core.autovars.NTupleVariable.name, cond::persistency::RUN_INFO::RUN_NUMBER.name, cond::persistency::TAG::TIME_TYPE.name, cond::persistency::GLOBAL_TAG::VALIDITY.name, cond::persistency::RUN_INFO::START_TIME.name, cond::persistency::TAG::OBJECT_TYPE.name, cond::persistency::GLOBAL_TAG::DESCRIPTION.name, cond::persistency::RUN_INFO::END_TIME.name, cond::persistency::TAG::SYNCHRONIZATION.name, cond::persistency::GLOBAL_TAG::RELEASE.name, MEPSet.name, cond::persistency::TAG::END_OF_VALIDITY.name, cond::persistency::GLOBAL_TAG::SNAPSHOT_TIME.name, cond::persistency::GTEditorData.name, cond::persistency::TAG::DESCRIPTION.name, cond::persistency::GLOBAL_TAG::INSERTION_TIME.name, nanoaod::MergeableCounterTable::SingleColumn< T >.name, cond::persistency::TAG::LAST_VALIDATED_TIME.name, cond::persistency::TAG::INSERTION_TIME.name, cond::persistency::TAG::MODIFICATION_TIME.name, preexistingValidation.PreexistingValidation.name, FWTGeoRecoGeometry::Info.name, Types._Untracked.name, dataset.BaseDataset.name, OutputMEPSet.name, personalPlayback.Applet.name, ParameterSet.name, PixelDCSObject< T >::Item.name, DQMRivetClient::LumiOption.name, MagCylinder.name, analyzer.Analyzer.name, ParSet.name, DQMRivetClient::ScaleFactorOption.name, EgHLTOfflineSummaryClient::SumHistBinData.name, SingleObjectCondition.name, cond::persistency::GTProxyData.name, core.autovars.NTupleObjectType.name, MyWatcher.name, edm::PathTimingSummary.name, nanoaod::MergeableCounterTable::VectorColumn< T >.name, cond::TimeTypeSpecs.name, lumi::TriggerInfo.name, alignment.Alignment.name, edm::PathSummary.name, PixelEndcapLinkMaker::Item.name, perftools::EdmEventSize::BranchRecord.name, cond::persistency::GLOBAL_TAG_MAP::GLOBAL_TAG_NAME.name, DQMGenericClient::EfficOption.name, FWTableViewManager::TableEntry.name, cond::persistency::GLOBAL_TAG_MAP::RECORD.name, PixelBarrelLinkMaker::Item.name, cms::DDAlgoArguments.name, EcalLogicID.name, cond::persistency::GLOBAL_TAG_MAP::LABEL.name, validateAlignments.ParallelMergeJob.name, MEtoEDM< T >::MEtoEDMObject.name, cond::persistency::GLOBAL_TAG_MAP::TAG_NAME.name, ExpressionHisto< T >.name, XMLProcessor::_loaderBaseConfig.name, cond::persistency::PAYLOAD::HASH.name, cond::persistency::PAYLOAD::OBJECT_TYPE.name, cond::persistency::PAYLOAD::DATA.name, genericValidation.GenericValidation.name, TreeCrawler.Package.name, cond::persistency::PAYLOAD::STREAMER_INFO.name, cond::persistency::PAYLOAD::VERSION.name, options.ConnectionHLTMenu.name, MagGeoBuilderFromDDD::volumeHandle.name, cond::persistency::PAYLOAD::INSERTION_TIME.name, DQMGenericClient::ProfileOption.name, dqmoffline::l1t::HistDefinition.name, DQMGenericClient::NormOption.name, emtf::Node.name, h4DSegm.name, PhysicsTools::Calibration::Variable.name, FastHFShowerLibrary.name, core.TriggerMatchAnalyzer.TriggerMatchAnalyzer.name, DQMGenericClient::CDOption.name, CounterChecker.name, cond::TagInfo_t.name, looper.Looper.name, DQMGenericClient::NoFlowOption.name, cond::persistency::IOV::TAG_NAME.name, EDMtoMEConverter.name, cond::persistency::IOV::SINCE.name, TrackerSectorStruct.name, Mapper::definition< ScannerT >.name, cond::persistency::IOV::PAYLOAD_HASH.name, classes.MonitorData.name, cond::persistency::IOV::INSERTION_TIME.name, HistogramManager.name, MuonGeometrySanityCheckPoint.name, classes.OutputData.name, options.HLTProcessOptions.name, h2DSegm.name, core.TriggerBitAnalyzer.TriggerBitAnalyzer.name, nanoaod::FlatTable::Column.name, geometry.Structure.name, config.Analyzer.name, core.autovars.NTupleSubObject.name, DQMNet::WaitObject.name, AlpgenParameterName.name, SiStripMonitorDigi.name, core.autovars.NTupleObject.name, config.Service.name, cond::persistency::TAG_LOG::TAG_NAME.name, cond::persistency::TAG_LOG::EVENT_TIME.name, cond::persistency::TAG_LOG::USER_NAME.name, cond::persistency::TAG_LOG::HOST_NAME.name, cond::persistency::TAG_LOG::COMMAND.name, cond::persistency::TAG_LOG::ACTION.name, cond::persistency::TAG_LOG::USER_TEXT.name, core.autovars.NTupleCollection.name, BPHRecoBuilder::BPHRecoSource.name, BPHRecoBuilder::BPHCompSource.name, personalPlayback.FrameworkJob.name, plotscripts.SawTeethFunction.name, crabFunctions.CrabTask.name, hTMaxCell.name, cscdqm::ParHistoDef.name, BeautifulSoup.Tag.name, SummaryOutputProducer::GenericSummary.name, BeautifulSoup.SoupStrainer.name, edm.print(), dataset.BaseDataset.user, EcalTPGParamReaderFromDB.user, popcon::RpcDataV.user, popcon::RpcObGasData.user, popcon::RPCObPVSSmapData.user, popcon::RpcDataT.user, popcon::RpcDataS.user, popcon::RpcDataFebmap.user, popcon::RpcDataGasMix.user, popcon::RpcDataI.user, popcon::RpcDataUXC.user, MatrixInjector.MatrixInjector.user, and EcalDBConnection.user.

    def printInfo(self):
        print('sample      :  ' + self.name)
        print('user        :  ' + self.user)

Member Data Documentation

dataset.BaseDataset.bad_files

Definition at line 49 of file dataset.py.

Referenced by dataset.BaseDataset.listOfGoodFiles(), and dataset.BaseDataset.printFiles().

dataset.BaseDataset.dbsInstance

MM.

Definition at line 30 of file dataset.py.

Referenced by dataset.PrivateDataset.getPrimaryDatasetEntries().

dataset.BaseDataset.files

Definition at line 40 of file dataset.py.

Referenced by dataset.BaseDataset.listOfFiles(), dataset.BaseDataset.listOfGoodFiles(), and dataset.BaseDataset.printFiles().

dataset.BaseDataset.filesAndSizes

Definition at line 45 of file dataset.py.

dataset.BaseDataset.good_files

Definition at line 48 of file dataset.py.

Referenced by dataset.BaseDataset.listOfGoodFiles(), dataset.BaseDataset.listOfGoodFilesWithPrescale(), and dataset.BaseDataset.printFiles().

dataset.BaseDataset.name

Definition at line 25 of file dataset.py.

Referenced by FWLite.ElectronMVAID.__call__(), dirstructure.Directory.__create_pie_image(), DisplayManager.DisplayManager.__del__(), dqm_interfaces.DirID.__eq__(), dirstructure.Directory.__get_full_path(), dirstructure.Comparison.__get_img_name(), dataset.Dataset.__getDataType(), dataset.Dataset.__getFileInfoList(), dirstructure.Comparison.__make_image(), dirstructure.Directory.__repr__(), dqm_interfaces.DirID.__repr__(), dirstructure.Comparison.__repr__(), config.CFG.__str__(), FWLite.WorkingPoints._reformat_cut_definitions(), dirstructure.Directory.calcStats(), validation.Sample.digest(), python.rootplot.utilities.Hist.divide(), python.rootplot.utilities.Hist.divide_wilson(), DisplayManager.DisplayManager.Draw(), utils.StatisticalTest.get_status(), production_tasks.Task.getname(), dataset.CMSDataset.getPrimaryDatasetEntries(), dataset.PrivateDataset.getPrimaryDatasetEntries(), VIDSelectorBase.VIDSelectorBase.initialize(), dirstructure.Directory.print_report(), dataset.BaseDataset.printInfo(), dataset.Dataset.printInfo(), production_tasks.MonitorJobs.run(), python.rootplot.utilities.Hist.TGraph(), python.rootplot.utilities.Hist.TH1F(), and Vispa.Views.PropertyView.Property.valueChanged().

dataset.BaseDataset.pattern

Definition at line 27 of file dataset.py.

Referenced by dataset.BaseDataset.printFiles().

dataset.BaseDataset.primaryDatasetEntries

MM.

Definition at line 32 of file dataset.py.

Referenced by dataset.BaseDataset.getPrimaryDatasetEntries(), and dataset.BaseDataset.printFiles().

dataset.BaseDataset.report

Definition at line 33 of file dataset.py.

Referenced by dataset.Dataset.getPrimaryDatasetEntries(), and addOnTests.testit.run().

dataset.BaseDataset.run_range

Definition at line 28 of file dataset.py.

Referenced by dataset.CMSDataset.buildListOfFiles(), dataset.CMSDataset.buildListOfFilesDBS(), dataset.CMSDataset.getPrimaryDatasetEntries(), and dataset.PrivateDataset.getPrimaryDatasetEntries().

dataset.BaseDataset.user

Definition at line 26 of file dataset.py.

Referenced by cmsPerfSuite.PerfSuite.optionParse(), dataset.BaseDataset.printInfo(), DTWorkflow.DTWorkflow.remote_out_path(), production_tasks.CheckDatasetExists.run(), production_tasks.GenerateMask.run(), production_tasks.SourceCFG.run(), production_tasks.FullCFG.run(), production_tasks.MonitorJobs.run(), and production_tasks.CleanJobFiles.run().