looper.Looper Class Reference

Inheritance diagram for looper.Looper:

Public Member Functions
def	__init__ (self, name, config, nEvents=None, firstEvent=0, nPrint=0, timeReport=False, quiet=False, memCheckFromEvent=-1)
def	loop (self)
def	process (self, iEv)
def	write (self)

Public Attributes
	analyzers
	cfg_comp
	classes
	config
	event
	events
	firstEvent
	iEvent
	logger
	memLast
	memReportFirstEvent
	name
	nEvents
	nPrint
	outDir
	setup
	start_time
	start_time_event
	timeReport

Private Member Functions
def	_build (self, cfg)
def	_prepareOutput (self, name)

Detailed Description

Creates a set of analyzers, and schedules the event processing.

Definition at line 52 of file looper.py.

Constructor & Destructor Documentation

__init__()

def looper.Looper.__init__	(		self,
			name,
			config,
			nEvents = None,
			firstEvent = 0,
			nPrint = 0,
			timeReport = False,
			quiet = False,
			memCheckFromEvent = -1
	)

Handles the processing of an event sample.
An Analyzer is built for each Config.Analyzer present
in sequence. The Looper can then be used to process an event,
or a collection of events.

Parameters:
name    : name of the Looper, will be used as the output directory name
config  : process configuration information, see Config
nEvents : number of events to process. Defaults to all.
firstEvent : first event to process. Defaults to the first one.
nPrint  : number of events to print at the beginning

Definition at line 62 of file looper.py.

                  memCheckFromEvent=-1):
        """Handles the processing of an event sample.
        An Analyzer is built for each Config.Analyzer present
        in sequence. The Looper can then be used to process an event,
        or a collection of events.

        Parameters:
        name    : name of the Looper, will be used as the output directory name
        config  : process configuration information, see Config
        nEvents : number of events to process. Defaults to all.
        firstEvent : first event to process. Defaults to the first one.
        nPrint  : number of events to print at the beginning
        """

        self.config = config
        self.name = self._prepareOutput(name)
        self.outDir = self.name
        self.logger = logging.getLogger( self.name )
        self.logger.addHandler(logging.FileHandler('/'.join([self.name,
                                                             'log.txt'])))
        self.logger.propagate = False
        if not quiet: 
            self.logger.addHandler( logging.StreamHandler(sys.stdout) )

        self.cfg_comp = config.components[0]
        self.classes = {}
        self.analyzers = map( self._build, config.sequence )
        self.nEvents = nEvents
        self.firstEvent = firstEvent
        self.nPrint = int(nPrint)
        self.timeReport = [ {'time':0.0,'events':0} for a in self.analyzers ] if timeReport else False
        self.memReportFirstEvent = memCheckFromEvent
        self.memLast=0
        tree_name = None
        if( hasattr(self.cfg_comp, 'tree_name') ):
            tree_name = self.cfg_comp.tree_name
        if len(self.cfg_comp.files)==0:
            errmsg = 'please provide at least an input file in the files attribute of this component\n' + str(self.cfg_comp)
            raise ValueError( errmsg )
        if hasattr(config,"preprocessor") and config.preprocessor is not None :
              self.cfg_comp = config.preprocessor.run(self.cfg_comp,self.outDir,firstEvent,nEvents)
        if hasattr(self.cfg_comp,"options"):
              print(self.cfg_comp.files,self.cfg_comp.options)
              self.events = config.events_class(self.cfg_comp.files, tree_name,options=self.cfg_comp.options)
        else :
              self.events = config.events_class(self.cfg_comp.files, tree_name)
        if hasattr(self.cfg_comp, 'fineSplit'):
            fineSplitIndex, fineSplitFactor = self.cfg_comp.fineSplit
            if fineSplitFactor > 1:
                if len(self.cfg_comp.files) != 1:
                    raise RuntimeError("Any component with fineSplit > 1 is supposed to have just a single file, while %s has %s" % (self.cfg_comp.name, self.cfg_comp.files))
                totevents = min(len(self.events),int(nEvents)) if (nEvents and int(nEvents) not in [-1,0]) else len(self.events)
                self.nEvents = int(ceil(totevents/float(fineSplitFactor)))
                self.firstEvent = firstEvent + fineSplitIndex * self.nEvents
                if self.firstEvent + self.nEvents >= totevents:
                    self.nEvents = totevents - self.firstEvent 
                #print "For component %s will process %d events starting from the %d one, ending at %d excluded" % (self.cfg_comp.name, self.nEvents, self.firstEvent, self.nEvents + self.firstEvent)
        # self.event is set in self.process
        self.event = None
        services = dict()
        for cfg_serv in config.services:
            service = self._build(cfg_serv)
            services[cfg_serv.name] = service
        # would like to provide a copy of the config to the setup,
        # so that analyzers cannot modify the config of other analyzers. 
        # but cannot copy the autofill config.
        self.setup = Setup(config, services)

Member Function Documentation

_build()

def looper.Looper._build (   self,   cfg  )

private

Definition at line 130 of file looper.py.

References analyzer.Analyzer.cfg_comp, looper.Looper.cfg_comp, and looper.Looper.outDir.

    def _build(self, cfg):
        theClass = cfg.class_object
        obj = theClass( cfg, self.cfg_comp, self.outDir )
        return obj
        

_prepareOutput()

def looper.Looper._prepareOutput (   self,   name  )

private

Definition at line 135 of file looper.py.

    def _prepareOutput(self, name):
        index = 0
        tmpname = name
        while True and index < 2000:
            try:
                # print 'mkdir', self.name
                os.mkdir( tmpname )
                break
            except OSError:
                index += 1
                tmpname = '%s_%d' % (name, index)
        if index == 2000:
              raise ValueError( "More than 2000 output folder with same name or 2000 attempts failed, please clean-up, change name or check permissions")
        return tmpname

loop()

def looper.Looper.loop

(

self

)

Loop on a given number of events.

At the beginning of the loop, 
Analyzer.beginLoop is called for each Analyzer.
At each event, self.process is called.
At the end of the loop, Analyzer.endLoop is called.

Definition at line 151 of file looper.py.

References looper.Looper.analyzers, Config.Process.analyzers, analyzer.Analyzer.cfg_comp, looper.Looper.cfg_comp, CSCFileDumper.events, eventsfwlite.Events.events, generateEDF.LumiInfo.events, emtf::Forest.events, JanAlignmentAlgorithm.events, ME::Header.events, emtf::Node.events, HitEff.events, HitResol.events, looper.Looper.events, MatrixUtil.InputInfo.events, CosmicRateAnalyzer.events, BeamSpotDipServer.events, options.HLTProcessOptions.events, FastTimerService::ResourcesPerModule.events, FastTimerService::ResourcesPerJob.events, JsonOutputProducer::JsonConfiguration.events, Printer.Printer.firstEvent, CSCFileReader.firstEvent, lhef::LHEReader.firstEvent, lhef::LH5Reader.firstEvent, ApeEstimatorSummary.firstEvent, looper.Looper.firstEvent, LeptonRecoSkim.firstEvent, SiStripMonitorPedestals.firstEvent, SiPixelPhase1Analyzer.firstEvent, SiStripMonitorDigi.firstEvent, SiStripMonitorCluster.firstEvent, SiStripMonitorTrack.firstEvent, info(), createfilelist.int, crabFunctions.CrabController.logger, looper.Looper.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, cmsHarvester.CMSHarvester.logger, CSCFileReader.nEvents, HIPTrackMonitorVariables.nEvents, WorkFlowRunner.WorkFlowRunner.nEvents, CTPPSPixelDQMSource.nEvents, QualityTester.nEvents, DTScalerInfoTask.nEvents, AlpgenHeader.nEvents, AlCaHcalNoiseProducer.nEvents, DTLocalTriggerLutTask.nEvents, looper.Looper.nEvents, hcalCalib.nEvents, MuonIsolationDQM.nEvents, MatacqProducer::stats_t.nEvents, cscdqm::Configuration.nEvents, print(), FastTimerService_cff.range, event.Event.setup, looper.Looper.setup, options.HLTProcessOptions.setup, and str.

    def loop(self):
        """Loop on a given number of events.

        At the beginning of the loop, 
        Analyzer.beginLoop is called for each Analyzer.
        At each event, self.process is called.
        At the end of the loop, Analyzer.endLoop is called.
        """
        nEvents = self.nEvents
        firstEvent = self.firstEvent
        iEv = firstEvent
        if nEvents is None or int(nEvents) > len(self.events) :
            nEvents = len(self.events)
        else:
            nEvents = int(nEvents)
        eventSize = nEvents
        self.logger.info(
            'starting loop at event {firstEvent} '\
                'to process {eventSize} events.'.format(firstEvent=firstEvent,
                                                        eventSize=eventSize))
        self.logger.info( str( self.cfg_comp ) )
        for analyzer in self.analyzers:
            analyzer.beginLoop(self.setup)
        try:
            for iEv in range(firstEvent, firstEvent+eventSize):
                # if iEv == nEvents:
                #     break
                if iEv%100 ==0:
                    # print 'event', iEv
                    if not hasattr(self,'start_time'):
                        print('event', iEv)
                        self.start_time = timeit.default_timer()
                        self.start_time_event = iEv
                    else:
                        print('event %d (%.1f ev/s)' % (iEv, (iEv-self.start_time_event)/float(timeit.default_timer() - self.start_time)))

                self.process( iEv )
                if iEv<self.nPrint:
                    print(self.event)

        except UserWarning:
            print('Stopped loop following a UserWarning exception')

        info = self.logger.info
        warning = self.logger.warning
        warning('number of events processed: {nEv}'.format(nEv=iEv+1))
        warning('')
        info( self.cfg_comp )
        info('')        
        for analyzer in self.analyzers:
            analyzer.endLoop(self.setup)
        if self.timeReport:
            allev = max([x['events'] for x in self.timeReport])
            warning("\n      ---- TimeReport (all times in ms; first evt is skipped) ---- ")
            warning("%9s   %9s    %9s   %9s %6s   %s" % ("processed","all evts","time/proc", " time/all", "  [%] ", "analyer"))
            warning("%9s   %9s    %9s   %9s %6s   %s" % ("---------","--------","---------", "---------", " -----", "-------------"))
            sumtime = sum(rep['time'] for rep in self.timeReport)
            passev  = self.timeReport[-1]['events']
            for ana,rep in zip(self.analyzers,self.timeReport):
                timePerProcEv = rep['time']/(rep['events']-1) if rep['events'] > 1 else 0
                timePerAllEv  = rep['time']/(allev-1)         if allev > 1         else 0
                fracAllEv     = rep['time']/sumtime
                warning( "%9d   %9d   %10.2f  %10.2f %5.1f%%   %s" % ( rep['events'], allev, 1000*timePerProcEv, 1000*timePerAllEv, 100.0*fracAllEv, ana.name))
            totPerProcEv = sumtime/(passev-1) if passev > 1 else 0
            totPerAllEv  = sumtime/(allev-1)  if allev > 1  else 0
            warning("%9s   %9s    %9s   %9s   %s" % ("---------","--------","---------", "---------", "-------------"))
            warning("%9d   %9d   %10.2f  %10.2f %5.1f%%   %s" % ( passev, allev, 1000*totPerProcEv, 1000*totPerAllEv, 100.0, "TOTAL"))
            warning("")
        if hasattr(self.events, 'endLoop'): self.events.endLoop()
        if hasattr(self.config,"preprocessor") and self.config.preprocessor is not None:
              if hasattr(self.config.preprocessor,"endLoop"):
                  self.config.preprocessor.endLoop(self.cfg_comp)

process()

def looper.Looper.process	(		self,
			iEv
	)

Run event processing for all analyzers in the sequence.

This function is called by self.loop,
but can also be called directly from
the python interpreter, to jump to a given event.

Definition at line 224 of file looper.py.

References TB06Reco.event, TB06RecoH2.event, ApeOverview.event, hgcal::econd::HGCalModuleTreeReader::HGCModuleTreeEvent.event, core.AutoHandle.AutoHandle.event, L1Analysis::L1AnalysisEventDataFormat.event, MuonPair.event, EcalPerEvtMatacqAnalyzer.event, LHAupLesHouches.event, WZInterestingEventSelector::event.event, lhef::LHEProxy.event, EcalStatusAnalyzer.event, EcalMatacqAnalyzer.event, L1TCaloLayer1RawToDigi.event, looper.Looper.event, EcalTestPulseAnalyzer.event, EcalABAnalyzer.event, HitEff.event, EcalPerEvtLaserAnalyzer.event, edm::service::SimpleMemoryCheck::SignificantEvent.event, EcalLaserAnalyzer.event, EcalLaserAnalyzer2.event, FastTimerService::ResourcesPerJob.event, cms::alpakatools::CachingAllocator< TDev, TQueue >::BlockDescriptor.event, l1ct::Event.event, JsonOutputProducer::JsonEvent.event, SummaryOutputProducer::Event.event, CSCFileDumper.events, eventsfwlite.Events.events, generateEDF.LumiInfo.events, emtf::Forest.events, JanAlignmentAlgorithm.events, ME::Header.events, emtf::Node.events, HitEff.events, HitResol.events, looper.Looper.events, MatrixUtil.InputInfo.events, CosmicRateAnalyzer.events, BeamSpotDipServer.events, options.HLTProcessOptions.events, FastTimerService::ResourcesPerModule.events, FastTimerService::ResourcesPerJob.events, JsonOutputProducer::JsonConfiguration.events, event.Event.setup, looper.Looper.setup, and options.HLTProcessOptions.setup.

    def process(self, iEv ):
        """Run event processing for all analyzers in the sequence.

        This function is called by self.loop,
        but can also be called directly from
        the python interpreter, to jump to a given event.
        """
        self.event = Event(iEv, self.events[iEv], self.setup)
        self.iEvent = iEv
        for i,analyzer in enumerate(self.analyzers):
            if not analyzer.beginLoopCalled:
                analyzer.beginLoop(self.setup)
            start = timeit.default_timer()
            if self.memReportFirstEvent >=0 and iEv >= self.memReportFirstEvent:           
                memNow=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
                if memNow > self.memLast :
                   print("Mem Jump detected before analyzer %s at event %s. RSS(before,after,difference) %s %s %s "%( analyzer.name, iEv, self.memLast, memNow, memNow-self.memLast))
                self.memLast=memNow
            ret = analyzer.process( self.event )
            if self.memReportFirstEvent >=0 and iEv >= self.memReportFirstEvent:           
                memNow=resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
                if memNow > self.memLast :
                   print("Mem Jump detected in analyzer %s at event %s. RSS(before,after,difference) %s %s %s "%( analyzer.name, iEv, self.memLast, memNow, memNow-self.memLast))
                self.memLast=memNow
            if self.timeReport:
                self.timeReport[i]['events'] += 1
                if self.timeReport[i]['events'] > 0:
                    self.timeReport[i]['time'] += timeit.default_timer() - start
            if ret == False:
                return (False, analyzer.name)
        if iEv<self.nPrint:
            self.logger.info( self.event.__str__() )
        return (True, analyzer.name)

write()

def looper.Looper.write

(

self

)

Writes all analyzers.

See Analyzer.Write for more information.

Definition at line 258 of file looper.py.

References looper.Looper.analyzers, Config.Process.analyzers, event.Event.setup, looper.Looper.setup, and options.HLTProcessOptions.setup.

    def write(self):
        """Writes all analyzers.

        See Analyzer.Write for more information.
        """
        for analyzer in self.analyzers:
            analyzer.write(self.setup)
        self.setup.close() 

Member Data Documentation

analyzers

looper.Looper.analyzers

Definition at line 88 of file looper.py.

Referenced by looper.Looper.loop(), and looper.Looper.write().

cfg_comp

looper.Looper.cfg_comp

Definition at line 86 of file looper.py.

Referenced by objects.LeptonAnalyzer.LeptonAnalyzer.__init__(), core.JSONAnalyzer.JSONAnalyzer.__init__(), objects.JetAnalyzer.JetAnalyzer.__init__(), looper.Looper._build(), JetAnalyzer.JetAnalyzer.declareHandles(), objects.VertexAnalyzer.VertexAnalyzer.declareHandles(), core.AutoFillTreeProducer.AutoFillTreeProducer.declareVariables(), core.AutoFillTreeProducer.AutoFillTreeProducer.fillCoreVariables(), core.AutoFillTreeProducer.AutoFillTreeProducer.fillTree(), core.AutoFillTreeProducer.AutoFillTreeProducer.getPythonWrapper(), looper.Looper.loop(), gen.LHEAnalyzer.LHEAnalyzer.process(), SimpleJetAnalyzer.SimpleJetAnalyzer.process(), gen.LHEWeightAnalyzer.LHEWeightAnalyzer.process(), gen.HiggsDecayModeAnalyzer.HiggsDecayModeAnalyzer.process(), gen.GenHeavyFlavourAnalyzer.GenHeavyFlavourAnalyzer.process(), gen.PDFWeightsAnalyzer.PDFWeightsAnalyzer.process(), core.JSONAnalyzer.JSONAnalyzer.process(), core.SkimAnalyzerCount.SkimAnalyzerCount.process(), AlphaTAnalyzer.AlphaTAnalyzer.process(), JetAnalyzer.JetAnalyzer.process(), core.PileUpAnalyzer.PileUpAnalyzer.process(), objects.VertexAnalyzer.VertexAnalyzer.process(), objects.TauAnalyzer.TauAnalyzer.process(), gen.GeneratorAnalyzer.GeneratorAnalyzer.process(), objects.PhotonAnalyzer.PhotonAnalyzer.process(), objects.LeptonAnalyzer.LeptonAnalyzer.process(), core.Analyzer.Analyzer.readCollections(), DiLeptonAnalyzer.DiLeptonAnalyzer.selectionSequence(), JetAnalyzer.JetAnalyzer.testBJet(), and core.PileUpAnalyzer.PileUpAnalyzer.write().

classes

looper.Looper.classes

Definition at line 87 of file looper.py.

config

looper.Looper.config

Definition at line 76 of file looper.py.

event

looper.Looper.event

Definition at line 120 of file looper.py.

Referenced by Types.EventID.cppID(), and looper.Looper.process().

events

looper.Looper.events

Definition at line 105 of file looper.py.

Referenced by looper.Looper.loop(), and looper.Looper.process().

firstEvent

looper.Looper.firstEvent

Definition at line 90 of file looper.py.

Referenced by looper.Looper.loop().

iEvent

looper.Looper.iEvent

Definition at line 232 of file looper.py.

logger

looper.Looper.logger

Definition at line 79 of file looper.py.

Referenced by looper.Looper.loop().

memLast

looper.Looper.memLast

Definition at line 94 of file looper.py.

memReportFirstEvent

looper.Looper.memReportFirstEvent

Definition at line 93 of file looper.py.

name

looper.Looper.name

Definition at line 77 of file looper.py.

Referenced by FWLite.ElectronMVAID.__call__(), dirstructure.Directory.__create_pie_image(), DisplayManager.DisplayManager.__del__(), dqm_interfaces.DirID.__eq__(), dirstructure.Directory.__get_full_path(), dirstructure.Comparison.__get_img_name(), dirstructure.Comparison.__make_image(), dirstructure.Directory.__repr__(), dqm_interfaces.DirID.__repr__(), dirstructure.Comparison.__repr__(), config.CFG.__str__(), counter.Counter.__str__(), average.Average.__str__(), FWLite.WorkingPoints._reformat_cut_definitions(), dataset.CMSDataset.buildListOfFiles(), dataset.LocalDataset.buildListOfFiles(), dataset.CMSDataset.buildListOfFilesDBS(), dirstructure.Directory.calcStats(), validation.Sample.digest(), python.rootplot.utilities.Hist.divide(), python.rootplot.utilities.Hist.divide_wilson(), DisplayManager.DisplayManager.Draw(), utils.StatisticalTest.get_status(), production_tasks.Task.getname(), dataset.CMSDataset.getPrimaryDatasetEntries(), dataset.PrivateDataset.getPrimaryDatasetEntries(), VIDSelectorBase.VIDSelectorBase.initialize(), dirstructure.Directory.print_report(), dataset.BaseDataset.printInfo(), dataset.Dataset.printInfo(), production_tasks.MonitorJobs.run(), python.rootplot.utilities.Hist.TGraph(), python.rootplot.utilities.Hist.TH1F(), counter.Counter.write(), and average.Average.write().

nEvents

looper.Looper.nEvents

Definition at line 89 of file looper.py.

Referenced by looper.Looper.loop().

nPrint

looper.Looper.nPrint

Definition at line 91 of file looper.py.

outDir

looper.Looper.outDir

Definition at line 78 of file looper.py.

Referenced by looper.Looper._build().

setup

looper.Looper.setup

Definition at line 128 of file looper.py.

Referenced by o2olib.O2OTool.execute(), looper.Looper.loop(), looper.Looper.process(), and looper.Looper.write().

start_time

looper.Looper.start_time

Definition at line 182 of file looper.py.

Referenced by progressbar.ProgressBar.__next__(), and progressbar.ProgressBar.update().

start_time_event

looper.Looper.start_time_event

Definition at line 183 of file looper.py.

timeReport

looper.Looper.timeReport

Definition at line 92 of file looper.py.