CMS 3D CMS Logo

 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes | Private Member Functions
looper.Looper Class Reference
Inheritance diagram for looper.Looper:

Public Member Functions

def __init__
 
def loop
 
def process
 
def write
 

Public Attributes

 analyzers
 
 cfg_comp
 
 classes
 
 event
 
 events
 
 firstEvent
 
 iEvent
 
 logger
 
 name
 
 nEvents
 
 nPrint
 
 outDir
 
 setup
 
 start_time
 
 start_time_event
 
 timeReport
 

Private Member Functions

def _build
 
def _prepareOutput
 

Detailed Description

Creates a set of analyzers, and schedules the event processing.

Definition at line 47 of file looper.py.

Constructor & Destructor Documentation

def looper.Looper.__init__ (   self,
  name,
  config,
  nEvents = None,
  firstEvent = 0,
  nPrint = 0,
  timeReport = False,
  quiet = False 
)
Handles the processing of an event sample.
An Analyzer is built for each Config.Analyzer present
in sequence. The Looper can then be used to process an event,
or a collection of events.

Parameters:
name    : name of the Looper, will be used as the output directory name
config  : process configuration information, see Config
nEvents : number of events to process. Defaults to all.
firstEvent : first event to process. Defaults to the first one.
nPrint  : number of events to print at the beginning

Definition at line 56 of file looper.py.

56 
57  quiet=False):
58  """Handles the processing of an event sample.
59  An Analyzer is built for each Config.Analyzer present
60  in sequence. The Looper can then be used to process an event,
61  or a collection of events.
62 
63  Parameters:
64  name : name of the Looper, will be used as the output directory name
65  config : process configuration information, see Config
66  nEvents : number of events to process. Defaults to all.
67  firstEvent : first event to process. Defaults to the first one.
68  nPrint : number of events to print at the beginning
69  """
70 
71  self.name = self._prepareOutput(name)
72  self.outDir = self.name
73  self.logger = logging.getLogger( self.name )
74  self.logger.addHandler(logging.FileHandler('/'.join([self.name,
75  'log.txt'])))
76  self.logger.propagate = False
77  if not quiet:
78  self.logger.addHandler( logging.StreamHandler(sys.stdout) )
79 
80  self.cfg_comp = config.components[0]
81  self.classes = {}
82  self.analyzers = map( self._build, config.sequence )
83  self.nEvents = nEvents
84  self.firstEvent = firstEvent
85  self.nPrint = int(nPrint)
86  self.timeReport = [ {'time':0.0,'events':0} for a in self.analyzers ] if timeReport else False
87  tree_name = None
88  if( hasattr(self.cfg_comp, 'tree_name') ):
89  tree_name = self.cfg_comp.tree_name
90  if len(self.cfg_comp.files)==0:
91  errmsg = 'please provide at least an input file in the files attribute of this component\n' + str(self.cfg_comp)
92  raise ValueError( errmsg )
93  if hasattr(config,"preprocessor") and config.preprocessor is not None :
94  self.cfg_comp = config.preprocessor.run(self.cfg_comp,self.outDir,firstEvent,nEvents)
95  if hasattr(self.cfg_comp,"options"):
96  print self.cfg_comp.files,self.cfg_comp.options
97  self.events = config.events_class(self.cfg_comp.files, tree_name,options=self.cfg_comp.options)
98  else :
99  self.events = config.events_class(self.cfg_comp.files, tree_name)
100  if hasattr(self.cfg_comp, 'fineSplit'):
101  fineSplitIndex, fineSplitFactor = self.cfg_comp.fineSplit
102  if fineSplitFactor > 1:
103  if len(self.cfg_comp.files) != 1:
104  raise RuntimeError, "Any component with fineSplit > 1 is supposed to have just a single file, while %s has %s" % (self.cfg_comp.name, self.cfg_comp.files)
105  totevents = min(len(self.events),int(nEvents)) if (nEvents and int(nEvents) not in [-1,0]) else len(self.events)
106  self.nEvents = int(ceil(totevents/float(fineSplitFactor)))
107  self.firstEvent = firstEvent + fineSplitIndex * self.nEvents
108  if self.firstEvent + self.nEvents >= totevents:
109  self.nEvents = totevents - self.firstEvent
110  #print "For component %s will process %d events starting from the %d one, ending at %d excluded" % (self.cfg_comp.name, self.nEvents, self.firstEvent, self.nEvents + self.firstEvent)
111  # self.event is set in self.process
112  self.event = None
113  services = dict()
114  for cfg_serv in config.services:
115  service = self._build(cfg_serv)
116  services[cfg_serv.name] = service
117  # would like to provide a copy of the config to the setup,
118  # so that analyzers cannot modify the config of other analyzers.
119  # but cannot copy the autofill config.
120  self.setup = Setup(config, services)
def _prepareOutput
Definition: looper.py:126
def _build
Definition: looper.py:121
T min(T a, T b)
Definition: MathUtil.h:58
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
if(conf.exists("allCellsPositionCalc"))

Member Function Documentation

def looper.Looper._build (   self,
  cfg 
)
private

Definition at line 121 of file looper.py.

References analyzer.Analyzer.cfg_comp, looper.Looper.cfg_comp, and looper.Looper.outDir.

122  def _build(self, cfg):
123  theClass = cfg.class_object
124  obj = theClass( cfg, self.cfg_comp, self.outDir )
125  return obj
def _build
Definition: looper.py:121
def looper.Looper._prepareOutput (   self,
  name 
)
private

Definition at line 126 of file looper.py.

127  def _prepareOutput(self, name):
128  index = 0
129  tmpname = name
130  while True and index < 2000:
131  try:
132  # print 'mkdir', self.name
133  os.mkdir( tmpname )
134  break
135  except OSError:
136  index += 1
137  tmpname = '%s_%d' % (name, index)
138  if index == 2000:
139  raise ValueError( "More than 2000 output folder with same name or 2000 attempts failed, please clean-up, change name or check permissions")
140  return tmpname
141 
def _prepareOutput
Definition: looper.py:126
def looper.Looper.loop (   self)
Loop on a given number of events.

At the beginning of the loop, 
Analyzer.beginLoop is called for each Analyzer.
At each event, self.process is called.
At the end of the loop, Analyzer.endLoop is called.

Definition at line 142 of file looper.py.

References looper.Looper.analyzers, Config.Process.analyzers, analyzer.Analyzer.cfg_comp, looper.Looper.cfg_comp, generateEDF.LumiInfo.events, relval_steps.InputInfo.events, HitEff.events, ME::Header.events, looper.Looper.events, options.HLTProcessOptions.events, Printer.Printer.firstEvent, lhef::LHEReader.firstEvent, StudyHLT.firstEvent, looper.Looper.firstEvent, LeptonRecoSkim.firstEvent, SiStripMonitorPedestals.firstEvent, IsoTrackCalib.firstEvent, SiStripMonitorCluster.firstEvent, SiStripMonitorDigi.firstEvent, TriggerValidator.firstEvent, SiStripMonitorTrack.firstEvent, HiggsTo2GammaSkim.nEvents, HiggsToZZ4LeptonsSkim.nEvents, JGJFilter.nEvents, HiggsToZZ4LeptonsSkimEff.nEvents, MCDijetResonance.nEvents, EgammaProbeSelector.nEvents, QualityTester.nEvents, AlCaHcalNoiseProducer.nEvents, DTScalerInfoTask.nEvents, MCatNLOSource.nEvents, AlpgenHeader.nEvents, HeavyChHiggsToTauNuSkim.nEvents, DTLocalTriggerLutTask.nEvents, looper.Looper.nEvents, DTLocalTriggerBaseTask.nEvents, MuonIsolationDQM.nEvents, MatacqProducer::stats_t.nEvents, cscdqm::Configuration.nEvents, event.Event.setup, and looper.Looper.setup.

143  def loop(self):
144  """Loop on a given number of events.
145 
146  At the beginning of the loop,
147  Analyzer.beginLoop is called for each Analyzer.
148  At each event, self.process is called.
149  At the end of the loop, Analyzer.endLoop is called.
150  """
151  nEvents = self.nEvents
152  firstEvent = self.firstEvent
153  iEv = firstEvent
154  if nEvents is None or int(nEvents) > len(self.events) :
155  nEvents = len(self.events)
156  else:
157  nEvents = int(nEvents)
158  eventSize = nEvents
159  self.logger.info(
160  'starting loop at event {firstEvent} '\
161  'to process {eventSize} events.'.format(firstEvent=firstEvent,
162  eventSize=eventSize))
163  self.logger.info( str( self.cfg_comp ) )
164  for analyzer in self.analyzers:
165  analyzer.beginLoop(self.setup)
166  try:
167  for iEv in range(firstEvent, firstEvent+eventSize):
168  # if iEv == nEvents:
169  # break
170  if iEv%100 ==0:
171  # print 'event', iEv
172  if not hasattr(self,'start_time'):
173  print 'event', iEv
174  self.start_time = timeit.default_timer()
175  self.start_time_event = iEv
176  else:
177  print 'event %d (%.1f ev/s)' % (iEv, (iEv-self.start_time_event)/float(timeit.default_timer() - self.start_time))
178 
179  self.process( iEv )
180  if iEv<self.nPrint:
181  print self.event
182 
183  except UserWarning:
184  print 'Stopped loop following a UserWarning exception'
185 
186  info = self.logger.info
187  warning = self.logger.warning
188  warning('number of events processed: {nEv}'.format(nEv=iEv+1))
189  warning('')
190  info( self.cfg_comp )
191  info('')
192  for analyzer in self.analyzers:
193  analyzer.endLoop(self.setup)
194  if self.timeReport:
195  allev = max([x['events'] for x in self.timeReport])
196  warning("\n ---- TimeReport (all times in ms; first evt is skipped) ---- ")
197  warning("%9s %9s %9s %9s %6s %s" % ("processed","all evts","time/proc", " time/all", " [%] ", "analyer"))
198  warning("%9s %9s %9s %9s %6s %s" % ("---------","--------","---------", "---------", " -----", "-------------"))
199  sumtime = sum(rep['time'] for rep in self.timeReport)
200  passev = self.timeReport[-1]['events']
201  for ana,rep in zip(self.analyzers,self.timeReport):
202  timePerProcEv = rep['time']/(rep['events']-1) if rep['events'] > 1 else 0
203  timePerAllEv = rep['time']/(allev-1) if allev > 1 else 0
204  fracAllEv = rep['time']/sumtime
205  warning( "%9d %9d %10.2f %10.2f %5.1f%% %s" % ( rep['events'], allev, 1000*timePerProcEv, 1000*timePerAllEv, 100.0*fracAllEv, ana.name))
206  totPerProcEv = sumtime/(passev-1) if passev > 1 else 0
207  totPerAllEv = sumtime/(allev-1) if allev > 1 else 0
208  warning("%9s %9s %9s %9s %s" % ("---------","--------","---------", "---------", "-------------"))
209  warning("%9d %9d %10.2f %10.2f %5.1f%% %s" % ( passev, allev, 1000*totPerProcEv, 1000*totPerAllEv, 100.0, "TOTAL"))
210  warning("")
static const TGPicture * info(bool iBackgroundIsBlack)
tuple zip
Definition: archive.py:476
def process
Definition: looper.py:211
def looper.Looper.process (   self,
  iEv 
)
Run event processing for all analyzers in the sequence.

This function is called by self.loop,
but can also be called directly from
the python interpreter, to jump to a given event.

Definition at line 211 of file looper.py.

References TB06Reco.event, TB06RecoH2.event, core.AutoHandle.AutoHandle.event, MuonPair.event, LHAupLesHouches.event, EcalPerEvtMatacqAnalyzer.event, WZInterestingEventSelector::event.event, EcalStatusAnalyzer.event, lhef::LHEProxy.event, HitEff.event, MCatNLOSource.event, EcalMatacqAnalyzer.event, looper.Looper.event, EcalTestPulseAnalyzer.event, EcalABAnalyzer.event, edm::service::SimpleMemoryCheck::SignificantEvent.event, EcalPerEvtLaserAnalyzer.event, EcalLaserAnalyzer.event, EcalLaserAnalyzer2.event, FastTimerService::Timing.event, FastTimerService::TimingPerProcess.event, FastTimerService::SummaryPlots.event, FastTimerService::SummaryPlotsPerProcess.event, FastTimerService::SummaryProfiles.event, FastTimerService::SummaryProfilesPerProcess.event, generateEDF.LumiInfo.events, relval_steps.InputInfo.events, HitEff.events, ME::Header.events, looper.Looper.events, options.HLTProcessOptions.events, event.Event.setup, and looper.Looper.setup.

Referenced by ConfigBuilder.ConfigBuilder.__init__(), ConfigBuilder.ConfigBuilder.addExtraStream(), ConfigBuilder.ConfigBuilder.anyOf(), ConfigBuilder.ConfigBuilder.completeInputCommand(), ConfigBuilder.ConfigBuilder.doNotInlineEventContent(), ConfigBuilder.ConfigBuilder.PrintAllModules.leave(), ConfigBuilder.ConfigBuilder.prepare(), ConfigBuilder.ConfigBuilder.prepare_ALCA(), ConfigBuilder.ConfigBuilder.prepare_DQM(), ConfigBuilder.ConfigBuilder.prepare_FASTSIM(), ConfigBuilder.ConfigBuilder.prepare_HARVESTING(), ConfigBuilder.ConfigBuilder.prepare_HLT(), ConfigBuilder.ConfigBuilder.prepare_LHE(), ConfigBuilder.ConfigBuilder.prepare_PATFILTER(), ConfigBuilder.ConfigBuilder.prepare_VALIDATION(), ConfigBuilder.ConfigBuilder.renameHLTprocessInSequence(), ConfigBuilder.ConfigBuilder.renameInputTagsInSequence(), and ConfigBuilder.ConfigBuilder.scheduleSequence().

212  def process(self, iEv ):
213  """Run event processing for all analyzers in the sequence.
214 
215  This function is called by self.loop,
216  but can also be called directly from
217  the python interpreter, to jump to a given event.
218  """
219  self.event = Event(iEv, self.events[iEv], self.setup)
220  self.iEvent = iEv
221  for i,analyzer in enumerate(self.analyzers):
222  if not analyzer.beginLoopCalled:
223  analyzer.beginLoop(self.setup)
224  start = timeit.default_timer()
225  ret = analyzer.process( self.event )
226  if self.timeReport:
227  self.timeReport[i]['events'] += 1
228  if self.timeReport[i]['events'] > 0:
229  self.timeReport[i]['time'] += timeit.default_timer() - start
230  if ret == False:
231  return (False, analyzer.name)
232  if iEv<self.nPrint:
233  self.logger.info( self.event.__str__() )
234  return (True, analyzer.name)
def process
Definition: looper.py:211
def looper.Looper.write (   self)
Writes all analyzers.

See Analyzer.Write for more information.

Definition at line 235 of file looper.py.

References looper.Looper.analyzers, Config.Process.analyzers, event.Event.setup, and looper.Looper.setup.

236  def write(self):
237  """Writes all analyzers.
238 
239  See Analyzer.Write for more information.
240  """
241  for analyzer in self.analyzers:
242  analyzer.write(self.setup)
243  self.setup.close()
244 

Member Data Documentation

looper.Looper.analyzers

Definition at line 81 of file looper.py.

Referenced by looper.Looper.loop(), and looper.Looper.write().

looper.Looper.cfg_comp

Definition at line 79 of file looper.py.

Referenced by looper.Looper._build(), core.AutoFillTreeProducer.AutoFillTreeProducer.fillCoreVariables(), looper.Looper.loop(), and JetAnalyzer.JetAnalyzer.process().

looper.Looper.classes

Definition at line 80 of file looper.py.

looper.Looper.event

Definition at line 111 of file looper.py.

Referenced by Types.EventID.cppID(), and looper.Looper.process().

looper.Looper.events

Definition at line 96 of file looper.py.

Referenced by eventsfwlite.Events.__getattr__(), eventsfwlite.Events.__init__(), looper.Looper.loop(), and looper.Looper.process().

looper.Looper.firstEvent

Definition at line 83 of file looper.py.

Referenced by looper.Looper.loop().

looper.Looper.iEvent

Definition at line 219 of file looper.py.

looper.Looper.logger

Definition at line 72 of file looper.py.

looper.Looper.name

Definition at line 70 of file looper.py.

Referenced by dirstructure.Directory.__create_pie_image(), dqm_interfaces.DirID.__eq__(), dirstructure.Directory.__get_full_path(), dirstructure.Comparison.__get_img_name(), dataset.Dataset.__getDataType(), dataset.Dataset.__getFileInfoList(), cuy.divideElement.__init__(), cuy.plotElement.__init__(), cuy.additionElement.__init__(), cuy.superimposeElement.__init__(), cuy.graphElement.__init__(), dirstructure.Comparison.__make_image(), dirstructure.Directory.__repr__(), dqm_interfaces.DirID.__repr__(), dirstructure.Comparison.__repr__(), config.CFG.__str__(), counter.Counter.__str__(), average.Average.__str__(), dirstructure.Directory.calcStats(), python.rootplot.utilities.Hist.divide(), python.rootplot.utilities.Hist.divide_wilson(), utils.StatisticalTest.get_status(), dataset.CMSDataset.getPrimaryDatasetEntries(), dataset.PrivateDataset.getPrimaryDatasetEntries(), VIDSelectorBase.VIDSelectorBase.initialize(), dirstructure.Directory.print_report(), dataset.BaseDataset.printInfo(), dataset.Dataset.printInfo(), python.rootplot.utilities.Hist.TGraph(), python.rootplot.utilities.Hist.TH1F(), Vispa.Views.PropertyView.Property.valueChanged(), counter.Counter.write(), and average.Average.write().

looper.Looper.nEvents

Definition at line 82 of file looper.py.

Referenced by looper.Looper.loop().

looper.Looper.nPrint

Definition at line 84 of file looper.py.

looper.Looper.outDir

Definition at line 71 of file looper.py.

Referenced by looper.Looper._build().

looper.Looper.setup

Definition at line 119 of file looper.py.

Referenced by looper.Looper.loop(), looper.Looper.process(), and looper.Looper.write().

looper.Looper.start_time

Definition at line 173 of file looper.py.

Referenced by progressbar.ProgressBar.__next__(), and progressbar.ProgressBar.update().

looper.Looper.start_time_event

Definition at line 174 of file looper.py.

looper.Looper.timeReport

Definition at line 85 of file looper.py.