CMS 3D CMS Logo

 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes | Private Member Functions
looper.Looper Class Reference
Inheritance diagram for looper.Looper:

Public Member Functions

def __init__
 
def loop
 
def process
 
def write
 

Public Attributes

 analyzers
 
 cfg_comp
 
 classes
 
 event
 
 events
 
 firstEvent
 
 iEvent
 
 logger
 
 name
 
 nEvents
 
 nPrint
 
 outDir
 
 setup
 
 start_time
 
 start_time_event
 
 timeReport
 

Private Member Functions

def _build
 
def _prepareOutput
 

Detailed Description

Creates a set of analyzers, and schedules the event processing.

Definition at line 49 of file looper.py.

Constructor & Destructor Documentation

def looper.Looper.__init__ (   self,
  name,
  config,
  nEvents = None,
  firstEvent = 0,
  nPrint = 0,
  timeReport = False 
)
Handles the processing of an event sample.
An Analyzer is built for each Config.Analyzer present
in sequence. The Looper can then be used to process an event,
or a collection of events.

Parameters:
name    : name of the Looper, will be used as the output directory name
config  : process configuration information, see Config
nEvents : number of events to process. Defaults to all.
firstEvent : first event to process. Defaults to the first one.
nPrint  : number of events to print at the beginning

Definition at line 55 of file looper.py.

55 
56  firstEvent=0, nPrint=0, timeReport=False ):
57  """Handles the processing of an event sample.
58  An Analyzer is built for each Config.Analyzer present
59  in sequence. The Looper can then be used to process an event,
60  or a collection of events.
61 
62  Parameters:
63  name : name of the Looper, will be used as the output directory name
64  config : process configuration information, see Config
65  nEvents : number of events to process. Defaults to all.
66  firstEvent : first event to process. Defaults to the first one.
67  nPrint : number of events to print at the beginning
68  """
69 
70  self.name = self._prepareOutput(name)
71  self.outDir = self.name
72  self.logger = logging.getLogger( self.name )
73  self.logger.addHandler(logging.FileHandler('/'.join([self.name,
74  'log.txt'])))
75  self.logger.addHandler( logging.StreamHandler(sys.stdout) )
76 
77  self.cfg_comp = config.components[0]
78  self.classes = {}
79  self.analyzers = map( self._build, config.sequence )
80  self.nEvents = nEvents
81  self.firstEvent = firstEvent
82  self.nPrint = int(nPrint)
83  self.timeReport = [ {'time':0.0,'events':0} for a in self.analyzers ] if timeReport else False
84  tree_name = None
85  if( hasattr(self.cfg_comp, 'tree_name') ):
86  tree_name = self.cfg_comp.tree_name
87  if len(self.cfg_comp.files)==0:
88  errmsg = 'please provide at least an input file in the files attribute of this component\n' + str(self.cfg_comp)
89  raise ValueError( errmsg )
90  self.events = config.events_class(self.cfg_comp.files, tree_name)
91  if hasattr(self.cfg_comp, 'fineSplit'):
92  fineSplitIndex, fineSplitFactor = self.cfg_comp.fineSplit
93  if fineSplitFactor > 1:
94  if len(self.cfg_comp.files) != 1:
95  raise RuntimeError, "Any component with fineSplit > 1 is supposed to have just a single file, while %s has %s" % (self.cfg_comp.name, self.cfg_comp.files)
96  totevents = min(len(self.events),int(nEvents)) if (nEvents and int(nEvents) not in [-1,0]) else len(self.events)
97  self.nEvents = int(ceil(totevents/float(fineSplitFactor)))
98  self.firstEvent = firstEvent + fineSplitIndex * self.nEvents
99  #print "For component %s will process %d events starting from the %d one" % (self.cfg_comp.name, self.nEvents, self.firstEvent)
100  # self.event is set in self.process
101  self.event = None
102  services = dict()
103  for cfg_serv in config.services:
104  service = self._build(cfg_serv)
105  services[cfg_serv.name] = service
106  # would like to provide a copy of the config to the setup,
107  # so that analyzers cannot modify the config of other analyzers.
108  # but cannot copy the autofill config.
109  self.setup = Setup(config, services)
def _prepareOutput
Definition: looper.py:115
def _build
Definition: looper.py:110
T min(T a, T b)
Definition: MathUtil.h:58
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
if(conf.exists("allCellsPositionCalc"))

Member Function Documentation

def looper.Looper._build (   self,
  cfg 
)
private

Definition at line 110 of file looper.py.

References analyzer.Analyzer.cfg_comp, looper.Looper.cfg_comp, and looper.Looper.outDir.

111  def _build(self, cfg):
112  theClass = cfg.class_object
113  obj = theClass( cfg, self.cfg_comp, self.outDir )
114  return obj
def _build
Definition: looper.py:110
def looper.Looper._prepareOutput (   self,
  name 
)
private

Definition at line 115 of file looper.py.

116  def _prepareOutput(self, name):
117  index = 0
118  tmpname = name
119  while True and index < 2000:
120  try:
121  # print 'mkdir', self.name
122  os.mkdir( tmpname )
123  break
124  except OSError:
125  index += 1
126  tmpname = '%s_%d' % (name, index)
127  if index == 2000:
128  raise ValueError( "More than 2000 output folder with same name or 2000 attempts failed, please clean-up, change name or check permissions")
129  return tmpname
130 
def _prepareOutput
Definition: looper.py:115
def looper.Looper.loop (   self)
Loop on a given number of events.

At the beginning of the loop, 
Analyzer.beginLoop is called for each Analyzer.
At each event, self.process is called.
At the end of the loop, Analyzer.endLoop is called.

Definition at line 131 of file looper.py.

References looper.Looper.analyzers, Config.Process.analyzers, analyzer.Analyzer.cfg_comp, looper.Looper.cfg_comp, eventsfwlite.Events.events, generateEDF.LumiInfo.events, relval_steps.InputInfo.events, HitEff.events, ME::Header.events, looper.Looper.events, options.HLTProcessOptions.events, Printer.Printer.firstEvent, lhef::LHEReader.firstEvent, StudyHLT.firstEvent, looper.Looper.firstEvent, LeptonRecoSkim.firstEvent, SiStripMonitorPedestals.firstEvent, IsoTrackCalib.firstEvent, SiStripMonitorCluster.firstEvent, SiStripMonitorDigi.firstEvent, TriggerValidator.firstEvent, SiStripMonitorTrack.firstEvent, HiggsTo2GammaSkim.nEvents, HiggsToZZ4LeptonsSkim.nEvents, JGJFilter.nEvents, HiggsToZZ4LeptonsSkimEff.nEvents, MCDijetResonance.nEvents, EgammaProbeSelector.nEvents, QualityTester.nEvents, AlCaHcalNoiseProducer.nEvents, DTFineDelayCorr.nEvents, MCatNLOSource.nEvents, HeavyChHiggsToTauNuSkim.nEvents, AlpgenHeader.nEvents, DTScalerInfoTask.nEvents, looper.Looper.nEvents, DTLocalTriggerLutTask.nEvents, DTLocalTriggerBaseTask.nEvents, MuonIsolationDQM.nEvents, MatacqProducer::stats_t.nEvents, cscdqm::Configuration.nEvents, event.Event.setup, and looper.Looper.setup.

132  def loop(self):
133  """Loop on a given number of events.
134 
135  At the beginning of the loop,
136  Analyzer.beginLoop is called for each Analyzer.
137  At each event, self.process is called.
138  At the end of the loop, Analyzer.endLoop is called.
139  """
140  nEvents = self.nEvents
141  firstEvent = self.firstEvent
142  iEv = firstEvent
143  if nEvents is None or int(nEvents) > len(self.events) :
144  nEvents = len(self.events)
145  else:
146  nEvents = int(nEvents)
147  eventSize = nEvents
148  self.logger.warning(
149  'starting loop at event {firstEvent} '\
150  'to process {eventSize} events.'.format(firstEvent=firstEvent,
151  eventSize=eventSize))
152  self.logger.warning( str( self.cfg_comp ) )
153  for analyzer in self.analyzers:
154  analyzer.beginLoop(self.setup)
155  try:
156  for iEv in range(firstEvent, firstEvent+eventSize):
157  # if iEv == nEvents:
158  # break
159  if iEv%100 ==0:
160  # print 'event', iEv
161  if not hasattr(self,'start_time'):
162  print 'event', iEv
163  self.start_time = timeit.default_timer()
164  self.start_time_event = iEv
165  else:
166  print 'event %d (%.1f ev/s)' % (iEv, (iEv-self.start_time_event)/float(timeit.default_timer() - self.start_time))
167 
168  self.process( iEv )
169  if iEv<self.nPrint:
170  print self.event
171 
172  except UserWarning:
173  print 'Stopped loop following a UserWarning exception'
174  for analyzer in self.analyzers:
175  analyzer.endLoop(self.setup)
176  warn = self.logger.warning
177  warn('')
178  warn( self.cfg_comp )
179  warn('')
180  warn('number of events processed: {nEv}'.format(nEv=iEv+1))
def process
Definition: looper.py:181
def warn
Definition: __init__.py:20
def looper.Looper.process (   self,
  iEv 
)
Run event processing for all analyzers in the sequence.

This function is called by self.loop,
but can also be called directly from
the python interpreter, to jump to a given event.

Definition at line 181 of file looper.py.

References TB06Reco.event, TB06RecoH2.event, MuonPair.event, LHAupLesHouches.event, EcalPerEvtMatacqAnalyzer.event, WZInterestingEventSelector::event.event, EcalStatusAnalyzer.event, lhef::LHEProxy.event, HitEff.event, MCatNLOSource.event, EcalMatacqAnalyzer.event, cms::GammaJetAnalysis.event, looper.Looper.event, EcalTestPulseAnalyzer.event, EcalABAnalyzer.event, edm::service::SimpleMemoryCheck::SignificantEvent.event, EcalPerEvtLaserAnalyzer.event, EcalLaserAnalyzer.event, EcalLaserAnalyzer2.event, FastTimerService::Timing.event, FastTimerService::TimingPerProcess.event, FastTimerService::SummaryPlots.event, FastTimerService::SummaryPlotsPerProcess.event, FastTimerService::SummaryProfiles.event, FastTimerService::SummaryProfilesPerProcess.event, eventsfwlite.Events.events, generateEDF.LumiInfo.events, relval_steps.InputInfo.events, HitEff.events, ME::Header.events, looper.Looper.events, options.HLTProcessOptions.events, event.Event.setup, and looper.Looper.setup.

Referenced by ConfigBuilder.ConfigBuilder.__init__(), ConfigBuilder.ConfigBuilder.addExtraStream(), ConfigBuilder.ConfigBuilder.anyOf(), ConfigBuilder.ConfigBuilder.completeInputCommand(), ConfigBuilder.ConfigBuilder.doNotInlineEventContent(), ConfigBuilder.ConfigBuilder.PrintAllModules.leave(), ConfigBuilder.ConfigBuilder.prepare(), ConfigBuilder.ConfigBuilder.prepare_ALCA(), ConfigBuilder.ConfigBuilder.prepare_DQM(), ConfigBuilder.ConfigBuilder.prepare_FASTSIM(), ConfigBuilder.ConfigBuilder.prepare_HARVESTING(), ConfigBuilder.ConfigBuilder.prepare_HLT(), ConfigBuilder.ConfigBuilder.prepare_LHE(), ConfigBuilder.ConfigBuilder.prepare_VALIDATION(), ConfigBuilder.ConfigBuilder.renameHLTprocessInSequence(), ConfigBuilder.ConfigBuilder.renameInputTagsInSequence(), and ConfigBuilder.ConfigBuilder.scheduleSequence().

182  def process(self, iEv ):
183  """Run event processing for all analyzers in the sequence.
184 
185  This function is called by self.loop,
186  but can also be called directly from
187  the python interpreter, to jump to a given event.
188  """
189  self.event = Event(iEv, self.events[iEv], self.setup)
190  self.iEvent = iEv
191  for i,analyzer in enumerate(self.analyzers):
192  if not analyzer.beginLoopCalled:
193  analyzer.beginLoop()
194  start = timeit.default_timer()
195  ret = analyzer.process( self.event )
196  if self.timeReport:
197  self.timeReport[i]['events'] += 1
198  if self.timeReport[i]['events'] > 0:
199  self.timeReport[i]['time'] += timeit.default_timer() - start
200  if ret == False:
201  return (False, analyzer.name)
202  return (True, analyzer.name)
def process
Definition: looper.py:181
def looper.Looper.write (   self)
Writes all analyzers.

See Analyzer.Write for more information.

Definition at line 203 of file looper.py.

References looper.Looper.analyzers, Config.Process.analyzers, bookConverter.max, event.Event.setup, looper.Looper.setup, looper.Looper.timeReport, and archive.zip.

204  def write(self):
205  """Writes all analyzers.
206 
207  See Analyzer.Write for more information.
208  """
209  for analyzer in self.analyzers:
210  analyzer.write(self.setup)
211  self.setup.close()
212 
213  if self.timeReport:
214  allev = max([x['events'] for x in self.timeReport])
215  print "\n ---- TimeReport (all times in ms; first evt is skipped) ---- "
216  print "%9s %9s %9s %9s %s" % ("processed","all evts","time/proc", " time/all", "analyer")
217  print "%9s %9s %9s %9s %s" % ("---------","--------","---------", "---------", "-------------")
218  for ana,rep in zip(self.analyzers,self.timeReport):
219  print "%9d %9d %10.2f %10.2f %s" % ( rep['events'], allev, 1000*rep['time']/(rep['events']-1) if rep['events']>1 else 0, 1000*rep['time']/(allev-1) if allev > 1 else 0, ana.name)
220  print ""
221  pass
222 
tuple zip
Definition: archive.py:476

Member Data Documentation

looper.Looper.analyzers

Definition at line 78 of file looper.py.

Referenced by looper.Looper.loop(), and looper.Looper.write().

looper.Looper.cfg_comp

Definition at line 76 of file looper.py.

Referenced by looper.Looper._build(), core.AutoFillTreeProducer.AutoFillTreeProducer.fillCoreVariables(), looper.Looper.loop(), and JetAnalyzer.JetAnalyzer.process().

looper.Looper.classes

Definition at line 77 of file looper.py.

looper.Looper.event

Definition at line 100 of file looper.py.

Referenced by Types.EventID.cppID(), core.AutoHandle.AutoHandle.Load(), looper.Looper.process(), and core.AutoHandle.AutoHandle.product().

looper.Looper.events

Definition at line 89 of file looper.py.

Referenced by looper.Looper.loop(), and looper.Looper.process().

looper.Looper.firstEvent

Definition at line 80 of file looper.py.

Referenced by looper.Looper.loop().

looper.Looper.iEvent

Definition at line 189 of file looper.py.

looper.Looper.logger

Definition at line 71 of file looper.py.

looper.Looper.name

Definition at line 69 of file looper.py.

Referenced by dirstructure.Directory.__create_pie_image(), dqm_interfaces.DirID.__eq__(), dirstructure.Directory.__get_full_path(), dirstructure.Comparison.__get_img_name(), dataset.Dataset.__getDataType(), dataset.Dataset.__getFileInfoList(), cuy.divideElement.__init__(), cuy.plotElement.__init__(), cuy.additionElement.__init__(), cuy.superimposeElement.__init__(), cuy.graphElement.__init__(), dirstructure.Comparison.__make_image(), dirstructure.Directory.__repr__(), dqm_interfaces.DirID.__repr__(), dirstructure.Comparison.__repr__(), config.CFG.__str__(), counter.Counter.__str__(), average.Average.__str__(), dirstructure.Directory.calcStats(), python.rootplot.utilities.Hist.divide(), python.rootplot.utilities.Hist.divide_wilson(), utils.StatisticalTest.get_status(), dirstructure.Directory.print_report(), python.rootplot.utilities.Hist.TGraph(), python.rootplot.utilities.Hist.TH1F(), Vispa.Views.PropertyView.Property.valueChanged(), counter.Counter.write(), and average.Average.write().

looper.Looper.nEvents

Definition at line 79 of file looper.py.

Referenced by looper.Looper.loop().

looper.Looper.nPrint

Definition at line 81 of file looper.py.

looper.Looper.outDir

Definition at line 70 of file looper.py.

Referenced by looper.Looper._build().

looper.Looper.setup

Definition at line 108 of file looper.py.

Referenced by looper.Looper.loop(), looper.Looper.process(), and looper.Looper.write().

looper.Looper.start_time

Definition at line 162 of file looper.py.

Referenced by progressbar.ProgressBar.__next__(), and progressbar.ProgressBar.update().

looper.Looper.start_time_event

Definition at line 163 of file looper.py.

looper.Looper.timeReport

Definition at line 82 of file looper.py.

Referenced by looper.Looper.write().