CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes
MatrixReader.MatrixReader Class Reference
Inheritance diagram for MatrixReader.MatrixReader:

Public Member Functions

def __init__
 
def createWorkFlows
 
def makeCmd
 
def makeStep
 
def prepare
 
def readMatrix
 
def reset
 
def show
 
def showRaw
 
def showWorkFlows
 
def updateDB
 
def workFlowsByLocation
 

Public Attributes

 addCommand
 
 apply
 
 commandLineWf
 
 files
 
 filesDefault
 
 filesPrefMap
 
 nameList
 
 noRun
 
 overWrite
 
 relvalModule
 
 revertDqmio
 maybe we want too level deep input More...
 
 what
 
 wm
 
 workFlows
 
 workFlowSteps
 

Detailed Description

Definition at line 16 of file MatrixReader.py.

Constructor & Destructor Documentation

def MatrixReader.MatrixReader.__init__ (   self,
  opt 
)

Definition at line 18 of file MatrixReader.py.

References PuppiAlgo.reset(), TPedResult.reset(), BinningPointByMap.reset(), ora::NamedReference.reset(), AlcaBeamSpotManager.reset(), ora::IRelationalOperation.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), ora::Handle< T >.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), ora::IteratorBuffer.reset(), EcalCondHeader.reset(), ora::ContainerIterator.reset(), IMACalibBlock.reset(), ora::OId.reset(), SamplingAnalysis.reset(), L3CalibBlock.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), DaqScopeModeAnalysis.reset(), ora::MultiRecordInsertOperation.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), TB06TreeH2.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), L1MuDTPhiLut.reset(), L1MuDTPtaLut.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), PedsOnlyAnalysis.reset(), TB06Reco.reset(), L1MuDTExtLut.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), coral_bridge::AuthenticationCredentialSet.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), ApvTimingAnalysis.reset(), CalibrationAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), FastFedCablingAnalysis.reset(), PedsFullNoiseAnalysis.reset(), ora::UniqueRef< T >.reset(), cond::persistency::GTProxy.reset(), cond::persistency::IOVProxy.reset(), CommissioningAnalysis.reset(), ora::InsertOperation.reset(), ora::Ptr< T >.reset(), TB06RecoH2.reset(), ora::BulkInsertOperation.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TrackerMap.reset(), ora::UpdateOperation.reset(), ora::DeleteOperation.reset(), DTTFBitArray< N >.reset(), data_sources.json_list.reset(), and ora::QueryableVector< Tp >.reset().

18 
19  def __init__(self, opt):
20 
21  self.reset(opt.what)
22 
23  self.wm=opt.wmcontrol
24  self.revertDqmio=opt.revertDqmio
25  self.addCommand=opt.command
26  self.apply=opt.apply
27  self.commandLineWf=opt.workflow
28  self.overWrite=opt.overWrite
29 
30  self.noRun = opt.noRun
31  return
MatrixReader.MatrixReader.revertDqmio
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23

Member Function Documentation

def MatrixReader.MatrixReader.createWorkFlows (   self,
  fileNameIn 
)

Definition at line 450 of file MatrixReader.py.

References MatrixReader.MatrixReader.filesPrefMap, MatrixReader.MatrixReader.nameList, and MatrixReader.MatrixReader.workFlowSteps.

Referenced by MatrixReader.MatrixReader.prepare().

450 
451  def createWorkFlows(self, fileNameIn):
452 
453  prefixIn = self.filesPrefMap[fileNameIn]
454 
455  # get through the list of items and update the requested workflows only
456  keyList = self.workFlowSteps.keys()
457  ids = []
458  for item in keyList:
459  id, pref = item
460  if pref != prefixIn : continue
461  ids.append(id)
462  ids.sort()
463  for key in ids:
464  val = self.workFlowSteps[(key,prefixIn)]
465  num, name, commands, stepList = val
466  nameId = str(num)+'_'+name
467  if nameId in self.nameList:
468  print "==> duplicate name found for ", nameId
469  print ' keeping : ', self.nameList[nameId]
470  print ' ignoring : ', val
471  else:
472  self.nameList[nameId] = val
473 
474  self.workFlows.append(WorkFlow(num, name, commands=commands))
475 
476  return
def MatrixReader.MatrixReader.makeCmd (   self,
  step 
)

Definition at line 90 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

90 
91  def makeCmd(self, step):
92 
93  cmd = ''
94  cfg = None
95  input = None
96  for k,v in step.items():
97  if 'no_exec' in k : continue # we want to really run it ...
98  if k.lower() == 'cfg':
99  cfg = v
100  continue # do not append to cmd, return separately
101  if k.lower() == 'input':
102  input = v
103  continue # do not append to cmd, return separately
104 
105  #chain the configs
106  #if k.lower() == '--python':
107  # v = 'step%d_%s'%(index,v)
108  cmd += ' ' + k + ' ' + str(v)
109  return cfg, input, cmd
def MatrixReader.MatrixReader.makeStep (   self,
  step,
  overrides 
)

Definition at line 110 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

110 
111  def makeStep(self,step,overrides):
112  from Configuration.PyReleaseValidation.relval_steps import merge
113  if len(overrides.keys()) > 0:
114  copyStep=merge([overrides]+[step])
115  return copyStep
116  else:
117  return step
merge
Definition: merge.py:1
relval_steps
def MatrixReader.MatrixReader.prepare (   self,
  useInput = None,
  refRel = '',
  fromScratch = None 
)

Definition at line 477 of file MatrixReader.py.

References MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.files, geometryComparison.GeometryComparison.files, MatrixReader.MatrixReader.filesDefault, MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.what.

477 
478  def prepare(self, useInput=None, refRel='', fromScratch=None):
479 
480  for matrixFile in self.files:
481  if self.what != 'all' and self.what not in matrixFile:
482  print "ignoring non-requested file",matrixFile
483  continue
484  if self.what == 'all' and not self.filesDefault[matrixFile]:
485  print "ignoring",matrixFile,"from default matrix"
486  continue
487 
488  try:
489  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
490  except Exception as e:
491  print "ERROR reading file:", matrixFile, str(e)
492  raise
493 
494  try:
495  self.createWorkFlows(matrixFile)
496  except Exception as e:
497  print "ERROR creating workflows :", str(e)
498  raise
499 
def MatrixReader.MatrixReader.readMatrix (   self,
  fileNameIn,
  useInput = None,
  refRel = None,
  fromScratch = None 
)

Definition at line 118 of file MatrixReader.py.

References MatrixReader.MatrixReader.addCommand, MatrixReader.MatrixReader.apply, MatrixReader.MatrixReader.filesPrefMap, join(), list(), MatrixReader.MatrixReader.makeCmd(), MatrixReader.MatrixReader.makeStep(), MatrixReader.MatrixReader.noRun, MatrixReader.MatrixReader.overWrite, MatrixReader.MatrixReader.relvalModule, MatrixReader.MatrixReader.revertDqmio, MatrixReader.MatrixReader.wm, and MatrixReader.MatrixReader.workFlowSteps.

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

118 
119  def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None):
120 
121  prefix = self.filesPrefMap[fileNameIn]
122 
123  print "processing", fileNameIn
124 
125  try:
126  _tmpMod = __import__( 'Configuration.PyReleaseValidation.'+fileNameIn )
127  self.relvalModule = sys.modules['Configuration.PyReleaseValidation.'+fileNameIn]
128  except Exception as e:
129  print "ERROR importing file ", fileNameIn, str(e)
130  return
131 
132  if useInput is not None:
133  print "request for INPUT for ", useInput
134 
135 
136  fromInput={}
137 
138  if useInput:
139  for i in useInput:
140  if ':' in i:
141  (ik,il)=i.split(':')
142  if ik=='all':
143  for k in self.relvalModule.workflows.keys():
144  fromInput[float(k)]=int(il)
145  else:
146  fromInput[float(ik)]=int(il)
147  else:
148  if i=='all':
149  for k in self.relvalModule.workflows.keys():
150  fromInput[float(k)]=0
151  else:
152  fromInput[float(i)]=0
153 
154  if fromScratch:
155  fromScratch=map(float,fromScratch)
156  for num in fromScratch:
157  if num in fromInput:
158  fromInput.pop(num)
159  #overwrite steps
160  if self.overWrite:
161  for p in self.overWrite:
162  self.relvalModule.steps.overwrite(p)
163 
164  #change the origin of dataset on the fly
165  if refRel:
166  if ',' in refRel:
167  refRels=refRel.split(',')
168  if len(refRels)!=len(self.relvalModule.baseDataSetRelease):
169  return
170  self.relvalModule.changeRefRelease(
171  self.relvalModule.steps,
172  list(zip(self.relvalModule.baseDataSetRelease,refRels))
173  )
174  else:
175  self.relvalModule.changeRefRelease(
176  self.relvalModule.steps,
177  [(x,refRel) for x in self.relvalModule.baseDataSetRelease]
178  )
179 
180 
181  for num, wfInfo in self.relvalModule.workflows.items():
182  commands=[]
183  wfName = wfInfo[0]
184  stepList = wfInfo[1]
185  stepOverrides=wfInfo.overrides
186  # if no explicit name given for the workflow, use the name of step1
187  if wfName.strip() == '': wfName = stepList[0]
188  # option to specialize the wf as the third item in the WF list
189  addTo=None
190  addCom=None
191  if len(wfInfo)>=3:
192  addCom=wfInfo[2]
193  if not type(addCom)==list: addCom=[addCom]
194  #print 'added dict',addCom
195  if len(wfInfo)>=4:
196  addTo=wfInfo[3]
197  #pad with 0
198  while len(addTo)!=len(stepList):
199  addTo.append(0)
200 
201  name=wfName
202  stepIndex=0
203  ranStepList=[]
204 
205  #first resolve INPUT possibilities
206  if num in fromInput:
207  ilevel=fromInput[num]
208  #print num,ilevel
209  for (stepIr,step) in enumerate(reversed(stepList)):
210  stepName=step
211  stepI=(len(stepList)-stepIr)-1
212  #print stepIr,step,stepI,ilevel
213  if stepI>ilevel:
214  #print "ignoring"
215  continue
216  if stepI!=0:
217  testName='__'.join(stepList[0:stepI+1])+'INPUT'
218  else:
219  testName=step+'INPUT'
220  #print "JR",stepI,stepIr,testName,stepList
221  if testName in self.relvalModule.steps.keys():
222  #print "JR",stepI,stepIr
223  stepList[stepI]=testName
224  #pop the rest in the list
225  #print "\tmod prepop",stepList
226  for p in range(stepI):
227  stepList.pop(0)
228  #print "\t\tmod",stepList
229  break
230 
231 
232  for (stepI,step) in enumerate(stepList):
233  stepName=step
234  if self.wm:
235  #cannot put a certain number of things in wm
236  if stepName in [
237  #'HARVEST','HARVESTD','HARVESTDreHLT',
238  'RECODFROMRAWRECO','SKIMD','SKIMCOSD','SKIMDreHLT'
239  ]:
240  continue
241 
242  #replace stepName is needed
243  #if stepName in self.replaceStep
244  if len(name) > 0 : name += '+'
245  #any step can be mirrored with INPUT
246  ## maybe we want too level deep input
247  """
248  if num in fromInput:
249  if step+'INPUT' in self.relvalModule.steps.keys():
250  stepName = step+"INPUT"
251  stepList.remove(step)
252  stepList.insert(stepIndex,stepName)
253  """
254  name += stepName
255  if addCom and (not addTo or addTo[stepIndex]==1):
256  from Configuration.PyReleaseValidation.relval_steps import merge
257  copyStep=merge(addCom+[self.makeStep(self.relvalModule.steps[stepName],stepOverrides)])
258  cfg, input, opts = self.makeCmd(copyStep)
259  else:
260  cfg, input, opts = self.makeCmd(self.makeStep(self.relvalModule.steps[stepName],stepOverrides))
261 
262  if input and cfg :
263  msg = "FATAL ERROR: found both cfg and input for workflow "+str(num)+' step '+stepName
264  raise MatrixException(msg)
265 
266  if input:
267  cmd = input
268  if self.noRun:
269  cmd.run=[]
270  else:
271  if cfg:
272  cmd = 'cmsDriver.py '+cfg+' '+opts
273  else:
274  cmd = 'cmsDriver.py step'+str(stepIndex+1)+' '+opts
275  if self.wm:
276  cmd+=' --io %s.io --python %s.py'%(stepName,stepName)
277  if self.addCommand:
278  if self.apply:
279  if stepIndex in self.apply or stepName in self.apply:
280  cmd +=' '+self.addCommand
281  else:
282  cmd +=' '+self.addCommand
283  if self.wm and self.revertDqmio=='yes':
284  cmd=cmd.replace('DQMIO','DQM')
285  cmd=cmd.replace('--filetype DQM','')
286  commands.append(cmd)
287  ranStepList.append(stepName)
288  stepIndex+=1
289 
290  self.workFlowSteps[(num,prefix)] = (num, name, commands, ranStepList)
291 
292  return
293 
merge
Definition: merge.py:1
MatrixReader.MatrixReader.revertDqmio
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
relval_steps
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
list
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*","!HLTx*"if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL.It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of"!*"before the partial wildcard feature was incorporated).Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run
def MatrixReader.MatrixReader.reset (   self,
  what = 'all' 
)

Definition at line 32 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.__init__(), and MatrixReader.MatrixReader.showRaw().

32 
33  def reset(self, what='all'):
34 
35  self.what = what
36 
37  #a bunch of information, but not yet the WorkFlow object
38  self.workFlowSteps = {}
39  #the actual WorkFlow objects
40  self.workFlows = []
41  self.nameList = {}
42 
43  self.filesPrefMap = {'relval_standard' : 'std-' ,
44  'relval_highstats': 'hi-' ,
45  'relval_pileup': 'PU-' ,
46  'relval_generator': 'gen-',
47  'relval_extendedgen': 'genExt-',
48  'relval_production': 'prod-' ,
49  'relval_ged': 'ged-',
50  'relval_upgrade':'upg-',
51  'relval_2017':'2017-',
52  'relval_identity':'id-',
53  'relval_machine': 'mach-',
54  'relval_unsch': 'unsch-',
55  'relval_premix': 'premix-'
56  }
57 
58  self.files = ['relval_standard' ,
59  'relval_highstats',
60  'relval_pileup',
61  'relval_generator',
62  'relval_extendedgen',
63  'relval_production',
64  'relval_ged',
65  'relval_upgrade',
66  'relval_2017',
67  'relval_identity',
68  'relval_machine',
69  'relval_unsch',
70  'relval_premix'
71  ]
72  self.filesDefault = {'relval_standard':True ,
73  'relval_highstats':True ,
74  'relval_pileup':True,
75  'relval_generator':True,
76  'relval_extendedgen':True,
77  'relval_production':True,
78  'relval_ged':True,
79  'relval_upgrade':False,
80  'relval_2017':True,
81  'relval_identity':False,
82  'relval_machine':True,
83  'relval_unsch':True,
84  'relval_premix':True
85  }
86 
87  self.relvalModule = None
88 
89  return
def MatrixReader.MatrixReader.show (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 500 of file MatrixReader.py.

References MatrixReader.MatrixReader.showWorkFlows().

Referenced by Vispa.Main.AboutDialog.AboutDialog.onScreen(), Vispa.Gui.BoxContentDialog.BoxContentDialog.onScreen(), and Vispa.Gui.FindDialog.FindDialog.onScreen().

500 
501  def show(self, selected=None, extended=True, cafVeto=True):
502 
503  self.showWorkFlows(selected, extended, cafVeto)
504  print '\n','-'*80,'\n'
505 
def MatrixReader.MatrixReader.showRaw (   self,
  useInput,
  refRel = None,
  fromScratch = None,
  what = 'all',
  step1Only = False,
  selected = None 
)

Definition at line 294 of file MatrixReader.py.

References Clusterizer1DCommons.add(), MatrixReader.MatrixReader.files, geometryComparison.GeometryComparison.files, MatrixReader.MatrixReader.filesDefault, join(), MatrixReader.MatrixReader.readMatrix(), PuppiAlgo.reset(), BinningPointByMap.reset(), TPedResult.reset(), ora::NamedReference.reset(), AlcaBeamSpotManager.reset(), ora::IRelationalOperation.reset(), ApvLatencyAnalysis.reset(), VEcalCalibBlock.reset(), ora::Handle< T >.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), ora::IteratorBuffer.reset(), EcalCondHeader.reset(), ora::ContainerIterator.reset(), IMACalibBlock.reset(), ora::OId.reset(), SamplingAnalysis.reset(), L3CalibBlock.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), DaqScopeModeAnalysis.reset(), ora::MultiRecordInsertOperation.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), TB06TreeH2.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), L1MuDTPhiLut.reset(), L1MuDTPtaLut.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), PedsOnlyAnalysis.reset(), TB06Reco.reset(), L1MuDTExtLut.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), coral_bridge::AuthenticationCredentialSet.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), ApvTimingAnalysis.reset(), CalibrationAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), FastFedCablingAnalysis.reset(), PedsFullNoiseAnalysis.reset(), ora::UniqueRef< T >.reset(), cond::persistency::GTProxy.reset(), cond::persistency::IOVProxy.reset(), CommissioningAnalysis.reset(), ora::InsertOperation.reset(), ora::Ptr< T >.reset(), TB06RecoH2.reset(), ora::BulkInsertOperation.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TrackerMap.reset(), ora::UpdateOperation.reset(), ora::DeleteOperation.reset(), DTTFBitArray< N >.reset(), data_sources.json_list.reset(), ora::QueryableVector< Tp >.reset(), MatrixReader.MatrixReader.revertDqmio, MatrixReader.MatrixReader.what, and MatrixReader.MatrixReader.workFlowSteps.

294 
295  def showRaw(self, useInput, refRel=None, fromScratch=None, what='all',step1Only=False,selected=None):
296 
297  if selected:
298  selected=map(float,selected)
299  for matrixFile in self.files:
300 
301  self.reset(what)
302 
303  if self.what != 'all' and self.what not in matrixFile:
304  print "ignoring non-requested file",matrixFile
305  continue
306 
307  if self.what == 'all' and not self.filesDefault[matrixFile]:
308  print "ignoring file not used by default (enable with -w)",matrixFile
309  continue
310 
311  try:
312  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
313  except Exception as e:
314  print "ERROR reading file:", matrixFile, str(e)
315  raise
316 
317  if not self.workFlowSteps: continue
318 
319  dataFileName = matrixFile.replace('relval_', 'cmsDriver_')+'_hlt.txt'
320  outFile = open(dataFileName,'w')
321 
322  print "found ", len(self.workFlowSteps.keys()), ' workflows for ', dataFileName
323  ids = self.workFlowSteps.keys()
324  ids.sort()
325  indexAndSteps=[]
326 
327  writtenWF=0
328  for key in ids:
329  if selected and not (key[0] in selected):
330  continue
331  #trick to skip the HImix IB test
332  if key[0]==203.1 or key[0]==204.1 or key[0]==205.1 or key[0]==4.51 or key[0]==4.52: continue
333  num, name, commands, stepList = self.workFlowSteps[key]
334 
335  wfName,stepNames= name.split('+',1)
336 
337  stepNames=stepNames.replace('+RECODFROMRAWRECO','')
338  stepNames=stepNames.replace('+SKIMCOSD','')
339  stepNames=stepNames.replace('+SKIMD','')
340  if 'HARVEST' in stepNames:
341  #find out automatically what to remove
342  exactb=stepNames.index('+HARVEST')
343  exacte=stepNames.index('+',exactb+1) if ('+' in stepNames[exactb+1:]) else (len(stepNames))
344  stepNames=stepNames.replace(stepNames[exactb:exacte],'')
345  otherSteps = None
346  if '+' in stepNames:
347  step1,otherSteps = stepNames.split('+',1)
348 
349  line = str(num) + ' ++ '+ wfName
350  if otherSteps and not step1Only:
351  line += ' ++ ' +otherSteps.replace('+',',')
352  else:
353  line += ' ++ none'
354  inputInfo=None
355  if not isinstance(commands[0],str):
356  inputInfo=commands[0]
357  if otherSteps:
358  for (i,c) in enumerate(otherSteps.split('+')):
359  #pad with set
360  for p in range(len(indexAndSteps),i+2):
361  indexAndSteps.append(set())
362  indexAndSteps[i+1].add((c,commands[i+1]))
363 
364  if inputInfo :
365  #skip the samples from INPUT when step1Only is on
366  if step1Only: continue
367  line += ' ++ REALDATA: '+inputInfo.dataSet
368  if inputInfo.run!=[]: line += ', RUN:'+'|'.join(map(str,inputInfo.run))
369  line += ', FILES: ' +str(inputInfo.files)
370  line += ', EVENTS: '+str(inputInfo.events)
371  if inputInfo.label!='':
372  line += ', LABEL: ' +inputInfo.label
373  line += ', LOCATION:'+inputInfo.location
374  line += ' @@@'
375  else:
376  line += ' @@@ '+commands[0]
377  if self.revertDqmio=='yes':
378  line=line.replace('DQMIO','DQM')
379  writtenWF+=1
380  outFile.write(line+'\n')
381 
382 
383  outFile.write('\n'+'\n')
384  if step1Only: continue
385 
386  for (index,s) in enumerate(indexAndSteps):
387  for (stepName,cmd) in s:
388  stepIndex=index+1
389  if 'dasquery.log' in cmd: continue
390  line = 'STEP%d ++ '%(stepIndex,) +stepName + ' @@@ '+cmd
391  if self.revertDqmio=='yes':
392  line=line.replace('DQMIO','DQM')
393  outFile.write(line+'\n')
394  outFile.write('\n'+'\n')
395  outFile.close()
396  print "wrote ",writtenWF, ' workflow'+('s' if (writtenWF!=1) else ''),' to ', outFile.name
397  return
MatrixReader.MatrixReader.revertDqmio
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
Clusterizer1DCommons::add
void add(const std::vector< const T * > &source, std::vector< const T * > &dest)
Definition: Clusterizer1DCommons.h:26
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
def MatrixReader.MatrixReader.showWorkFlows (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 413 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlowsByLocation().

Referenced by MatrixReader.MatrixReader.show().

413 
414  def showWorkFlows(self, selected=None, extended=True, cafVeto=True):
415  if selected: selected = map(float,selected)
416  wfs = self.workFlowsByLocation(cafVeto)
417  maxLen = 100 # for summary, limit width of output
418  fmt1 = "%-6s %-35s [1]: %s ..."
419  fmt2 = " %35s [%d]: %s ..."
420  print "\nfound a total of ", len(wfs), ' workflows:'
421  if selected:
422  print " of which the following", len(selected), 'were selected:'
423  #-ap for now:
424  maxLen = -1 # for individual listing, no limit on width
425  fmt1 = "%-6s %-35s [1]: %s "
426  fmt2 = " %35s [%d]: %s"
427 
428  N=[]
429  for wf in wfs:
430  if selected and float(wf.numId) not in selected: continue
431  if extended: print ''
432  #pad with zeros
433  for i in range(len(N),len(wf.cmds)): N.append(0)
434  N[len(wf.cmds)-1]+=1
435  wfName, stepNames = wf.nameId.split('+',1)
436  for i,s in enumerate(wf.cmds):
437  if extended:
438  if i==0:
439  print fmt1 % (wf.numId, stepNames, (str(s)+' ')[:maxLen])
440  else:
441  print fmt2 % ( ' ', i+1, (str(s)+' ')[:maxLen])
442  else:
443  print "%-6s %-35s "% (wf.numId, stepNames)
444  break
445  print ''
446  for i,n in enumerate(N):
447  if n: print n,'workflows with',i+1,'steps'
448 
449  return
def MatrixReader.MatrixReader.updateDB (   self)

Definition at line 506 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

506 
507  def updateDB(self):
508 
509  import pickle
510  pickle.dump(self.workFlows, open('theMatrix.pkl', 'w') )
511 
512  return
513 
def MatrixReader.MatrixReader.workFlowsByLocation (   self,
  cafVeto = True 
)

Definition at line 398 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

Referenced by MatrixReader.MatrixReader.showWorkFlows().

398 
399  def workFlowsByLocation(self, cafVeto=True):
400  # Check if we are on CAF
401  onCAF = False
402  if 'cms/caf/cms' in os.environ['CMS_PATH']:
403  onCAF = True
404 
405  workflows = []
406  for workflow in self.workFlows:
407  if isinstance(workflow.cmds[0], InputInfo):
408  if cafVeto and (workflow.cmds[0].location == 'CAF' and not onCAF):
409  continue
410  workflows.append(workflow)
411 
412  return workflows

Member Data Documentation

MatrixReader.MatrixReader.addCommand

Definition at line 24 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.apply

Definition at line 25 of file MatrixReader.py.

Referenced by heavyIonTools.ConfigureHeavyIons.__call__(), editorTools.UserCodeTool.__call__(), HiCoreTools.RestrictInputToAOD.__call__(), coreTools.RunOnData.__call__(), trackTools.MakeAODTrackCandidates.__call__(), runJetUncertainties.RunJetUncertainties.__call__(), metTools.AddMETCollection.__call__(), heavyIonTools.ProductionDefaults.__call__(), editorTools.ChangeSource.__call__(), HiCoreTools.RemoveMCMatching.__call__(), cmsswVersionTools.PickRelValInputFiles.__call__(), coreTools.RemoveMCMatching.__call__(), trackTools.MakePATTrackCandidates.__call__(), trigTools.SwitchOnTrigger.__call__(), runMETCorrectionsAndUncertainties.RunMETCorrectionsAndUncertainties.__call__(), heavyIonTools.SelectionDefaults.__call__(), HiCoreTools.RemoveAllPATObjectsBut.__call__(), heavyIonTools.DisbaleMonteCarloDeps.__call__(), HiCoreTools.RemoveSpecificPATObjects.__call__(), trigTools.SwitchOnTriggerStandAlone.__call__(), trackTools.MakeTrackCandidates.__call__(), tauTools.AddTauCollection.__call__(), trigTools.SwitchOnTriggerMatching.__call__(), HiCoreTools.RemoveCleaning.__call__(), HiCoreTools.AddCleaning.__call__(), jetTools.AddJetCollection.__call__(), trigTools.SwitchOnTriggerMatchingStandAlone.__call__(), trigTools.SwitchOnTriggerMatchEmbedding.__call__(), jetTools.SwitchJetCollection.__call__(), jetTools.UpdateJetCollection.__call__(), jetTools.AddJetID.__call__(), jetTools.SetTagInfos.__call__(), and MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.commandLineWf

Definition at line 26 of file MatrixReader.py.

MatrixReader.MatrixReader.files

Definition at line 57 of file MatrixReader.py.

Referenced by chain.Chain._guessTreeName(), dataset.BaseDataset.listOfFiles(), dataset.BaseDataset.listOfGoodFiles(), MatrixReader.MatrixReader.prepare(), dataset.BaseDataset.printFiles(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.filesDefault

Definition at line 71 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.filesPrefMap

Definition at line 42 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows(), and MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.nameList

Definition at line 40 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows().

MatrixReader.MatrixReader.noRun

Definition at line 29 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.overWrite

Definition at line 27 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.relvalModule

Definition at line 86 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.revertDqmio

maybe we want too level deep input

Definition at line 23 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.what

Definition at line 34 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.wm

Definition at line 22 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.workFlows

Definition at line 39 of file MatrixReader.py.

Referenced by MatrixRunner.MatrixRunner.runTests(), MatrixReader.MatrixReader.updateDB(), and MatrixReader.MatrixReader.workFlowsByLocation().

MatrixReader.MatrixReader.workFlowSteps

Definition at line 37 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

Generated for CMSSW Reference Manual by doxygen 1.8.5