CMS 3D CMS Logo

List of all members | Public Member Functions | Public Attributes
MatrixReader.MatrixReader Class Reference
Inheritance diagram for MatrixReader.MatrixReader:

Public Member Functions

def __init__ (self, opt)
 
def createWorkFlows (self, fileNameIn)
 
def makeCmd (self, step)
 
def makeStep (self, step, overrides)
 
def prepare (self, useInput=None, refRel='', fromScratch=None)
 
def readMatrix (self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
 
def reset (self, what='all')
 
def show (self, selected=None, extended=True, cafVeto=True)
 
def showRaw (self, useInput, refRel=None, fromScratch=None, what='all', step1Only=False, selected=None)
 
def showWorkFlows (self, selected=None, extended=True, cafVeto=True)
 
def updateDB (self)
 
def workFlowsByLocation (self, cafVeto=True)
 

Public Attributes

 addCommand
 
 apply
 
 commandLineWf
 
 files
 
 filesDefault
 
 filesPrefMap
 
 nameList
 
 noRun
 
 overWrite
 
 relvalModule
 
 revertDqmio
 maybe we want too level deep input More...
 
 what
 
 wm
 
 workFlows
 
 workFlowSteps
 

Detailed Description

Definition at line 16 of file MatrixReader.py.

Constructor & Destructor Documentation

def MatrixReader.MatrixReader.__init__ (   self,
  opt 
)

Definition at line 18 of file MatrixReader.py.

References PuppiAlgo.reset(), TPedResult.reset(), BinningPointByMap.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), ApvLatencyAnalysis.reset(), VEcalCalibBlock.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), EcalCondHeader.reset(), IMACalibBlock.reset(), SamplingAnalysis.reset(), L3CalibBlock.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), DaqScopeModeAnalysis.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), TB06TreeH2.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), RPCDCCLink.reset(), RPCAMCLink.reset(), L1MuDTPhiLut.reset(), L1MuDTPtaLut.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), PedsOnlyAnalysis.reset(), L1MuDTQualPatternLut.reset(), TB06Reco.reset(), L1MuDTExtLut.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), coral_bridge::AuthenticationCredentialSet.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), cond::persistency::RunInfoProxy.reset(), ApvTimingAnalysis.reset(), CalibrationAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), FastFedCablingAnalysis.reset(), PedsFullNoiseAnalysis.reset(), RPCLBLink.reset(), cond::persistency::GTProxy.reset(), AlignableBeamSpot.reset(), cond::persistency::IOVProxy.reset(), CommissioningAnalysis.reset(), TB06RecoH2.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TrackerMap.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), and mps_create_file_lists._DasCache.reset().

18  def __init__(self, opt):
19 
20  self.reset(opt.what)
21 
22  self.wm=opt.wmcontrol
23  self.revertDqmio=opt.revertDqmio
24  self.addCommand=opt.command
25  self.apply=opt.apply
26  self.commandLineWf=opt.workflow
27  self.overWrite=opt.overWrite
28 
29  self.noRun = opt.noRun
30  return
31 
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
def reset(self, what='all')
Definition: MatrixReader.py:32
def __init__(self, opt)
Definition: MatrixReader.py:18

Member Function Documentation

def MatrixReader.MatrixReader.createWorkFlows (   self,
  fileNameIn 
)

Definition at line 453 of file MatrixReader.py.

References MatrixReader.MatrixReader.filesPrefMap, MatrixReader.MatrixReader.nameList, harvestTrackValidationPlots.str, and MatrixReader.MatrixReader.workFlowSteps.

Referenced by MatrixReader.MatrixReader.prepare().

453  def createWorkFlows(self, fileNameIn):
454 
455  prefixIn = self.filesPrefMap[fileNameIn]
456 
457  # get through the list of items and update the requested workflows only
458  keyList = self.workFlowSteps.keys()
459  ids = []
460  for item in keyList:
461  id, pref = item
462  if pref != prefixIn : continue
463  ids.append(id)
464  ids.sort()
465  for key in ids:
466  val = self.workFlowSteps[(key,prefixIn)]
467  num, name, commands, stepList = val
468  nameId = str(num)+'_'+name
469  if nameId in self.nameList:
470  print "==> duplicate name found for ", nameId
471  print ' keeping : ', self.nameList[nameId]
472  print ' ignoring : ', val
473  else:
474  self.nameList[nameId] = val
475 
476  self.workFlows.append(WorkFlow(num, name, commands=commands))
477 
478  return
479 
def createWorkFlows(self, fileNameIn)
def MatrixReader.MatrixReader.makeCmd (   self,
  step 
)

Definition at line 93 of file MatrixReader.py.

References harvestTrackValidationPlots.str.

Referenced by MatrixReader.MatrixReader.readMatrix().

93  def makeCmd(self, step):
94 
95  cmd = ''
96  cfg = None
97  input = None
98  for k,v in step.items():
99  if 'no_exec' in k : continue # we want to really run it ...
100  if k.lower() == 'cfg':
101  cfg = v
102  continue # do not append to cmd, return separately
103  if k.lower() == 'input':
104  input = v
105  continue # do not append to cmd, return separately
106 
107  #chain the configs
108  #if k.lower() == '--python':
109  # v = 'step%d_%s'%(index,v)
110  cmd += ' ' + k + ' ' + str(v)
111  return cfg, input, cmd
112 
def makeCmd(self, step)
Definition: MatrixReader.py:93
def MatrixReader.MatrixReader.makeStep (   self,
  step,
  overrides 
)

Definition at line 113 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

113  def makeStep(self,step,overrides):
115  if len(overrides.keys()) > 0:
116  copyStep=merge([overrides]+[step])
117  return copyStep
118  else:
119  return step
120 
Definition: merge.py:1
def makeStep(self, step, overrides)
def MatrixReader.MatrixReader.prepare (   self,
  useInput = None,
  refRel = '',
  fromScratch = None 
)

Definition at line 480 of file MatrixReader.py.

References MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, MatrixReader.MatrixReader.readMatrix(), harvestTrackValidationPlots.str, and MatrixReader.MatrixReader.what.

480  def prepare(self, useInput=None, refRel='', fromScratch=None):
481 
482  for matrixFile in self.files:
483  if self.what != 'all' and self.what not in matrixFile:
484  print "ignoring non-requested file",matrixFile
485  continue
486  if self.what == 'all' and not self.filesDefault[matrixFile]:
487  print "ignoring",matrixFile,"from default matrix"
488  continue
489 
490  try:
491  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
492  except Exception as e:
493  print "ERROR reading file:", matrixFile, str(e)
494  raise
495 
496  try:
497  self.createWorkFlows(matrixFile)
498  except Exception as e:
499  print "ERROR creating workflows :", str(e)
500  raise
501 
502 
def prepare(self, useInput=None, refRel='', fromScratch=None)
def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
def createWorkFlows(self, fileNameIn)
def MatrixReader.MatrixReader.readMatrix (   self,
  fileNameIn,
  useInput = None,
  refRel = None,
  fromScratch = None 
)

Definition at line 121 of file MatrixReader.py.

References MatrixReader.MatrixReader.addCommand, MatrixReader.MatrixReader.apply, MatrixReader.MatrixReader.filesPrefMap, objects.autophobj.float, createfilelist.int, join(), list(), MatrixReader.MatrixReader.makeCmd(), MatrixReader.MatrixReader.makeStep(), genParticles_cff.map, MatrixReader.MatrixReader.noRun, MatrixReader.MatrixReader.overWrite, MatrixReader.MatrixReader.relvalModule, MatrixReader.MatrixReader.revertDqmio, harvestTrackValidationPlots.str, MatrixReader.MatrixReader.wm, MatrixReader.MatrixReader.workFlowSteps, and ComparisonHelper.zip().

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

121  def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None):
122 
123  prefix = self.filesPrefMap[fileNameIn]
124 
125  print "processing", fileNameIn
126 
127  try:
128  _tmpMod = __import__( 'Configuration.PyReleaseValidation.'+fileNameIn )
129  self.relvalModule = sys.modules['Configuration.PyReleaseValidation.'+fileNameIn]
130  except Exception as e:
131  print "ERROR importing file ", fileNameIn, str(e)
132  return
133 
134  if useInput is not None:
135  print "request for INPUT for ", useInput
136 
137 
138  fromInput={}
139 
140  if useInput:
141  for i in useInput:
142  if ':' in i:
143  (ik,il)=i.split(':')
144  if ik=='all':
145  for k in self.relvalModule.workflows.keys():
146  fromInput[float(k)]=int(il)
147  else:
148  fromInput[float(ik)]=int(il)
149  else:
150  if i=='all':
151  for k in self.relvalModule.workflows.keys():
152  fromInput[float(k)]=0
153  else:
154  fromInput[float(i)]=0
155 
156  if fromScratch:
157  fromScratch=map(float,fromScratch)
158  for num in fromScratch:
159  if num in fromInput:
160  fromInput.pop(num)
161  #overwrite steps
162  if self.overWrite:
163  for p in self.overWrite:
164  self.relvalModule.steps.overwrite(p)
165 
166  #change the origin of dataset on the fly
167  if refRel:
168  if ',' in refRel:
169  refRels=refRel.split(',')
170  if len(refRels)!=len(self.relvalModule.baseDataSetRelease):
171  return
172  self.relvalModule.changeRefRelease(
173  self.relvalModule.steps,
174  list(zip(self.relvalModule.baseDataSetRelease,refRels))
175  )
176  else:
177  self.relvalModule.changeRefRelease(
178  self.relvalModule.steps,
179  [(x,refRel) for x in self.relvalModule.baseDataSetRelease]
180  )
181 
182 
183  for num, wfInfo in self.relvalModule.workflows.items():
184  commands=[]
185  wfName = wfInfo[0]
186  stepList = wfInfo[1]
187  stepOverrides=wfInfo.overrides
188  # if no explicit name given for the workflow, use the name of step1
189  if wfName.strip() == '': wfName = stepList[0]
190  # option to specialize the wf as the third item in the WF list
191  addTo=None
192  addCom=None
193  if len(wfInfo)>=3:
194  addCom=wfInfo[2]
195  if not type(addCom)==list: addCom=[addCom]
196  #print 'added dict',addCom
197  if len(wfInfo)>=4:
198  addTo=wfInfo[3]
199  #pad with 0
200  while len(addTo)!=len(stepList):
201  addTo.append(0)
202 
203  name=wfName
204  stepIndex=0
205  ranStepList=[]
206 
207  #first resolve INPUT possibilities
208  if num in fromInput:
209  ilevel=fromInput[num]
210  #print num,ilevel
211  for (stepIr,step) in enumerate(reversed(stepList)):
212  stepName=step
213  stepI=(len(stepList)-stepIr)-1
214  #print stepIr,step,stepI,ilevel
215  if stepI>ilevel:
216  #print "ignoring"
217  continue
218  if stepI!=0:
219  testName='__'.join(stepList[0:stepI+1])+'INPUT'
220  else:
221  testName=step+'INPUT'
222  #print "JR",stepI,stepIr,testName,stepList
223  if testName in self.relvalModule.steps.keys():
224  #print "JR",stepI,stepIr
225  stepList[stepI]=testName
226  #pop the rest in the list
227  #print "\tmod prepop",stepList
228  for p in range(stepI):
229  stepList.pop(0)
230  #print "\t\tmod",stepList
231  break
232 
233 
234  for (stepI,step) in enumerate(stepList):
235  stepName=step
236  if self.wm:
237  #cannot put a certain number of things in wm
238  if stepName in [
239  #'HARVEST','HARVESTD','HARVESTDreHLT',
240  'RECODFROMRAWRECO','SKIMD','SKIMCOSD','SKIMDreHLT'
241  ]:
242  continue
243 
244  #replace stepName is needed
245  #if stepName in self.replaceStep
246  if len(name) > 0 : name += '+'
247  #any step can be mirrored with INPUT
248  ## maybe we want too level deep input
249  """
250  if num in fromInput:
251  if step+'INPUT' in self.relvalModule.steps.keys():
252  stepName = step+"INPUT"
253  stepList.remove(step)
254  stepList.insert(stepIndex,stepName)
255  """
256  name += stepName
257  if addCom and (not addTo or addTo[stepIndex]==1):
259  copyStep=merge(addCom+[self.makeStep(self.relvalModule.steps[stepName],stepOverrides)])
260  cfg, input, opts = self.makeCmd(copyStep)
261  else:
262  cfg, input, opts = self.makeCmd(self.makeStep(self.relvalModule.steps[stepName],stepOverrides))
263 
264  if input and cfg :
265  msg = "FATAL ERROR: found both cfg and input for workflow "+str(num)+' step '+stepName
266  raise MatrixException(msg)
267 
268  if input:
269  cmd = input
270  if self.noRun:
271  cmd.run=[]
272  else:
273  if cfg:
274  cmd = 'cmsDriver.py '+cfg+' '+opts
275  else:
276  cmd = 'cmsDriver.py step'+str(stepIndex+1)+' '+opts
277  if self.wm:
278  cmd+=' --io %s.io --python %s.py'%(stepName,stepName)
279  if self.addCommand:
280  if self.apply:
281  if stepIndex in self.apply or stepName in self.apply:
282  cmd +=' '+self.addCommand
283  else:
284  cmd +=' '+self.addCommand
285  if self.wm and self.revertDqmio=='yes':
286  cmd=cmd.replace('DQMIO','DQM')
287  cmd=cmd.replace('--filetype DQM','')
288  commands.append(cmd)
289  ranStepList.append(stepName)
290  stepIndex+=1
291 
292  self.workFlowSteps[(num,prefix)] = (num, name, commands, ranStepList)
293 
294  return
295 
296 
Definition: merge.py:1
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
def makeCmd(self, step)
Definition: MatrixReader.py:93
def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
OutputIterator zip(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, InputIterator2 last2, OutputIterator result, Compare comp)
def makeStep(self, step, overrides)
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*","!HLTx*"if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL.It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of"!*"before the partial wildcard feature was incorporated).Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run
def MatrixReader.MatrixReader.reset (   self,
  what = 'all' 
)

Definition at line 32 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.__init__(), and MatrixReader.MatrixReader.showRaw().

32  def reset(self, what='all'):
33 
34  self.what = what
35 
36  #a bunch of information, but not yet the WorkFlow object
37  self.workFlowSteps = {}
38  #the actual WorkFlow objects
39  self.workFlows = []
40  self.nameList = {}
41 
42  self.filesPrefMap = {'relval_standard' : 'std-' ,
43  'relval_highstats': 'hi-' ,
44  'relval_pileup': 'PU-' ,
45  'relval_generator': 'gen-',
46  'relval_extendedgen': 'genExt-',
47  'relval_production': 'prod-' ,
48  'relval_ged': 'ged-',
49  'relval_upgrade':'upg-',
50  'relval_2017':'2017-',
51  'relval_2023':'2023-',
52  'relval_identity':'id-',
53  'relval_machine': 'mach-',
54  'relval_unsch': 'unsch-',
55  'relval_premix': 'premix-'
56  }
57 
58  self.files = ['relval_standard' ,
59  'relval_highstats',
60  'relval_pileup',
61  'relval_generator',
62  'relval_extendedgen',
63  'relval_production',
64  'relval_ged',
65  'relval_upgrade',
66  'relval_2017',
67  'relval_2023',
68  'relval_identity',
69  'relval_machine',
70  'relval_unsch',
71  'relval_premix'
72  ]
73  self.filesDefault = {'relval_standard':True ,
74  'relval_highstats':True ,
75  'relval_pileup':True,
76  'relval_generator':True,
77  'relval_extendedgen':True,
78  'relval_production':True,
79  'relval_ged':True,
80  'relval_upgrade':False,
81  'relval_2017':True,
82  'relval_2023':True,
83  'relval_identity':False,
84  'relval_machine':True,
85  'relval_unsch':True,
86  'relval_premix':True
87  }
88 
89  self.relvalModule = None
90 
91  return
92 
def reset(self, what='all')
Definition: MatrixReader.py:32
def MatrixReader.MatrixReader.show (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 503 of file MatrixReader.py.

References MatrixReader.MatrixReader.showWorkFlows().

Referenced by Vispa.Main.AboutDialog.AboutDialog.onScreen(), Vispa.Gui.BoxContentDialog.BoxContentDialog.onScreen(), and Vispa.Gui.FindDialog.FindDialog.onScreen().

503  def show(self, selected=None, extended=True, cafVeto=True):
504 
505  self.showWorkFlows(selected, extended, cafVeto)
506  print '\n','-'*80,'\n'
507 
508 
def showWorkFlows(self, selected=None, extended=True, cafVeto=True)
def show(self, selected=None, extended=True, cafVeto=True)
def MatrixReader.MatrixReader.showRaw (   self,
  useInput,
  refRel = None,
  fromScratch = None,
  what = 'all',
  step1Only = False,
  selected = None 
)

Definition at line 297 of file MatrixReader.py.

References Clusterizer1DCommons.add(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, join(), genParticles_cff.map, MatrixReader.MatrixReader.readMatrix(), PuppiAlgo.reset(), BinningPointByMap.reset(), TPedResult.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), EcalCondHeader.reset(), IMACalibBlock.reset(), SamplingAnalysis.reset(), L3CalibBlock.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), DaqScopeModeAnalysis.reset(), FedTimingAnalysis.reset(), pos::PixelROCStatus.reset(), TB06TreeH2.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), RPCAMCLink.reset(), RPCDCCLink.reset(), L1MuDTPhiLut.reset(), L1MuDTPtaLut.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), PedsOnlyAnalysis.reset(), L1MuDTQualPatternLut.reset(), TB06Reco.reset(), VpspScanAnalysis.reset(), L1MuDTExtLut.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), coral_bridge::AuthenticationCredentialSet.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), cond::persistency::RunInfoProxy.reset(), ApvTimingAnalysis.reset(), CalibrationAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), FastFedCablingAnalysis.reset(), PedsFullNoiseAnalysis.reset(), RPCLBLink.reset(), cond::persistency::GTProxy.reset(), AlignableBeamSpot.reset(), cond::persistency::IOVProxy.reset(), CommissioningAnalysis.reset(), TB06RecoH2.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TrackerMap.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), mps_create_file_lists._DasCache.reset(), MatrixReader.MatrixReader.revertDqmio, harvestTrackValidationPlots.str, MatrixReader.MatrixReader.what, and MatrixReader.MatrixReader.workFlowSteps.

297  def showRaw(self, useInput, refRel=None, fromScratch=None, what='all',step1Only=False,selected=None):
298 
299  if selected:
300  selected=map(float,selected)
301  for matrixFile in self.files:
302 
303  self.reset(what)
304 
305  if self.what != 'all' and self.what not in matrixFile:
306  print "ignoring non-requested file",matrixFile
307  continue
308 
309  if self.what == 'all' and not self.filesDefault[matrixFile]:
310  print "ignoring file not used by default (enable with -w)",matrixFile
311  continue
312 
313  try:
314  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
315  except Exception as e:
316  print "ERROR reading file:", matrixFile, str(e)
317  raise
318 
319  if not self.workFlowSteps: continue
320 
321  dataFileName = matrixFile.replace('relval_', 'cmsDriver_')+'_hlt.txt'
322  outFile = open(dataFileName,'w')
323 
324  print "found ", len(self.workFlowSteps.keys()), ' workflows for ', dataFileName
325  ids = self.workFlowSteps.keys()
326  ids.sort()
327  indexAndSteps=[]
328 
329  writtenWF=0
330  for key in ids:
331  if selected and not (key[0] in selected):
332  continue
333  #trick to skip the HImix IB test
334  if key[0]==203.1 or key[0]==204.1 or key[0]==205.1 or key[0]==4.51 or key[0]==4.52: continue
335  num, name, commands, stepList = self.workFlowSteps[key]
336 
337  wfName,stepNames= name.split('+',1)
338 
339  stepNames=stepNames.replace('+RECODFROMRAWRECO','')
340  stepNames=stepNames.replace('+SKIMCOSD','')
341  stepNames=stepNames.replace('+SKIMD','')
342  if 'HARVEST' in stepNames:
343  #find out automatically what to remove
344  exactb=stepNames.index('+HARVEST')
345  exacte=stepNames.index('+',exactb+1) if ('+' in stepNames[exactb+1:]) else (len(stepNames))
346  stepNames=stepNames.replace(stepNames[exactb:exacte],'')
347  otherSteps = None
348  if '+' in stepNames:
349  step1,otherSteps = stepNames.split('+',1)
350 
351  line = str(num) + ' ++ '+ wfName
352  if otherSteps and not step1Only:
353  line += ' ++ ' +otherSteps.replace('+',',')
354  else:
355  line += ' ++ none'
356  inputInfo=None
357  if not isinstance(commands[0],str):
358  inputInfo=commands[0]
359  if otherSteps:
360  for (i,c) in enumerate(otherSteps.split('+')):
361  #pad with set
362  for p in range(len(indexAndSteps),i+2):
363  indexAndSteps.append(set())
364  indexAndSteps[i+1].add((c,commands[i+1]))
365 
366  if inputInfo :
367  #skip the samples from INPUT when step1Only is on
368  if step1Only: continue
369  line += ' ++ REALDATA: '+inputInfo.dataSet
370  if inputInfo.run!=[]: line += ', RUN:'+'|'.join(map(str,inputInfo.run))
371  line += ', FILES: ' +str(inputInfo.files)
372  line += ', EVENTS: '+str(inputInfo.events)
373  if inputInfo.label!='':
374  line += ', LABEL: ' +inputInfo.label
375  line += ', LOCATION:'+inputInfo.location
376  line += ' @@@'
377  else:
378  line += ' @@@ '+commands[0]
379  if self.revertDqmio=='yes':
380  line=line.replace('DQMIO','DQM')
381  writtenWF+=1
382  outFile.write(line+'\n')
383 
384 
385  outFile.write('\n'+'\n')
386  if step1Only: continue
387 
388  for (index,s) in enumerate(indexAndSteps):
389  for (stepName,cmd) in s:
390  stepIndex=index+1
391  if 'dasquery.log' in cmd: continue
392  line = 'STEP%d ++ '%(stepIndex,) +stepName + ' @@@ '+cmd
393  if self.revertDqmio=='yes':
394  line=line.replace('DQMIO','DQM')
395  outFile.write(line+'\n')
396  outFile.write('\n'+'\n')
397  outFile.close()
398  print "wrote ",writtenWF, ' workflow'+('s' if (writtenWF!=1) else ''),' to ', outFile.name
399  return
400 
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
void add(const std::vector< const T * > &source, std::vector< const T * > &dest)
def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
def showRaw(self, useInput, refRel=None, fromScratch=None, what='all', step1Only=False, selected=None)
def reset(self, what='all')
Definition: MatrixReader.py:32
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
def MatrixReader.MatrixReader.showWorkFlows (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 416 of file MatrixReader.py.

References objects.autophobj.float, genParticles_cff.map, harvestTrackValidationPlots.str, and MatrixReader.MatrixReader.workFlowsByLocation().

Referenced by MatrixReader.MatrixReader.show().

416  def showWorkFlows(self, selected=None, extended=True, cafVeto=True):
417  if selected: selected = map(float,selected)
418  wfs = self.workFlowsByLocation(cafVeto)
419  maxLen = 100 # for summary, limit width of output
420  fmt1 = "%-6s %-35s [1]: %s ..."
421  fmt2 = " %35s [%d]: %s ..."
422  print "\nfound a total of ", len(wfs), ' workflows:'
423  if selected:
424  print " of which the following", len(selected), 'were selected:'
425  #-ap for now:
426  maxLen = -1 # for individual listing, no limit on width
427  fmt1 = "%-6s %-35s [1]: %s "
428  fmt2 = " %35s [%d]: %s"
429 
430  N=[]
431  for wf in wfs:
432  if selected and float(wf.numId) not in selected: continue
433  if extended: print ''
434  #pad with zeros
435  for i in range(len(N),len(wf.cmds)): N.append(0)
436  N[len(wf.cmds)-1]+=1
437  wfName, stepNames = wf.nameId.split('+',1)
438  for i,s in enumerate(wf.cmds):
439  if extended:
440  if i==0:
441  print fmt1 % (wf.numId, stepNames, (str(s)+' ')[:maxLen])
442  else:
443  print fmt2 % ( ' ', i+1, (str(s)+' ')[:maxLen])
444  else:
445  print "%-6s %-35s "% (wf.numId, stepNames)
446  break
447  print ''
448  for i,n in enumerate(N):
449  if n: print n,'workflows with',i+1,'steps'
450 
451  return
452 
def showWorkFlows(self, selected=None, extended=True, cafVeto=True)
def workFlowsByLocation(self, cafVeto=True)
def MatrixReader.MatrixReader.updateDB (   self)

Definition at line 509 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

509  def updateDB(self):
510 
511  import pickle
512  pickle.dump(self.workFlows, open('theMatrix.pkl', 'w') )
513 
514  return
515 
def MatrixReader.MatrixReader.workFlowsByLocation (   self,
  cafVeto = True 
)

Definition at line 401 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

Referenced by MatrixReader.MatrixReader.showWorkFlows().

401  def workFlowsByLocation(self, cafVeto=True):
402  # Check if we are on CAF
403  onCAF = False
404  if 'cms/caf/cms' in os.environ['CMS_PATH']:
405  onCAF = True
406 
407  workflows = []
408  for workflow in self.workFlows:
409  if isinstance(workflow.cmds[0], InputInfo):
410  if cafVeto and (workflow.cmds[0].location == 'CAF' and not onCAF):
411  continue
412  workflows.append(workflow)
413 
414  return workflows
415 
def workFlowsByLocation(self, cafVeto=True)

Member Data Documentation

MatrixReader.MatrixReader.addCommand

Definition at line 24 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.apply

Definition at line 25 of file MatrixReader.py.

Referenced by heavyIonTools.ConfigureHeavyIons.__call__(), editorTools.UserCodeTool.__call__(), HiCoreTools.RestrictInputToAOD.__call__(), coreTools.RunOnData.__call__(), trackTools.MakeAODTrackCandidates.__call__(), runJetUncertainties.RunJetUncertainties.__call__(), metTools.AddMETCollection.__call__(), heavyIonTools.ProductionDefaults.__call__(), editorTools.ChangeSource.__call__(), HiCoreTools.RemoveMCMatching.__call__(), cmsswVersionTools.PickRelValInputFiles.__call__(), coreTools.RemoveMCMatching.__call__(), trackTools.MakePATTrackCandidates.__call__(), trigTools.SwitchOnTrigger.__call__(), runMETCorrectionsAndUncertainties.RunMETCorrectionsAndUncertainties.__call__(), heavyIonTools.SelectionDefaults.__call__(), HiCoreTools.RemoveAllPATObjectsBut.__call__(), heavyIonTools.DisbaleMonteCarloDeps.__call__(), HiCoreTools.RemoveSpecificPATObjects.__call__(), trigTools.SwitchOnTriggerStandAlone.__call__(), trackTools.MakeTrackCandidates.__call__(), tauTools.AddTauCollection.__call__(), trigTools.SwitchOnTriggerMatching.__call__(), HiCoreTools.RemoveCleaning.__call__(), HiCoreTools.AddCleaning.__call__(), trigTools.SwitchOnTriggerMatchingStandAlone.__call__(), trigTools.SwitchOnTriggerMatchEmbedding.__call__(), jetTools.AddJetCollection.__call__(), jetTools.SwitchJetCollection.__call__(), jetTools.UpdateJetCollection.__call__(), jetTools.AddJetID.__call__(), jetTools.SetTagInfos.__call__(), and MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.commandLineWf

Definition at line 26 of file MatrixReader.py.

MatrixReader.MatrixReader.filesDefault
MatrixReader.MatrixReader.filesPrefMap
MatrixReader.MatrixReader.nameList

Definition at line 40 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows().

MatrixReader.MatrixReader.noRun

Definition at line 29 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.overWrite

Definition at line 27 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.relvalModule

Definition at line 89 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.revertDqmio

maybe we want too level deep input

Definition at line 23 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.what
MatrixReader.MatrixReader.wm

Definition at line 22 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.workFlows
MatrixReader.MatrixReader.workFlowSteps