CMS 3D CMS Logo

List of all members | Public Member Functions | Public Attributes
MatrixReader.MatrixReader Class Reference
Inheritance diagram for MatrixReader.MatrixReader:

Public Member Functions

def __init__ (self, opt)
 
def createWorkFlows (self, fileNameIn)
 
def makeCmd (self, step)
 
def makeStep (self, step, overrides)
 
def prepare (self, useInput=None, refRel='', fromScratch=None)
 
def readMatrix (self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
 
def reset (self, what='all')
 
def show (self, selected=None, extended=True, cafVeto=True)
 
def showRaw (self, useInput, refRel=None, fromScratch=None, what='all', step1Only=False, selected=None)
 
def showWorkFlows (self, selected=None, extended=True, cafVeto=True)
 
def updateDB (self)
 
def workFlowsByLocation (self, cafVeto=True)
 

Public Attributes

 addCommand
 
 apply
 
 commandLineWf
 
 files
 
 filesDefault
 
 filesPrefMap
 
 nameList
 
 noRun
 
 overWrite
 
 relvalModule
 
 revertDqmio
 maybe we want too level deep input More...
 
 what
 
 wm
 
 workFlows
 
 workFlowSteps
 

Detailed Description

Definition at line 16 of file MatrixReader.py.

Constructor & Destructor Documentation

def MatrixReader.MatrixReader.__init__ (   self,
  opt 
)

Definition at line 18 of file MatrixReader.py.

References PuppiAlgo.reset(), BinningPointByMap.reset(), TPedResult.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), ApvLatencyAnalysis.reset(), VEcalCalibBlock.reset(), SiPixelStatusManager.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), SiStripMiscalibrate::Entry.reset(), EcalCondHeader.reset(), IMACalibBlock.reset(), SamplingAnalysis.reset(), L3CalibBlock.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), SiStripPI::Entry.reset(), TB06TreeH2.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), RPCDCCLink.reset(), RPCAMCLink.reset(), L1MuDTPhiLut.reset(), L1MuDTPtaLut.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), PedsOnlyAnalysis.reset(), TB06Reco.reset(), VpspScanAnalysis.reset(), L1MuDTExtLut.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), coral_bridge::AuthenticationCredentialSet.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), cond::persistency::RunInfoProxy.reset(), ApvTimingAnalysis.reset(), DaqScopeModeAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), FastFedCablingAnalysis.reset(), RPCLBLink.reset(), cond::persistency::GTProxy.reset(), AlignableBeamSpot.reset(), cond::persistency::IOVProxy.reset(), CommissioningAnalysis.reset(), PedsFullNoiseAnalysis.reset(), CalibrationAnalysis.reset(), TB06RecoH2.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TrackerMap.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), LHCInfoImpl::LumiSectionFilter.reset(), and mps_create_file_lists._DasCache.reset().

18  def __init__(self, opt):
19 
20  self.reset(opt.what)
21 
22  self.wm=opt.wmcontrol
23  self.revertDqmio=opt.revertDqmio
24  self.addCommand=opt.command
25  self.apply=opt.apply
26  self.commandLineWf=opt.workflow
27  self.overWrite=opt.overWrite
28 
29  self.noRun = opt.noRun
30  return
31 
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
def reset(self, what='all')
Definition: MatrixReader.py:32
def __init__(self, opt)
Definition: MatrixReader.py:18

Member Function Documentation

def MatrixReader.MatrixReader.createWorkFlows (   self,
  fileNameIn 
)

Definition at line 452 of file MatrixReader.py.

References MatrixReader.MatrixReader.filesPrefMap, MatrixReader.MatrixReader.nameList, str, and MatrixReader.MatrixReader.workFlowSteps.

Referenced by MatrixReader.MatrixReader.prepare().

452  def createWorkFlows(self, fileNameIn):
453 
454  prefixIn = self.filesPrefMap[fileNameIn]
455 
456  # get through the list of items and update the requested workflows only
457  keyList = self.workFlowSteps.keys()
458  ids = []
459  for item in keyList:
460  id, pref = item
461  if pref != prefixIn : continue
462  ids.append(id)
463  ids.sort()
464  for key in ids:
465  val = self.workFlowSteps[(key,prefixIn)]
466  num, name, commands, stepList = val
467  nameId = str(num)+'_'+name
468  if nameId in self.nameList:
469  print "==> duplicate name found for ", nameId
470  print ' keeping : ', self.nameList[nameId]
471  print ' ignoring : ', val
472  else:
473  self.nameList[nameId] = val
474 
475  self.workFlows.append(WorkFlow(num, name, commands=commands))
476 
477  return
478 
def createWorkFlows(self, fileNameIn)
#define str(s)
def MatrixReader.MatrixReader.makeCmd (   self,
  step 
)

Definition at line 93 of file MatrixReader.py.

References str.

Referenced by MatrixReader.MatrixReader.readMatrix().

93  def makeCmd(self, step):
94 
95  cmd = ''
96  cfg = None
97  input = None
98  for k,v in step.items():
99  if 'no_exec' in k : continue # we want to really run it ...
100  if k.lower() == 'cfg':
101  cfg = v
102  continue # do not append to cmd, return separately
103  if k.lower() == 'input':
104  input = v
105  continue # do not append to cmd, return separately
106 
107  #chain the configs
108  #if k.lower() == '--python':
109  # v = 'step%d_%s'%(index,v)
110  cmd += ' ' + k + ' ' + str(v)
111  return cfg, input, cmd
112 
def makeCmd(self, step)
Definition: MatrixReader.py:93
#define str(s)
def MatrixReader.MatrixReader.makeStep (   self,
  step,
  overrides 
)

Definition at line 113 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

113  def makeStep(self,step,overrides):
115  if len(overrides) > 0:
116  copyStep=merge([overrides]+[step])
117  return copyStep
118  else:
119  return step
120 
Definition: merge.py:1
def makeStep(self, step, overrides)
def MatrixReader.MatrixReader.prepare (   self,
  useInput = None,
  refRel = '',
  fromScratch = None 
)

Definition at line 479 of file MatrixReader.py.

References MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, MatrixReader.MatrixReader.readMatrix(), str, and MatrixReader.MatrixReader.what.

479  def prepare(self, useInput=None, refRel='', fromScratch=None):
480 
481  for matrixFile in self.files:
482  if self.what != 'all' and self.what not in matrixFile:
483  print "ignoring non-requested file",matrixFile
484  continue
485  if self.what == 'all' and not self.filesDefault[matrixFile]:
486  print "ignoring",matrixFile,"from default matrix"
487  continue
488 
489  try:
490  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
491  except Exception as e:
492  print "ERROR reading file:", matrixFile, str(e)
493  raise
494 
495  try:
496  self.createWorkFlows(matrixFile)
497  except Exception as e:
498  print "ERROR creating workflows :", str(e)
499  raise
500 
501 
def prepare(self, useInput=None, refRel='', fromScratch=None)
def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
def createWorkFlows(self, fileNameIn)
#define str(s)
def MatrixReader.MatrixReader.readMatrix (   self,
  fileNameIn,
  useInput = None,
  refRel = None,
  fromScratch = None 
)

Definition at line 121 of file MatrixReader.py.

References MatrixReader.MatrixReader.addCommand, MatrixReader.MatrixReader.apply, MatrixReader.MatrixReader.filesPrefMap, objects.autophobj.float, createfilelist.int, join(), list(), MatrixReader.MatrixReader.makeCmd(), MatrixReader.MatrixReader.makeStep(), genParticles_cff.map, MatrixReader.MatrixReader.noRun, MatrixReader.MatrixReader.overWrite, MatrixReader.MatrixReader.relvalModule, MatrixReader.MatrixReader.revertDqmio, str, MatrixReader.MatrixReader.wm, MatrixReader.MatrixReader.workFlowSteps, and ComparisonHelper.zip().

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

121  def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None):
122 
123  prefix = self.filesPrefMap[fileNameIn]
124 
125  print "processing", fileNameIn
126 
127  try:
128  _tmpMod = __import__( 'Configuration.PyReleaseValidation.'+fileNameIn )
129  self.relvalModule = sys.modules['Configuration.PyReleaseValidation.'+fileNameIn]
130  except Exception as e:
131  print "ERROR importing file ", fileNameIn, str(e)
132  return
133 
134  if useInput is not None:
135  print "request for INPUT for ", useInput
136 
137 
138  fromInput={}
139 
140  if useInput:
141  for i in useInput:
142  if ':' in i:
143  (ik,il)=i.split(':')
144  if ik=='all':
145  for k in self.relvalModule.workflows.keys():
146  fromInput[float(k)]=int(il)
147  else:
148  fromInput[float(ik)]=int(il)
149  else:
150  if i=='all':
151  for k in self.relvalModule.workflows.keys():
152  fromInput[float(k)]=0
153  else:
154  fromInput[float(i)]=0
155 
156  if fromScratch:
157  fromScratch=map(float,fromScratch)
158  for num in fromScratch:
159  if num in fromInput:
160  fromInput.pop(num)
161  #overwrite steps
162  if self.overWrite:
163  for p in self.overWrite:
164  self.relvalModule.steps.overwrite(p)
165 
166  #change the origin of dataset on the fly
167  if refRel:
168  if ',' in refRel:
169  refRels=refRel.split(',')
170  if len(refRels)!=len(self.relvalModule.baseDataSetRelease):
171  return
172  self.relvalModule.changeRefRelease(
173  self.relvalModule.steps,
174  list(zip(self.relvalModule.baseDataSetRelease,refRels))
175  )
176  else:
177  self.relvalModule.changeRefRelease(
178  self.relvalModule.steps,
179  [(x,refRel) for x in self.relvalModule.baseDataSetRelease]
180  )
181 
182 
183  for num, wfInfo in self.relvalModule.workflows.items():
184  commands=[]
185  wfName = wfInfo[0]
186  stepList = wfInfo[1]
187  stepOverrides=wfInfo.overrides
188  # if no explicit name given for the workflow, use the name of step1
189  if wfName.strip() == '': wfName = stepList[0]
190  # option to specialize the wf as the third item in the WF list
191  addTo=None
192  addCom=None
193  if len(wfInfo)>=3:
194  addCom=wfInfo[2]
195  if not isinstance(addCom, list): addCom=[addCom]
196  #print 'added dict',addCom
197  if len(wfInfo)>=4:
198  addTo=wfInfo[3]
199  #pad with 0
200  while len(addTo)!=len(stepList):
201  addTo.append(0)
202 
203  name=wfName
204  stepIndex=0
205  ranStepList=[]
206 
207  #first resolve INPUT possibilities
208  if num in fromInput:
209  ilevel=fromInput[num]
210  #print num,ilevel
211  for (stepIr,step) in enumerate(reversed(stepList)):
212  stepName=step
213  stepI=(len(stepList)-stepIr)-1
214  #print stepIr,step,stepI,ilevel
215  if stepI>ilevel:
216  #print "ignoring"
217  continue
218  if stepI!=0:
219  testName='__'.join(stepList[0:stepI+1])+'INPUT'
220  else:
221  testName=step+'INPUT'
222  #print "JR",stepI,stepIr,testName,stepList
223  if testName in self.relvalModule.steps.keys():
224  #print "JR",stepI,stepIr
225  stepList[stepI]=testName
226  #pop the rest in the list
227  #print "\tmod prepop",stepList
228  for p in range(stepI):
229  stepList.pop(0)
230  #print "\t\tmod",stepList
231  break
232 
233 
234  for (stepI,step) in enumerate(stepList):
235  stepName=step
236  if self.wm:
237  #cannot put a certain number of things in wm
238  if stepName in [
239  #'HARVEST','HARVESTD','HARVESTDreHLT',
240  'RECODFROMRAWRECO','SKIMD','SKIMCOSD','SKIMDreHLT'
241  ]:
242  continue
243 
244  #replace stepName is needed
245  #if stepName in self.replaceStep
246  if len(name) > 0 : name += '+'
247  #any step can be mirrored with INPUT
248  ## maybe we want too level deep input
249  """
250  if num in fromInput:
251  if step+'INPUT' in self.relvalModule.steps.keys():
252  stepName = step+"INPUT"
253  stepList.remove(step)
254  stepList.insert(stepIndex,stepName)
255  """
256  name += stepName
257  if addCom and (not addTo or addTo[stepIndex]==1):
259  copyStep=merge(addCom+[self.makeStep(self.relvalModule.steps[stepName],stepOverrides)])
260  cfg, input, opts = self.makeCmd(copyStep)
261  else:
262  cfg, input, opts = self.makeCmd(self.makeStep(self.relvalModule.steps[stepName],stepOverrides))
263 
264  if input and cfg :
265  msg = "FATAL ERROR: found both cfg and input for workflow "+str(num)+' step '+stepName
266  raise MatrixException(msg)
267 
268  if input:
269  cmd = input
270  if self.noRun:
271  cmd.run=[]
272  else:
273  if cfg:
274  cmd = 'cmsDriver.py '+cfg+' '+opts
275  else:
276  cmd = 'cmsDriver.py step'+str(stepIndex+1)+' '+opts
277  if self.wm:
278  cmd+=' --io %s.io --python %s.py'%(stepName,stepName)
279  if self.addCommand:
280  if self.apply:
281  if stepIndex in self.apply or stepName in self.apply:
282  cmd +=' '+self.addCommand
283  else:
284  cmd +=' '+self.addCommand
285  if self.wm and self.revertDqmio=='yes':
286  cmd=cmd.replace('DQMIO','DQM')
287  cmd=cmd.replace('--filetype DQM','')
288  commands.append(cmd)
289  ranStepList.append(stepName)
290  stepIndex+=1
291 
292  self.workFlowSteps[(num,prefix)] = (num, name, commands, ranStepList)
293 
294  return
295 
296 
Definition: merge.py:1
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
def makeCmd(self, step)
Definition: MatrixReader.py:93
def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
OutputIterator zip(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, InputIterator2 last2, OutputIterator result, Compare comp)
def makeStep(self, step, overrides)
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
#define str(s)
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*","!HLTx*"if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL.It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of"!*"before the partial wildcard feature was incorporated).Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run
def MatrixReader.MatrixReader.reset (   self,
  what = 'all' 
)

Definition at line 32 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.__init__(), and MatrixReader.MatrixReader.showRaw().

32  def reset(self, what='all'):
33 
34  self.what = what
35 
36  #a bunch of information, but not yet the WorkFlow object
37  self.workFlowSteps = {}
38  #the actual WorkFlow objects
39  self.workFlows = []
40  self.nameList = {}
41 
42  self.filesPrefMap = {'relval_standard' : 'std-' ,
43  'relval_highstats': 'hi-' ,
44  'relval_pileup': 'PU-' ,
45  'relval_generator': 'gen-',
46  'relval_extendedgen': 'genExt-',
47  'relval_production': 'prod-' ,
48  'relval_ged': 'ged-',
49  'relval_upgrade':'upg-',
50  'relval_2017':'2017-',
51  'relval_2023':'2023-',
52  'relval_identity':'id-',
53  'relval_machine': 'mach-',
54  'relval_unsch': 'unsch-',
55  'relval_premix': 'premix-'
56  }
57 
58  self.files = ['relval_standard' ,
59  'relval_highstats',
60  'relval_pileup',
61  'relval_generator',
62  'relval_extendedgen',
63  'relval_production',
64  'relval_ged',
65  'relval_upgrade',
66  'relval_2017',
67  'relval_2023',
68  'relval_identity',
69  'relval_machine',
70  'relval_unsch',
71  'relval_premix'
72  ]
73  self.filesDefault = {'relval_standard':True ,
74  'relval_highstats':True ,
75  'relval_pileup':True,
76  'relval_generator':True,
77  'relval_extendedgen':True,
78  'relval_production':True,
79  'relval_ged':True,
80  'relval_upgrade':False,
81  'relval_2017':True,
82  'relval_2023':True,
83  'relval_identity':False,
84  'relval_machine':True,
85  'relval_unsch':True,
86  'relval_premix':True
87  }
88 
89  self.relvalModule = None
90 
91  return
92 
def reset(self, what='all')
Definition: MatrixReader.py:32
def MatrixReader.MatrixReader.show (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 502 of file MatrixReader.py.

References MatrixReader.MatrixReader.showWorkFlows().

Referenced by Vispa.Main.AboutDialog.AboutDialog.onScreen(), Vispa.Gui.BoxContentDialog.BoxContentDialog.onScreen(), and Vispa.Gui.FindDialog.FindDialog.onScreen().

502  def show(self, selected=None, extended=True, cafVeto=True):
503 
504  self.showWorkFlows(selected, extended, cafVeto)
505  print '\n','-'*80,'\n'
506 
507 
def showWorkFlows(self, selected=None, extended=True, cafVeto=True)
def show(self, selected=None, extended=True, cafVeto=True)
def MatrixReader.MatrixReader.showRaw (   self,
  useInput,
  refRel = None,
  fromScratch = None,
  what = 'all',
  step1Only = False,
  selected = None 
)

Definition at line 297 of file MatrixReader.py.

References PVValHelper.add(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, join(), genParticles_cff.map, MatrixReader.MatrixReader.readMatrix(), PuppiAlgo.reset(), BinningPointByMap.reset(), TPedResult.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), SiPixelStatusManager.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), SiStripMiscalibrate::Entry.reset(), EcalCondHeader.reset(), IMACalibBlock.reset(), SamplingAnalysis.reset(), L3CalibBlock.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), FedTimingAnalysis.reset(), pos::PixelROCStatus.reset(), SiStripPI::Entry.reset(), TB06TreeH2.reset(), L1MuDTTFParameters.reset(), L1MuDTTFMasks.reset(), RPCAMCLink.reset(), RPCDCCLink.reset(), L1MuDTPhiLut.reset(), L1MuDTPtaLut.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), PedsOnlyAnalysis.reset(), TB06Reco.reset(), L1MuDTExtLut.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), coral_bridge::AuthenticationCredentialSet.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), cond::persistency::RunInfoProxy.reset(), ApvTimingAnalysis.reset(), DaqScopeModeAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), FastFedCablingAnalysis.reset(), RPCLBLink.reset(), cond::persistency::GTProxy.reset(), AlignableBeamSpot.reset(), cond::persistency::IOVProxy.reset(), CommissioningAnalysis.reset(), PedsFullNoiseAnalysis.reset(), CalibrationAnalysis.reset(), TB06RecoH2.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TrackerMap.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), LHCInfoImpl::LumiSectionFilter.reset(), mps_create_file_lists._DasCache.reset(), MatrixReader.MatrixReader.revertDqmio, str, MatrixReader.MatrixReader.what, and MatrixReader.MatrixReader.workFlowSteps.

297  def showRaw(self, useInput, refRel=None, fromScratch=None, what='all',step1Only=False,selected=None):
298 
299  if selected:
300  selected=map(float,selected)
301  for matrixFile in self.files:
302 
303  self.reset(what)
304 
305  if self.what != 'all' and self.what not in matrixFile:
306  print "ignoring non-requested file",matrixFile
307  continue
308 
309  if self.what == 'all' and not self.filesDefault[matrixFile]:
310  print "ignoring file not used by default (enable with -w)",matrixFile
311  continue
312 
313  try:
314  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
315  except Exception as e:
316  print "ERROR reading file:", matrixFile, str(e)
317  raise
318 
319  if not self.workFlowSteps: continue
320 
321  dataFileName = matrixFile.replace('relval_', 'cmsDriver_')+'_hlt.txt'
322  outFile = open(dataFileName,'w')
323 
324  print "found ", len(self.workFlowSteps), ' workflows for ', dataFileName
325  ids = sorted(self.workFlowSteps.keys())
326  indexAndSteps=[]
327 
328  writtenWF=0
329  for key in ids:
330  if selected and not (key[0] in selected):
331  continue
332  #trick to skip the HImix IB test
333  if key[0]==203.1 or key[0]==204.1 or key[0]==205.1 or key[0]==4.51 or key[0]==4.52: continue
334  num, name, commands, stepList = self.workFlowSteps[key]
335 
336  wfName,stepNames= name.split('+',1)
337 
338  stepNames=stepNames.replace('+RECODFROMRAWRECO','')
339  stepNames=stepNames.replace('+SKIMCOSD','')
340  stepNames=stepNames.replace('+SKIMD','')
341  if 'HARVEST' in stepNames:
342  #find out automatically what to remove
343  exactb=stepNames.index('+HARVEST')
344  exacte=stepNames.index('+',exactb+1) if ('+' in stepNames[exactb+1:]) else (len(stepNames))
345  stepNames=stepNames.replace(stepNames[exactb:exacte],'')
346  otherSteps = None
347  if '+' in stepNames:
348  step1,otherSteps = stepNames.split('+',1)
349 
350  line = str(num) + ' ++ '+ wfName
351  if otherSteps and not step1Only:
352  line += ' ++ ' +otherSteps.replace('+',',')
353  else:
354  line += ' ++ none'
355  inputInfo=None
356  if not isinstance(commands[0],str):
357  inputInfo=commands[0]
358  if otherSteps:
359  for (i,c) in enumerate(otherSteps.split('+')):
360  #pad with set
361  for p in range(len(indexAndSteps),i+2):
362  indexAndSteps.append(set())
363  indexAndSteps[i+1].add((c,commands[i+1]))
364 
365  if inputInfo :
366  #skip the samples from INPUT when step1Only is on
367  if step1Only: continue
368  line += ' ++ REALDATA: '+inputInfo.dataSet
369  if inputInfo.run!=[]: line += ', RUN:'+'|'.join(map(str,inputInfo.run))
370  line += ', FILES: ' +str(inputInfo.files)
371  line += ', EVENTS: '+str(inputInfo.events)
372  if inputInfo.label!='':
373  line += ', LABEL: ' +inputInfo.label
374  line += ', LOCATION:'+inputInfo.location
375  line += ' @@@'
376  else:
377  line += ' @@@ '+commands[0]
378  if self.revertDqmio=='yes':
379  line=line.replace('DQMIO','DQM')
380  writtenWF+=1
381  outFile.write(line+'\n')
382 
383 
384  outFile.write('\n'+'\n')
385  if step1Only: continue
386 
387  for (index,s) in enumerate(indexAndSteps):
388  for (stepName,cmd) in s:
389  stepIndex=index+1
390  if 'dasquery.log' in cmd: continue
391  line = 'STEP%d ++ '%(stepIndex,) +stepName + ' @@@ '+cmd
392  if self.revertDqmio=='yes':
393  line=line.replace('DQMIO','DQM')
394  outFile.write(line+'\n')
395  outFile.write('\n'+'\n')
396  outFile.close()
397  print "wrote ",writtenWF, ' workflow'+('s' if (writtenWF!=1) else ''),' to ', outFile.name
398  return
399 
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:23
def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
def showRaw(self, useInput, refRel=None, fromScratch=None, what='all', step1Only=False, selected=None)
def reset(self, what='all')
Definition: MatrixReader.py:32
void add(std::map< std::string, TH1 * > &h, TH1 *hist)
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
#define str(s)
def MatrixReader.MatrixReader.showWorkFlows (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 415 of file MatrixReader.py.

References objects.autophobj.float, genParticles_cff.map, str, and MatrixReader.MatrixReader.workFlowsByLocation().

Referenced by MatrixReader.MatrixReader.show().

415  def showWorkFlows(self, selected=None, extended=True, cafVeto=True):
416  if selected: selected = map(float,selected)
417  wfs = self.workFlowsByLocation(cafVeto)
418  maxLen = 100 # for summary, limit width of output
419  fmt1 = "%-6s %-35s [1]: %s ..."
420  fmt2 = " %35s [%d]: %s ..."
421  print "\nfound a total of ", len(wfs), ' workflows:'
422  if selected:
423  print " of which the following", len(selected), 'were selected:'
424  #-ap for now:
425  maxLen = -1 # for individual listing, no limit on width
426  fmt1 = "%-6s %-35s [1]: %s "
427  fmt2 = " %35s [%d]: %s"
428 
429  N=[]
430  for wf in wfs:
431  if selected and float(wf.numId) not in selected: continue
432  if extended: print ''
433  #pad with zeros
434  for i in range(len(N),len(wf.cmds)): N.append(0)
435  N[len(wf.cmds)-1]+=1
436  wfName, stepNames = wf.nameId.split('+',1)
437  for i,s in enumerate(wf.cmds):
438  if extended:
439  if i==0:
440  print fmt1 % (wf.numId, stepNames, (str(s)+' ')[:maxLen])
441  else:
442  print fmt2 % ( ' ', i+1, (str(s)+' ')[:maxLen])
443  else:
444  print "%-6s %-35s "% (wf.numId, stepNames)
445  break
446  print ''
447  for i,n in enumerate(N):
448  if n: print n,'workflows with',i+1,'steps'
449 
450  return
451 
def showWorkFlows(self, selected=None, extended=True, cafVeto=True)
def workFlowsByLocation(self, cafVeto=True)
#define str(s)
def MatrixReader.MatrixReader.updateDB (   self)

Definition at line 508 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

508  def updateDB(self):
509 
510  import pickle
511  pickle.dump(self.workFlows, open('theMatrix.pkl', 'w') )
512 
513  return
514 
def MatrixReader.MatrixReader.workFlowsByLocation (   self,
  cafVeto = True 
)

Definition at line 400 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

Referenced by MatrixReader.MatrixReader.showWorkFlows().

400  def workFlowsByLocation(self, cafVeto=True):
401  # Check if we are on CAF
402  onCAF = False
403  if 'cms/caf/cms' in os.environ['CMS_PATH']:
404  onCAF = True
405 
406  workflows = []
407  for workflow in self.workFlows:
408  if isinstance(workflow.cmds[0], InputInfo):
409  if cafVeto and (workflow.cmds[0].location == 'CAF' and not onCAF):
410  continue
411  workflows.append(workflow)
412 
413  return workflows
414 
def workFlowsByLocation(self, cafVeto=True)

Member Data Documentation

MatrixReader.MatrixReader.addCommand

Definition at line 24 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.apply

Definition at line 25 of file MatrixReader.py.

Referenced by heavyIonTools.ConfigureHeavyIons.__call__(), editorTools.UserCodeTool.__call__(), HiCoreTools.RestrictInputToAOD.__call__(), coreTools.RunOnData.__call__(), trackTools.MakeAODTrackCandidates.__call__(), runJetUncertainties.RunJetUncertainties.__call__(), metTools.AddMETCollection.__call__(), heavyIonTools.ProductionDefaults.__call__(), editorTools.ChangeSource.__call__(), HiCoreTools.RemoveMCMatching.__call__(), cmsswVersionTools.PickRelValInputFiles.__call__(), coreTools.RemoveMCMatching.__call__(), trackTools.MakePATTrackCandidates.__call__(), trigTools.SwitchOnTrigger.__call__(), runMETCorrectionsAndUncertainties.RunMETCorrectionsAndUncertainties.__call__(), heavyIonTools.SelectionDefaults.__call__(), HiCoreTools.RemoveAllPATObjectsBut.__call__(), heavyIonTools.DisbaleMonteCarloDeps.__call__(), HiCoreTools.RemoveSpecificPATObjects.__call__(), trigTools.SwitchOnTriggerStandAlone.__call__(), trackTools.MakeTrackCandidates.__call__(), tauTools.AddTauCollection.__call__(), trigTools.SwitchOnTriggerMatching.__call__(), HiCoreTools.RemoveCleaning.__call__(), HiCoreTools.AddCleaning.__call__(), trigTools.SwitchOnTriggerMatchingStandAlone.__call__(), trigTools.SwitchOnTriggerMatchEmbedding.__call__(), jetTools.AddJetCollection.__call__(), jetTools.SwitchJetCollection.__call__(), jetTools.UpdateJetCollection.__call__(), jetTools.AddJetID.__call__(), jetTools.SetTagInfos.__call__(), and MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.commandLineWf

Definition at line 26 of file MatrixReader.py.

MatrixReader.MatrixReader.filesDefault
MatrixReader.MatrixReader.filesPrefMap
MatrixReader.MatrixReader.nameList

Definition at line 40 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows().

MatrixReader.MatrixReader.noRun

Definition at line 29 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.overWrite

Definition at line 27 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.relvalModule

Definition at line 89 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.revertDqmio

maybe we want too level deep input

Definition at line 23 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.what
MatrixReader.MatrixReader.wm

Definition at line 22 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.workFlows
MatrixReader.MatrixReader.workFlowSteps