CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
List of all members | Public Member Functions | Public Attributes
MatrixReader.MatrixReader Class Reference
Inheritance diagram for MatrixReader.MatrixReader:

Public Member Functions

def __init__
 
def createWorkFlows
 
def makeCmd
 
def makeStep
 
def prepare
 
def readMatrix
 
def reset
 
def show
 
def showRaw
 
def showWorkFlows
 
def updateDB
 
def workFlowsByLocation
 

Public Attributes

 addCommand
 
 apply
 
 commandLineWf
 
 files
 
 filesDefault
 
 filesPrefMap
 
 nameList
 
 noRun
 
 overWrite
 
 relvalModule
 
 revertDqmio
 maybe we want too level deep input More...
 
 what
 
 wm
 
 workFlows
 
 workFlowSteps
 

Detailed Description

Definition at line 17 of file MatrixReader.py.

Constructor & Destructor Documentation

def MatrixReader.MatrixReader.__init__ (   self,
  opt 
)

Definition at line 19 of file MatrixReader.py.

References PuppiAlgo.reset(), TPedResult.reset(), BinningPointByMap.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), EcalCondHeader.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), SiPixelStatusManager.reset(), SiStripMiscalibrate::Entry.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), IMACalibBlock.reset(), SamplingAnalysis.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), L3CalibBlock.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), L1MuDTTFParameters.reset(), L1MuDTTFMasks.reset(), SiStripPI::Entry.reset(), L1MuDTPtaLut.reset(), L1MuDTPhiLut.reset(), RPCAMCLink.reset(), RPCDCCLink.reset(), EcalPhiSymRecHit.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), PedsOnlyAnalysis.reset(), L1MuDTExtLut.reset(), TB06Reco.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), coral_bridge::AuthenticationCredentialSet.reset(), CalibrationScanAnalysis.reset(), TB06TreeH2.reset(), ApvTimingAnalysis.reset(), DaqScopeModeAnalysis.reset(), cond::persistency::RunInfoProxy.reset(), FastFedCablingAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), RPCLBLink.reset(), cond::persistency::GTProxy.reset(), AlignableBeamSpot.reset(), PedsFullNoiseAnalysis.reset(), CalibrationAnalysis.reset(), CommissioningAnalysis.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TB06RecoH2.reset(), cond::persistency::IOVProxy.reset(), TrackerMap.reset(), LHCInfoImpl::LumiSectionFilter.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), and tkal_create_file_lists._DasCache.reset().

19 
20  def __init__(self, opt):
21 
22  self.reset(opt.what)
23 
24  self.wm=opt.wmcontrol
25  self.revertDqmio=opt.revertDqmio
26  self.addCommand=opt.command
27  self.apply=opt.apply
28  self.commandLineWf=opt.workflow
29  self.overWrite=opt.overWrite
30 
31  self.noRun = opt.noRun
32  return
MatrixReader.MatrixReader.revertDqmio
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:24

Member Function Documentation

def MatrixReader.MatrixReader.createWorkFlows (   self,
  fileNameIn 
)

Definition at line 468 of file MatrixReader.py.

References MatrixReader.MatrixReader.filesPrefMap, MatrixReader.MatrixReader.nameList, print(), str, and MatrixReader.MatrixReader.workFlowSteps.

Referenced by MatrixReader.MatrixReader.prepare().

468 
469  def createWorkFlows(self, fileNameIn):
470 
471  prefixIn = self.filesPrefMap[fileNameIn]
472 
473  # get through the list of items and update the requested workflows only
474  keyList = self.workFlowSteps.keys()
475  ids = []
476  for item in keyList:
477  id, pref = item
478  if pref != prefixIn : continue
479  ids.append(id)
480  ids.sort()
481  for key in ids:
482  val = self.workFlowSteps[(key,prefixIn)]
483  num, name, commands, stepList = val
484  nameId = str(num)+'_'+name
485  if nameId in self.nameList:
486  print("==> duplicate name found for ", nameId)
487  print(' keeping : ', self.nameList[nameId])
488  print(' ignoring : ', val)
489  else:
490  self.nameList[nameId] = val
491 
492  self.workFlows.append(WorkFlow(num, name, commands=commands))
493 
494  return
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
str
#define str(s)
Definition: TestProcessor.cc:54
def MatrixReader.MatrixReader.makeCmd (   self,
  step 
)

Definition at line 97 of file MatrixReader.py.

References str.

Referenced by MatrixReader.MatrixReader.readMatrix().

97 
98  def makeCmd(self, step):
99 
100  cmd = ''
101  cfg = None
102  input = None
103  for k,v in step.items():
104  if 'no_exec' in k : continue # we want to really run it ...
105  if k.lower() == 'cfg':
106  cfg = v
107  continue # do not append to cmd, return separately
108  if k.lower() == 'input':
109  input = v
110  continue # do not append to cmd, return separately
111 
112  #chain the configs
113  #if k.lower() == '--python':
114  # v = 'step%d_%s'%(index,v)
115  cmd += ' ' + k + ' ' + str(v)
116  return cfg, input, cmd
str
#define str(s)
Definition: TestProcessor.cc:54
def MatrixReader.MatrixReader.makeStep (   self,
  step,
  overrides 
)

Definition at line 117 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

117 
118  def makeStep(self,step,overrides):
119  from Configuration.PyReleaseValidation.relval_steps import merge
120  if len(overrides) > 0:
121  copyStep=merge([overrides]+[step])
122  return copyStep
123  else:
124  return step
merge
Definition: merge.py:1
relval_steps
def MatrixReader.MatrixReader.prepare (   self,
  useInput = None,
  refRel = '',
  fromScratch = None 
)

Definition at line 495 of file MatrixReader.py.

References any(), MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, print(), MatrixReader.MatrixReader.readMatrix(), str, and MatrixReader.MatrixReader.what.

495 
496  def prepare(self, useInput=None, refRel='', fromScratch=None):
497 
498  for matrixFile in self.files:
499  if self.what != 'all' and not any('_'+el in matrixFile for el in self.what.split(",")):
500  print("ignoring non-requested file",matrixFile)
501  continue
502  if self.what == 'all' and not self.filesDefault[matrixFile]:
503  print("ignoring",matrixFile,"from default matrix")
504  continue
505 
506  try:
507  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
508  except Exception as e:
509  print("ERROR reading file:", matrixFile, str(e))
510  raise
511 
512  try:
513  self.createWorkFlows(matrixFile)
514  except Exception as e:
515  print("ERROR creating workflows :", str(e))
516  raise
517 
any
bool any(const std::vector< T > &v, const T &what)
Definition: ECalSD.cc:37
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
str
#define str(s)
Definition: TestProcessor.cc:54
def MatrixReader.MatrixReader.readMatrix (   self,
  fileNameIn,
  useInput = None,
  refRel = None,
  fromScratch = None 
)

Definition at line 125 of file MatrixReader.py.

References MatrixReader.MatrixReader.addCommand, MatrixReader.MatrixReader.apply, MatrixReader.MatrixReader.filesPrefMap, join(), MatrixReader.MatrixReader.makeCmd(), MatrixReader.MatrixReader.makeStep(), MatrixReader.MatrixReader.noRun, MatrixReader.MatrixReader.overWrite, print(), sistrip::SpyUtilities.range(), MatrixReader.MatrixReader.relvalModule, MatrixReader.MatrixReader.revertDqmio, str, MatrixReader.MatrixReader.wm, MatrixReader.MatrixReader.workFlowSteps, and ComparisonHelper.zip().

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

125 
126  def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None):
127 
128  prefix = self.filesPrefMap[fileNameIn]
129 
130  print("processing", fileNameIn)
131 
132  try:
133  _tmpMod = __import__( 'Configuration.PyReleaseValidation.'+fileNameIn )
134  self.relvalModule = sys.modules['Configuration.PyReleaseValidation.'+fileNameIn]
135  except Exception as e:
136  print("ERROR importing file ", fileNameIn, str(e))
137  return
138 
139  if useInput is not None:
140  print("request for INPUT for ", useInput)
141 
142 
143  fromInput={}
144 
145  if useInput:
146  for i in useInput:
147  if ':' in i:
148  (ik,il)=i.split(':')
149  if ik=='all':
150  for k in self.relvalModule.workflows.keys():
151  fromInput[float(k)]=int(il)
152  else:
153  fromInput[float(ik)]=int(il)
154  else:
155  if i=='all':
156  for k in self.relvalModule.workflows.keys():
157  fromInput[float(k)]=0
158  else:
159  fromInput[float(i)]=0
160 
161  if fromScratch:
162  fromScratch=map(float,fromScratch)
163  for num in fromScratch:
164  if num in fromInput:
165  fromInput.pop(num)
166  #overwrite steps
167  if self.overWrite:
168  for p in self.overWrite:
169  self.relvalModule.steps.overwrite(p)
170 
171  #change the origin of dataset on the fly
172  if refRel:
173  if ',' in refRel:
174  refRels=refRel.split(',')
175  if len(refRels)!=len(self.relvalModule.baseDataSetRelease):
176  return
177  self.relvalModule.changeRefRelease(
178  self.relvalModule.steps,
179  list(zip(self.relvalModule.baseDataSetRelease,refRels))
180  )
181  else:
182  self.relvalModule.changeRefRelease(
183  self.relvalModule.steps,
184  [(x,refRel) for x in self.relvalModule.baseDataSetRelease]
185  )
186 
187 
188  for num, wfInfo in self.relvalModule.workflows.items():
189  commands=[]
190  wfName = wfInfo[0]
191  stepList = wfInfo[1]
192  stepOverrides=wfInfo.overrides
193  # upgrade case: workflow has basic name, key[, suffix (only special workflows)]
194  wfKey = ""
195  wfSuffix = ""
196  if isinstance(wfName, list) and len(wfName)>1:
197  if len(wfName)>2: wfSuffix = wfName[2]
198  wfKey = wfName[1]
199  wfName = wfName[0]
200  # if no explicit name given for the workflow, use the name of step1
201  if wfName.strip() == '': wfName = stepList[0]
202  # option to specialize the wf as the third item in the WF list
203  addTo=None
204  addCom=None
205  if len(wfInfo)>=3:
206  addCom=wfInfo[2]
207  if not isinstance(addCom, list): addCom=[addCom]
208  #print 'added dict',addCom
209  if len(wfInfo)>=4:
210  addTo=wfInfo[3]
211  #pad with 0
212  while len(addTo)!=len(stepList):
213  addTo.append(0)
214 
215  name=wfName
216  # separate suffixes by + because show() excludes first part of name
217  if len(wfKey)>0:
218  name = name+'+'+wfKey
219  if len(wfSuffix)>0: name = name+wfSuffix
220  stepIndex=0
221  ranStepList=[]
222 
223  #first resolve INPUT possibilities
224  if num in fromInput:
225  ilevel=fromInput[num]
226  #print num,ilevel
227  for (stepIr,step) in enumerate(reversed(stepList)):
228  stepName=step
229  stepI=(len(stepList)-stepIr)-1
230  #print stepIr,step,stepI,ilevel
231  if stepI>ilevel:
232  #print "ignoring"
233  continue
234  if stepI!=0:
235  testName='__'.join(stepList[0:stepI+1])+'INPUT'
236  else:
237  testName=step+'INPUT'
238  #print "JR",stepI,stepIr,testName,stepList
239  if testName in self.relvalModule.steps:
240  #print "JR",stepI,stepIr
241  stepList[stepI]=testName
242  #pop the rest in the list
243  #print "\tmod prepop",stepList
244  for p in range(stepI):
245  stepList.pop(0)
246  #print "\t\tmod",stepList
247  break
248 
249 
250  for (stepI,step) in enumerate(stepList):
251  stepName=step
252  if self.relvalModule.steps[stepName] is None:
253  continue
254  if self.wm:
255  #cannot put a certain number of things in wm
256  if stepName in ['SKIMD','SKIMCOSD','SKIMDreHLT']:
257  continue
258 
259  #replace stepName is needed
260  #if stepName in self.replaceStep
261  if len(name) > 0 : name += '+'
262  #any step can be mirrored with INPUT
263  ## maybe we want too level deep input
264  """
265  if num in fromInput:
266  if step+'INPUT' in self.relvalModule.steps.keys():
267  stepName = step+"INPUT"
268  stepList.remove(step)
269  stepList.insert(stepIndex,stepName)
270  """
271  stepNameTmp = stepName
272  if len(wfKey)>0: stepNameTmp = stepNameTmp.replace('_'+wfKey,"")
273  if len(wfSuffix)>0: stepNameTmp = stepNameTmp.replace(wfSuffix,"")
274  name += stepNameTmp
275  if addCom and (not addTo or addTo[stepIndex]==1):
276  from Configuration.PyReleaseValidation.relval_steps import merge
277  copyStep=merge(addCom+[self.makeStep(self.relvalModule.steps[stepName],stepOverrides)])
278  cfg, input, opts = self.makeCmd(copyStep)
279  else:
280  cfg, input, opts = self.makeCmd(self.makeStep(self.relvalModule.steps[stepName],stepOverrides))
281 
282  if input and cfg :
283  msg = "FATAL ERROR: found both cfg and input for workflow "+str(num)+' step '+stepName
284  raise MatrixException(msg)
285 
286  if input:
287  cmd = input
288  if self.noRun:
289  cmd.run=[]
290  else:
291  if cfg:
292  cmd = 'cmsDriver.py '+cfg+' '+opts
293  else:
294  cmd = 'cmsDriver.py step'+str(stepIndex+1)+' '+opts
295  if self.wm:
296  cmd+=' --io %s.io --python %s.py'%(stepName,stepName)
297  if self.addCommand:
298  if self.apply:
299  if stepIndex in self.apply or stepName in self.apply:
300  cmd +=' '+self.addCommand
301  else:
302  cmd +=' '+self.addCommand
303  if self.wm and self.revertDqmio=='yes':
304  cmd=cmd.replace('DQMIO','DQM')
305  cmd=cmd.replace('--filetype DQM','')
306  commands.append(cmd)
307  ranStepList.append(stepName)
308  stepIndex+=1
309 
310  self.workFlowSteps[(num,prefix)] = (num, name, commands, ranStepList)
311 
312  return
313 
merge
Definition: merge.py:1
MatrixReader.MatrixReader.revertDqmio
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:24
relval_steps
sistrip::SpyUtilities::range
const uint16_t range(const Frame &aFrame)
Definition: SiStripSpyUtilities.cc:76
ComparisonHelper::zip
OutputIterator zip(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, InputIterator2 last2, OutputIterator result, Compare comp)
Definition: L1TStage2CaloLayer1.h:39
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
str
#define str(s)
Definition: TestProcessor.cc:54
def MatrixReader.MatrixReader.reset (   self,
  what = 'all' 
)

Definition at line 33 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.__init__(), and MatrixReader.MatrixReader.showRaw().

33 
34  def reset(self, what='all'):
35 
36  self.what = what
37 
38  #a bunch of information, but not yet the WorkFlow object
39  self.workFlowSteps = {}
40  #the actual WorkFlow objects
41  self.workFlows = []
42  self.nameList = {}
43 
44  self.filesPrefMap = {'relval_standard' : 'std-' ,
45  'relval_highstats': 'hi-' ,
46  'relval_pileup': 'PU-' ,
47  'relval_generator': 'gen-',
48  'relval_extendedgen': 'genExt-',
49  'relval_production': 'prod-' ,
50  'relval_ged': 'ged-',
51  'relval_upgrade':'upg-',
52  'relval_cleanedupgrade':'clnupg-',
53  'relval_gpu':'gpu-',
54  'relval_2017':'2017-',
55  'relval_2026':'2026-',
56  'relval_identity':'id-',
57  'relval_machine': 'mach-',
58  'relval_premix': 'premix-'
59  }
60 
61  self.files = ['relval_standard' ,
62  'relval_highstats',
63  'relval_pileup',
64  'relval_generator',
65  'relval_extendedgen',
66  'relval_production',
67  'relval_ged',
68  'relval_upgrade',
69  'relval_cleanedupgrade',
70  'relval_gpu',
71  'relval_2017',
72  'relval_2026',
73  'relval_identity',
74  'relval_machine',
75  'relval_premix'
76  ]
77  self.filesDefault = {'relval_standard':True ,
78  'relval_highstats':True ,
79  'relval_pileup':True,
80  'relval_generator':True,
81  'relval_extendedgen':True,
82  'relval_production':True,
83  'relval_ged':True,
84  'relval_upgrade':False,
85  'relval_cleanedupgrade':False,
86  'relval_gpu':False,
87  'relval_2017':True,
88  'relval_2026':True,
89  'relval_identity':False,
90  'relval_machine':True,
91  'relval_premix':True
92  }
93 
94  self.relvalModule = None
95 
96  return
def MatrixReader.MatrixReader.show (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 518 of file MatrixReader.py.

References print(), and MatrixReader.MatrixReader.showWorkFlows().

518 
519  def show(self, selected=None, extended=True, cafVeto=True):
520 
521  self.showWorkFlows(selected, extended, cafVeto)
522  print('\n','-'*80,'\n')
523 
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
def MatrixReader.MatrixReader.showRaw (   self,
  useInput,
  refRel = None,
  fromScratch = None,
  what = 'all',
  step1Only = False,
  selected = None 
)

Definition at line 314 of file MatrixReader.py.

References PVValHelper.add(), any(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, join(), print(), sistrip::SpyUtilities.range(), MatrixReader.MatrixReader.readMatrix(), PuppiAlgo.reset(), TPedResult.reset(), BinningPointByMap.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), EcalCondHeader.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), SiPixelStatusManager.reset(), SiStripMiscalibrate::Entry.reset(), MatrixReader.MatrixReader.reset(), IMACalibBlock.reset(), TB06Tree.reset(), SamplingAnalysis.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), L3CalibBlock.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), SiStripPI::Entry.reset(), L1MuDTPtaLut.reset(), RPCAMCLink.reset(), RPCDCCLink.reset(), L1MuDTPhiLut.reset(), EcalPhiSymRecHit.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), PedsOnlyAnalysis.reset(), L1MuDTExtLut.reset(), TB06Reco.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), coral_bridge::AuthenticationCredentialSet.reset(), CalibrationScanAnalysis.reset(), TB06TreeH2.reset(), ApvTimingAnalysis.reset(), DaqScopeModeAnalysis.reset(), cond::persistency::RunInfoProxy.reset(), FastFedCablingAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), RPCLBLink.reset(), cond::persistency::GTProxy.reset(), AlignableBeamSpot.reset(), PedsFullNoiseAnalysis.reset(), CalibrationAnalysis.reset(), CommissioningAnalysis.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TB06RecoH2.reset(), cond::persistency::IOVProxy.reset(), TrackerMap.reset(), LHCInfoImpl::LumiSectionFilter.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), tkal_create_file_lists._DasCache.reset(), MatrixReader.MatrixReader.revertDqmio, str, MatrixReader.MatrixReader.what, and MatrixReader.MatrixReader.workFlowSteps.

314 
315  def showRaw(self, useInput, refRel=None, fromScratch=None, what='all',step1Only=False,selected=None):
316 
317  if selected:
318  selected=map(float,selected)
319  for matrixFile in self.files:
320 
321  self.reset(what)
322 
323  if self.what != 'all' and not any('_'+el in matrixFile for el in self.what.split(",")):
324  print("ignoring non-requested file",matrixFile)
325  continue
326 
327  if self.what == 'all' and not self.filesDefault[matrixFile]:
328  print("ignoring file not used by default (enable with -w)",matrixFile)
329  continue
330 
331  try:
332  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
333  except Exception as e:
334  print("ERROR reading file:", matrixFile, str(e))
335  raise
336 
337  if not self.workFlowSteps: continue
338 
339  dataFileName = matrixFile.replace('relval_', 'cmsDriver_')+'_hlt.txt'
340  outFile = open(dataFileName,'w')
341 
342  print("found ", len(self.workFlowSteps), ' workflows for ', dataFileName)
343  ids = sorted(self.workFlowSteps.keys())
344  indexAndSteps=[]
345 
346  writtenWF=0
347  for key in ids:
348  if selected and not (key[0] in selected):
349  continue
350  #trick to skip the HImix IB test
351  if key[0]==203.1 or key[0]==204.1 or key[0]==205.1 or key[0]==4.51 or key[0]==4.52: continue
352  num, name, commands, stepList = self.workFlowSteps[key]
353 
354  wfName,stepNames= name.split('+',1)
355 
356  stepNames=stepNames.replace('+SKIMCOSD','')
357  stepNames=stepNames.replace('+SKIMD','')
358  if 'HARVEST' in stepNames:
359  #find out automatically what to remove
360  exactb=stepNames.index('+HARVEST')
361  exacte=stepNames.index('+',exactb+1) if ('+' in stepNames[exactb+1:]) else (len(stepNames))
362  stepNames=stepNames.replace(stepNames[exactb:exacte],'')
363  otherSteps = None
364  if '+' in stepNames:
365  step1,otherSteps = stepNames.split('+',1)
366 
367  line = str(num) + ' ++ '+ wfName
368  if otherSteps and not step1Only:
369  line += ' ++ ' +otherSteps.replace('+',',')
370  else:
371  line += ' ++ none'
372  inputInfo=None
373  if not isinstance(commands[0],str):
374  inputInfo=commands[0]
375  if otherSteps:
376  for (i,c) in enumerate(otherSteps.split('+')):
377  #pad with set
378  for p in range(len(indexAndSteps),i+2):
379  indexAndSteps.append(set())
380  indexAndSteps[i+1].add((c,commands[i+1]))
381 
382  if inputInfo :
383  #skip the samples from INPUT when step1Only is on
384  if step1Only: continue
385  line += ' ++ REALDATA: '+inputInfo.dataSet
386  if inputInfo.run!=[]: line += ', RUN:'+'|'.join(map(str,inputInfo.run))
387  line += ', FILES: ' +str(inputInfo.files)
388  line += ', EVENTS: '+str(inputInfo.events)
389  if inputInfo.label!='':
390  line += ', LABEL: ' +inputInfo.label
391  line += ', LOCATION:'+inputInfo.location
392  line += ' @@@'
393  else:
394  line += ' @@@ '+commands[0]
395  if self.revertDqmio=='yes':
396  line=line.replace('DQMIO','DQM')
397  writtenWF+=1
398  outFile.write(line+'\n')
399 
400 
401  outFile.write('\n'+'\n')
402  if step1Only: continue
403 
404  for (index,s) in enumerate(indexAndSteps):
405  for (stepName,cmd) in s:
406  stepIndex=index+1
407  if 'dasquery.log' in cmd: continue
408  line = 'STEP%d ++ '%(stepIndex,) +stepName + ' @@@ '+cmd
409  if self.revertDqmio=='yes':
410  line=line.replace('DQMIO','DQM')
411  outFile.write(line+'\n')
412  outFile.write('\n'+'\n')
413  outFile.close()
414  print("wrote ",writtenWF, ' workflow'+('s' if (writtenWF!=1) else ''),' to ', outFile.name)
415  return
MatrixReader.MatrixReader.revertDqmio
revertDqmio
maybe we want too level deep input
Definition: MatrixReader.py:24
any
bool any(const std::vector< T > &v, const T &what)
Definition: ECalSD.cc:37
sistrip::SpyUtilities::range
const uint16_t range(const Frame &aFrame)
Definition: SiStripSpyUtilities.cc:76
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
PVValHelper::add
void add(std::map< std::string, TH1 * > &h, TH1 *hist)
Definition: PVValidationHelpers.cc:12
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
str
#define str(s)
Definition: TestProcessor.cc:54
def MatrixReader.MatrixReader.showWorkFlows (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 431 of file MatrixReader.py.

References print(), sistrip::SpyUtilities.range(), str, and MatrixReader.MatrixReader.workFlowsByLocation().

Referenced by MatrixReader.MatrixReader.show().

431 
432  def showWorkFlows(self, selected=None, extended=True, cafVeto=True):
433  if selected: selected = list(map(float,selected))
434  wfs = self.workFlowsByLocation(cafVeto)
435  maxLen = 100 # for summary, limit width of output
436  fmt1 = "%-6s %-35s [1]: %s ..."
437  fmt2 = " %35s [%d]: %s ..."
438  print("\nfound a total of ", len(wfs), ' workflows:')
439  if selected:
440  print(" of which the following", len(selected), 'were selected:')
441  #-ap for now:
442  maxLen = -1 # for individual listing, no limit on width
443  fmt1 = "%-6s %-35s [1]: %s "
444  fmt2 = " %35s [%d]: %s"
445 
446  N=[]
447  for wf in wfs:
448  if selected and float(wf.numId) not in selected: continue
449  if extended: print('')
450  #pad with zeros
451  for i in range(len(N),len(wf.cmds)): N.append(0)
452  N[len(wf.cmds)-1]+=1
453  wfName, stepNames = wf.nameId.split('+',1)
454  for i,s in enumerate(wf.cmds):
455  if extended:
456  if i==0:
457  print(fmt1 % (wf.numId, stepNames, (str(s)+' ')[:maxLen]))
458  else:
459  print(fmt2 % ( ' ', i+1, (str(s)+' ')[:maxLen]))
460  else:
461  print("%-6s %-35s "% (wf.numId, stepNames))
462  break
463  print('')
464  for i,n in enumerate(N):
465  if n: print(n,'workflows with',i+1,'steps')
466 
467  return
sistrip::SpyUtilities::range
const uint16_t range(const Frame &aFrame)
Definition: SiStripSpyUtilities.cc:76
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
str
#define str(s)
Definition: TestProcessor.cc:54
def MatrixReader.MatrixReader.updateDB (   self)

Definition at line 524 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

524 
525  def updateDB(self):
526 
527  import pickle
528  pickle.dump(self.workFlows, open('theMatrix.pkl', 'w') )
529 
530  return
531 
def MatrixReader.MatrixReader.workFlowsByLocation (   self,
  cafVeto = True 
)

Definition at line 416 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

Referenced by MatrixReader.MatrixReader.showWorkFlows().

416 
417  def workFlowsByLocation(self, cafVeto=True):
418  # Check if we are on CAF
419  onCAF = False
420  if 'cms/caf/cms' in os.environ['CMS_PATH']:
421  onCAF = True
422 
423  workflows = []
424  for workflow in self.workFlows:
425  if isinstance(workflow.cmds[0], InputInfo):
426  if cafVeto and (workflow.cmds[0].location == 'CAF' and not onCAF):
427  continue
428  workflows.append(workflow)
429 
430  return workflows

Member Data Documentation

MatrixReader.MatrixReader.addCommand

Definition at line 25 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.apply

Definition at line 26 of file MatrixReader.py.

Referenced by heavyIonTools.ConfigureHeavyIons.__call__(), coreTools.RunOnData.__call__(), trackTools.MakeAODTrackCandidates.__call__(), runJetUncertainties.RunJetUncertainties.__call__(), metTools.AddMETCollection.__call__(), heavyIonTools.ProductionDefaults.__call__(), cmsswVersionTools.PickRelValInputFiles.__call__(), coreTools.RemoveMCMatching.__call__(), trackTools.MakePATTrackCandidates.__call__(), trigTools.SwitchOnTrigger.__call__(), runMETCorrectionsAndUncertainties.RunMETCorrectionsAndUncertainties.__call__(), heavyIonTools.SelectionDefaults.__call__(), heavyIonTools.DisbaleMonteCarloDeps.__call__(), trigTools.SwitchOnTriggerStandAlone.__call__(), tauTools.AddTauCollection.__call__(), trackTools.MakeTrackCandidates.__call__(), trigTools.SwitchOnTriggerMatching.__call__(), trigTools.SwitchOnTriggerMatchingStandAlone.__call__(), trigTools.SwitchOnTriggerMatchEmbedding.__call__(), jetTools.AddJetCollection.__call__(), jetTools.SwitchJetCollection.__call__(), jetTools.UpdateJetCollection.__call__(), jetTools.AddJetID.__call__(), jetTools.SetTagInfos.__call__(), and MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.commandLineWf

Definition at line 27 of file MatrixReader.py.

MatrixReader.MatrixReader.files

Definition at line 60 of file MatrixReader.py.

Referenced by chain.Chain._guessTreeName(), dataset.BaseDataset.listOfFiles(), dataset.BaseDataset.listOfGoodFiles(), MatrixReader.MatrixReader.prepare(), dataset.BaseDataset.printFiles(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.filesDefault

Definition at line 76 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.filesPrefMap

Definition at line 43 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows(), and MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.nameList

Definition at line 41 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows().

MatrixReader.MatrixReader.noRun

Definition at line 30 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.overWrite

Definition at line 28 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.relvalModule

Definition at line 93 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.revertDqmio

maybe we want too level deep input

Definition at line 24 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.what

Definition at line 35 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

MatrixReader.MatrixReader.wm

Definition at line 23 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

MatrixReader.MatrixReader.workFlows

Definition at line 40 of file MatrixReader.py.

Referenced by MatrixRunner.MatrixRunner.runTests(), MatrixReader.MatrixReader.updateDB(), and MatrixReader.MatrixReader.workFlowsByLocation().

MatrixReader.MatrixReader.workFlowSteps

Definition at line 38 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

Generated for CMSSW Reference Manual by doxygen 1.8.5