CMS 3D CMS Logo

List of all members | Public Member Functions | Public Attributes
MatrixReader.MatrixReader Class Reference
Inheritance diagram for MatrixReader.MatrixReader:

Public Member Functions

def __init__ (self, opt)
 
def createWorkFlows (self, fileNameIn)
 
def makeCmd (self, step)
 
def makeStep (self, step, overrides)
 
def prepare (self, useInput=None, refRel='', fromScratch=None)
 
def readMatrix (self, fileNameIn, useInput=None, refRel=None, fromScratch=None)
 
def reset (self, what='all')
 
def show (self, selected=None, extended=True, cafVeto=True)
 
def showRaw (self, useInput, refRel=None, fromScratch=None, what='all', step1Only=False, selected=None)
 
def showWorkFlows (self, selected=None, extended=True, cafVeto=True)
 
def updateDB (self)
 
def workFlowsByLocation (self, cafVeto=True)
 

Public Attributes

 addCommand
 
 apply
 
 commandLineWf
 
 files
 
 filesDefault
 
 filesPrefMap
 
 nameList
 
 noRun
 
 overWrite
 
 relvalModule
 
 revertDqmio
 maybe we want too level deep input More...
 
 what
 
 wm
 
 workFlows
 
 workFlowSteps
 

Detailed Description

Definition at line 17 of file MatrixReader.py.

Constructor & Destructor Documentation

◆ __init__()

def MatrixReader.MatrixReader.__init__ (   self,
  opt 
)

Definition at line 19 of file MatrixReader.py.

References PuppiAlgo.reset(), LumiSectionFilter< T >.reset(), TPedResult.reset(), BinningPointByMap.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), EcalCondHeader.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), SiPixelStatusManager.reset(), SiStripMiscalibrate::Entry.reset(), MatrixReader.MatrixReader.reset(), TB06Tree.reset(), IMACalibBlock.reset(), SamplingAnalysis.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), L3CalibBlock.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), L1MuDTTFParameters.reset(), L1MuDTTFMasks.reset(), L1MuDTPtaLut.reset(), RPCAMCLink.reset(), RPCDCCLink.reset(), L1MuDTPhiLut.reset(), EcalPhiSymRecHit.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), SiStripPI::Entry.reset(), PedsOnlyAnalysis.reset(), L1MuDTExtLut.reset(), TB06Reco.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), coral_bridge::AuthenticationCredentialSet.reset(), CalibrationScanAnalysis.reset(), TB06TreeH2.reset(), ApvTimingAnalysis.reset(), DaqScopeModeAnalysis.reset(), FastFedCablingAnalysis.reset(), RPCLBLink.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), cond::persistency::RunInfoProxy.reset(), AlignableBeamSpot.reset(), cond::persistency::GTProxy.reset(), PedsFullNoiseAnalysis.reset(), CalibrationAnalysis.reset(), CommissioningAnalysis.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TB06RecoH2.reset(), cond::persistency::IOVProxy.reset(), TrackerMap.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), and tkal_create_file_lists._DasCache.reset().

19  def __init__(self, opt):
20 
21  self.reset(opt.what)
22 
23  self.wm=opt.wmcontrol
24  self.revertDqmio=opt.revertDqmio
25  self.addCommand=opt.command
26  self.apply=opt.apply
27  self.commandLineWf=opt.workflow
28  self.overWrite=opt.overWrite
29 
30  self.noRun = opt.noRun
31  return
32 
def __init__(self, dataset, job_number, job_id, job_name, isDA, isMC, applyBOWS, applyEXTRACOND, extraconditions, runboundary, lumilist, intlumi, maxevents, gt, allFromGT, alignmentDB, alignmentTAG, apeDB, apeTAG, bowDB, bowTAG, vertextype, tracktype, refittertype, ttrhtype, applyruncontrol, ptcut, CMSSW_dir, the_dir)

Member Function Documentation

◆ createWorkFlows()

def MatrixReader.MatrixReader.createWorkFlows (   self,
  fileNameIn 
)

Definition at line 469 of file MatrixReader.py.

References mps_setup.append, MatrixReader.MatrixReader.filesPrefMap, relativeConstraints.keys, MatrixReader.MatrixReader.nameList, print(), str, MatrixReader.MatrixReader.workFlows, and MatrixReader.MatrixReader.workFlowSteps.

Referenced by MatrixReader.MatrixReader.prepare().

469  def createWorkFlows(self, fileNameIn):
470 
471  prefixIn = self.filesPrefMap[fileNameIn]
472 
473  # get through the list of items and update the requested workflows only
474  keyList = self.workFlowSteps.keys()
475  ids = []
476  for item in keyList:
477  id, pref = item
478  if pref != prefixIn : continue
479  ids.append(id)
480  ids.sort()
481  for key in ids:
482  val = self.workFlowSteps[(key,prefixIn)]
483  num, name, commands, stepList = val
484  nameId = str(num)+'_'+name
485  if nameId in self.nameList:
486  print("==> duplicate name found for ", nameId)
487  print(' keeping : ', self.nameList[nameId])
488  print(' ignoring : ', val)
489  else:
490  self.nameList[nameId] = val
491 
492  self.workFlows.append(WorkFlow(num, name, commands=commands, stepList=stepList))
493 
494  return
495 
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
#define str(s)

◆ makeCmd()

def MatrixReader.MatrixReader.makeCmd (   self,
  step 
)

Definition at line 100 of file MatrixReader.py.

References str.

Referenced by MatrixReader.MatrixReader.readMatrix().

100  def makeCmd(self, step):
101 
102  cmd = ''
103  cfg = None
104  input = None
105  for k,v in step.items():
106  if 'no_exec' in k : continue # we want to really run it ...
107  if k.lower() == 'cfg':
108  cfg = v
109  continue # do not append to cmd, return separately
110  if k.lower() == 'input':
111  input = v
112  continue # do not append to cmd, return separately
113 
114  #chain the configs
115  #if k.lower() == '--python':
116  # v = 'step%d_%s'%(index,v)
117  cmd += ' ' + k + ' ' + str(v)
118  return cfg, input, cmd
119 
#define str(s)

◆ makeStep()

def MatrixReader.MatrixReader.makeStep (   self,
  step,
  overrides 
)

Definition at line 120 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

120  def makeStep(self,step,overrides):
122  if len(overrides) > 0:
123  copyStep=merge([overrides]+[step])
124  return copyStep
125  else:
126  return step
127 
Definition: merge.py:1

◆ prepare()

def MatrixReader.MatrixReader.prepare (   self,
  useInput = None,
  refRel = '',
  fromScratch = None 
)

Definition at line 496 of file MatrixReader.py.

References any(), MatrixReader.MatrixReader.createWorkFlows(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, print(), MatrixReader.MatrixReader.readMatrix(), submitPVValidationJobs.split(), str, and MatrixReader.MatrixReader.what.

496  def prepare(self, useInput=None, refRel='', fromScratch=None):
497 
498  for matrixFile in self.files:
499  if self.what != 'all' and not any('_'+el in matrixFile for el in self.what.split(",")):
500  print("ignoring non-requested file",matrixFile)
501  continue
502  if self.what == 'all' and not self.filesDefault[matrixFile]:
503  print("ignoring",matrixFile,"from default matrix")
504  continue
505 
506  try:
507  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
508  except Exception as e:
509  print("ERROR reading file:", matrixFile, str(e))
510  raise
511 
512  try:
513  self.createWorkFlows(matrixFile)
514  except Exception as e:
515  print("ERROR creating workflows :", str(e))
516  raise
517 
518 
bool any(const std::vector< T > &v, const T &what)
Definition: ECalSD.cc:37
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
#define str(s)

◆ readMatrix()

def MatrixReader.MatrixReader.readMatrix (   self,
  fileNameIn,
  useInput = None,
  refRel = None,
  fromScratch = None 
)

Definition at line 128 of file MatrixReader.py.

References MatrixReader.MatrixReader.addCommand, MatrixReader.MatrixReader.apply, MatrixUtil.changeRefRelease(), MatrixReader.MatrixReader.filesPrefMap, nano_mu_digi_cff.float, createfilelist.int, join(), MatrixReader.MatrixReader.makeCmd(), MatrixReader.MatrixReader.makeStep(), genParticles_cff.map, MatrixReader.MatrixReader.noRun, MatrixReader.MatrixReader.overWrite, print(), FastTimerService_cff.range, MatrixReader.MatrixReader.relvalModule, MatrixReader.MatrixReader.revertDqmio, str, MatrixReader.MatrixReader.wm, MatrixReader.MatrixReader.workFlowSteps, and ComparisonHelper.zip().

Referenced by MatrixReader.MatrixReader.prepare(), and MatrixReader.MatrixReader.showRaw().

128  def readMatrix(self, fileNameIn, useInput=None, refRel=None, fromScratch=None):
129 
130  prefix = self.filesPrefMap[fileNameIn]
131 
132  print("processing", fileNameIn)
133 
134  try:
135  _tmpMod = __import__( 'Configuration.PyReleaseValidation.'+fileNameIn )
136  self.relvalModule = sys.modules['Configuration.PyReleaseValidation.'+fileNameIn]
137  except Exception as e:
138  print("ERROR importing file ", fileNameIn, str(e))
139  return
140 
141  if useInput is not None:
142  print("request for INPUT for ", useInput)
143 
144 
145  fromInput={}
146 
147  if useInput:
148  for i in useInput:
149  if ':' in i:
150  (ik,il)=i.split(':')
151  if ik=='all':
152  for k in self.relvalModule.workflows.keys():
153  fromInput[float(k)]=int(il)
154  else:
155  fromInput[float(ik)]=int(il)
156  else:
157  if i=='all':
158  for k in self.relvalModule.workflows.keys():
159  fromInput[float(k)]=0
160  else:
161  fromInput[float(i)]=0
162 
163  if fromScratch:
164  fromScratch=map(float,fromScratch)
165  for num in fromScratch:
166  if num in fromInput:
167  fromInput.pop(num)
168  #overwrite steps
169  if self.overWrite:
170  for p in self.overWrite:
171  self.relvalModule.steps.overwrite(p)
172 
173  #change the origin of dataset on the fly
174  if refRel:
175  if ',' in refRel:
176  refRels=refRel.split(',')
177  if len(refRels)!=len(self.relvalModule.baseDataSetRelease):
178  return
179  self.relvalModule.changeRefRelease(
180  self.relvalModule.steps,
181  list(zip(self.relvalModule.baseDataSetRelease,refRels))
182  )
183  else:
184  self.relvalModule.changeRefRelease(
185  self.relvalModule.steps,
186  [(x,refRel) for x in self.relvalModule.baseDataSetRelease]
187  )
188 
189  for num, wfInfo in self.relvalModule.workflows.items():
190  commands=[]
191  wfName = wfInfo[0]
192  stepList = wfInfo[1]
193  stepOverrides=wfInfo.overrides
194  # upgrade case: workflow has basic name, key[, suffix (only special workflows)]
195  wfKey = ""
196  wfSuffix = ""
197  if isinstance(wfName, list) and len(wfName)>1:
198  if len(wfName)>2: wfSuffix = wfName[2]
199  wfKey = wfName[1]
200  wfName = wfName[0]
201  # if no explicit name given for the workflow, use the name of step1
202  if wfName.strip() == '': wfName = stepList[0]
203  # option to specialize the wf as the third item in the WF list
204  addTo=None
205  addCom=None
206  if len(wfInfo)>=3:
207  addCom=wfInfo[2]
208  if not isinstance(addCom, list): addCom=[addCom]
209  #print 'added dict',addCom
210  if len(wfInfo)>=4:
211  addTo=wfInfo[3]
212  #pad with 0
213  while len(addTo)!=len(stepList):
214  addTo.append(0)
215 
216  name=wfName
217  # separate suffixes by + because show() excludes first part of name
218  if len(wfKey)>0:
219  name = name+'+'+wfKey
220  if len(wfSuffix)>0: name = name+wfSuffix
221  stepIndex=0
222  ranStepList=[]
223  name_for_workflow = name
224 
225  #first resolve INPUT possibilities
226  if num in fromInput:
227  ilevel=fromInput[num]
228  #print num,ilevel
229  for (stepIr,step) in enumerate(reversed(stepList)):
230  stepName=step
231  stepI=(len(stepList)-stepIr)-1
232  #print stepIr,step,stepI,ilevel
233  if stepI>ilevel:
234  #print "ignoring"
235  continue
236  if stepI!=0:
237  testName='__'.join(stepList[0:stepI+1])+'INPUT'
238  else:
239  testName=step+'INPUT'
240  #print "JR",stepI,stepIr,testName,stepList
241  if testName in self.relvalModule.steps:
242  #print "JR",stepI,stepIr
243  stepList[stepI]=testName
244  #pop the rest in the list
245  #print "\tmod prepop",stepList
246  for p in range(stepI):
247  stepList.pop(0)
248  #print "\t\tmod",stepList
249  break
250 
251  for (stepI,step) in enumerate(stepList):
252  stepName=step
253  if self.relvalModule.steps[stepName] is None:
254  continue
255  if self.wm:
256  #cannot put a certain number of things in wm
257  if stepName in ['SKIMD','SKIMCOSD','SKIMDreHLT']:
258  continue
259 
260  #replace stepName is needed
261  #if stepName in self.replaceStep
262  if len(name) > 0 : name += '+'
263  #any step can be mirrored with INPUT
264 
265  """
266  if num in fromInput:
267  if step+'INPUT' in self.relvalModule.steps.keys():
268  stepName = step+"INPUT"
269  stepList.remove(step)
270  stepList.insert(stepIndex,stepName)
271  """
272  stepNameTmp = stepName
273  if len(wfKey)>0: stepNameTmp = stepNameTmp.replace('_'+wfKey,"")
274  if len(wfSuffix)>0: stepNameTmp = stepNameTmp.replace(wfSuffix,"")
275  name += stepNameTmp
276  if addCom and (not addTo or addTo[stepIndex]==1):
278  copyStep=merge(addCom+[self.makeStep(self.relvalModule.steps[stepName],stepOverrides)])
279  cfg, input, opts = self.makeCmd(copyStep)
280  else:
281  cfg, input, opts = self.makeCmd(self.makeStep(self.relvalModule.steps[stepName],stepOverrides))
282 
283  if input and cfg :
284  msg = "FATAL ERROR: found both cfg and input for workflow "+str(num)+' step '+stepName
285  raise MatrixException(msg)
286 
287  if input:
288  cmd = input
289  if self.noRun:
290  cmd.run=[]
291  else:
292  if cfg:
293  cmd = 'cmsDriver.py '+cfg+' '+opts
294  else:
295  cmd = 'cmsDriver.py step'+str(stepIndex+1)+' '+opts
296  if self.wm:
297  cmd+=' --io %s.io --python %s.py'%(stepName,stepName)
298  if self.addCommand:
299  if self.apply:
300  if stepIndex in self.apply or stepName in self.apply:
301  cmd +=' '+self.addCommand
302  else:
303  cmd +=' '+self.addCommand
304  if self.wm and self.revertDqmio=='yes':
305  cmd=cmd.replace('DQMIO','DQM')
306  cmd=cmd.replace('--filetype DQM','')
307  commands.append(cmd)
308  ranStepList.append(stepName)
309  stepIndex+=1
310  self.workFlowSteps[(num,prefix)] = (num, name_for_workflow, commands, ranStepList)
311 
312  return
313 
314 
Definition: merge.py:1
OutputIterator zip(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, InputIterator2 last2, OutputIterator result, Compare comp)
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
def changeRefRelease(steps, listOfPairs)
Definition: MatrixUtil.py:240
#define str(s)

◆ reset()

def MatrixReader.MatrixReader.reset (   self,
  what = 'all' 
)

Definition at line 33 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.__init__(), and MatrixReader.MatrixReader.showRaw().

33  def reset(self, what='all'):
34 
35  self.what = what
36 
37  #a bunch of information, but not yet the WorkFlow object
38  self.workFlowSteps = {}
39  #the actual WorkFlow objects
40  self.workFlows = []
41  self.nameList = {}
42 
43  self.filesPrefMap = {'relval_standard' : 'std-' ,
44  'relval_highstats': 'hi-' ,
45  'relval_pileup': 'PU-' ,
46  'relval_generator': 'gen-',
47  'relval_extendedgen': 'genExt-',
48  'relval_production': 'prod-' ,
49  'relval_ged': 'ged-',
50  'relval_upgrade':'upg-',
51  'relval_cleanedupgrade':'clnupg-',
52  'relval_gpu':'gpu-',
53  'relval_2017':'2017-',
54  'relval_2026':'2026-',
55  'relval_identity':'id-',
56  'relval_machine': 'mach-',
57  'relval_premix': 'premix-',
58  'relval_nano':'nano-'
59  }
60 
61  self.files = ['relval_standard' ,
62  'relval_highstats',
63  'relval_pileup',
64  'relval_generator',
65  'relval_extendedgen',
66  'relval_production',
67  'relval_ged',
68  'relval_upgrade',
69  'relval_cleanedupgrade',
70  'relval_gpu',
71  'relval_2017',
72  'relval_2026',
73  'relval_identity',
74  'relval_machine',
75  'relval_premix',
76  'relval_nano'
77  ]
78  self.filesDefault = {'relval_standard':True ,
79  'relval_highstats':True ,
80  'relval_pileup':True,
81  'relval_generator':True,
82  'relval_extendedgen':True,
83  'relval_production':True,
84  'relval_ged':True,
85  'relval_upgrade':False,
86  'relval_cleanedupgrade':False,
87  'relval_gpu':False,
88  'relval_2017':True,
89  'relval_2026':True,
90  'relval_identity':False,
91  'relval_machine':True,
92  'relval_premix':True,
93  'relval_nano':True
94  }
95 
96  self.relvalModule = None
97 
98  return
99 
void reset(double vett[256])
Definition: TPedValues.cc:11

◆ show()

def MatrixReader.MatrixReader.show (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 519 of file MatrixReader.py.

References print(), and MatrixReader.MatrixReader.showWorkFlows().

519  def show(self, selected=None, extended=True, cafVeto=True):
520 
521  self.showWorkFlows(selected, extended, cafVeto)
522  print('\n','-'*80,'\n')
523 
524 
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47

◆ showRaw()

def MatrixReader.MatrixReader.showRaw (   self,
  useInput,
  refRel = None,
  fromScratch = None,
  what = 'all',
  step1Only = False,
  selected = None 
)

Definition at line 315 of file MatrixReader.py.

References PVValHelper.add(), any(), MatrixReader.MatrixReader.files, MatrixReader.MatrixReader.filesDefault, join(), relativeConstraints.keys, genParticles_cff.map, print(), FastTimerService_cff.range, MatrixReader.MatrixReader.readMatrix(), PuppiAlgo.reset(), LumiSectionFilter< T >.reset(), TPedResult.reset(), BinningPointByMap.reset(), AlcaBeamSpotManager.reset(), RPCFebConnector.reset(), EcalCondHeader.reset(), VEcalCalibBlock.reset(), ApvLatencyAnalysis.reset(), SiPixelStatusManager.reset(), SiStripMiscalibrate::Entry.reset(), MatrixReader.MatrixReader.reset(), IMACalibBlock.reset(), TB06Tree.reset(), SamplingAnalysis.reset(), FactorizedJetCorrectorCalculator::VariableValues.reset(), L3CalibBlock.reset(), pos::PixelROCStatus.reset(), FedTimingAnalysis.reset(), L1MuDTTFMasks.reset(), L1MuDTTFParameters.reset(), L1MuDTPtaLut.reset(), RPCAMCLink.reset(), RPCDCCLink.reset(), L1MuDTPhiLut.reset(), EcalPhiSymRecHit.reset(), FedCablingAnalysis.reset(), L1MuDTEtaPatternLut.reset(), L1MuDTQualPatternLut.reset(), SiStripPI::Entry.reset(), PedsOnlyAnalysis.reset(), L1MuDTExtLut.reset(), TB06Reco.reset(), VpspScanAnalysis.reset(), PhysicsTools::Calibration::Histogram2D< Value_t, AxisX_t, AxisY_t >.reset(), PhysicsTools::Calibration::Histogram< Value_t, Axis_t >.reset(), NoiseAnalysis.reset(), PedestalsAnalysis.reset(), OptoScanAnalysis.reset(), coral_bridge::AuthenticationCredentialSet.reset(), CalibrationScanAnalysis.reset(), TB06TreeH2.reset(), ApvTimingAnalysis.reset(), DaqScopeModeAnalysis.reset(), FastFedCablingAnalysis.reset(), PhysicsTools::Calibration::Histogram3D< Value_t, AxisX_t, AxisY_t, AxisZ_t >.reset(), RPCLBLink.reset(), cond::persistency::RunInfoProxy.reset(), AlignableBeamSpot.reset(), cond::persistency::GTProxy.reset(), PedsFullNoiseAnalysis.reset(), CalibrationAnalysis.reset(), CommissioningAnalysis.reset(), cond::XMLAuthenticationService::XMLAuthenticationService.reset(), TB06RecoH2.reset(), cond::persistency::IOVProxy.reset(), TrackerMap.reset(), data_sources.json_list.reset(), DTTFBitArray< N >.reset(), tkal_create_file_lists._DasCache.reset(), MatrixReader.MatrixReader.revertDqmio, submitPVValidationJobs.split(), str, MatrixReader.MatrixReader.what, and MatrixReader.MatrixReader.workFlowSteps.

315  def showRaw(self, useInput, refRel=None, fromScratch=None, what='all',step1Only=False,selected=None):
316 
317  if selected:
318  selected=map(float,selected)
319  for matrixFile in self.files:
320 
321  self.reset(what)
322 
323  if self.what != 'all' and not any('_'+el in matrixFile for el in self.what.split(",")):
324  print("ignoring non-requested file",matrixFile)
325  continue
326 
327  if self.what == 'all' and not self.filesDefault[matrixFile]:
328  print("ignoring file not used by default (enable with -w)",matrixFile)
329  continue
330 
331  try:
332  self.readMatrix(matrixFile, useInput, refRel, fromScratch)
333  except Exception as e:
334  print("ERROR reading file:", matrixFile, str(e))
335  raise
336 
337  if not self.workFlowSteps: continue
338 
339  dataFileName = matrixFile.replace('relval_', 'cmsDriver_')+'_hlt.txt'
340  outFile = open(dataFileName,'w')
341 
342  print("found ", len(self.workFlowSteps), ' workflows for ', dataFileName)
343  ids = sorted(self.workFlowSteps.keys())
344  indexAndSteps=[]
345 
346  writtenWF=0
347  for key in ids:
348  if selected and not (key[0] in selected):
349  continue
350  #trick to skip the HImix IB test
351  if key[0]==203.1 or key[0]==204.1 or key[0]==205.1 or key[0]==4.51 or key[0]==4.52: continue
352  num, name, commands, stepList = self.workFlowSteps[key]
353  wfName,stepNames= name.split('+',1)
354 
355  stepNames=stepNames.replace('+SKIMCOSD','')
356  stepNames=stepNames.replace('+SKIMD','')
357  if 'HARVEST' in stepNames:
358  #find out automatically what to remove
359  exactb=stepNames.index('+HARVEST')
360  exacte=stepNames.index('+',exactb+1) if ('+' in stepNames[exactb+1:]) else (len(stepNames))
361  stepNames=stepNames.replace(stepNames[exactb:exacte],'')
362  otherSteps = None
363  if '+' in stepNames:
364  step1,otherSteps = stepNames.split('+',1)
365 
366  line = str(num) + ' ++ '+ wfName
367  if otherSteps and not step1Only:
368  line += ' ++ ' +otherSteps.replace('+',',')
369  else:
370  line += ' ++ none'
371  inputInfo=None
372  if not isinstance(commands[0],str):
373  inputInfo=commands[0]
374  if otherSteps:
375  for (i,c) in enumerate(otherSteps.split('+')):
376  #pad with set
377  for p in range(len(indexAndSteps),i+2):
378  indexAndSteps.append(set())
379  indexAndSteps[i+1].add((c,commands[i+1]))
380 
381  if inputInfo :
382  #skip the samples from INPUT when step1Only is on
383  if step1Only: continue
384  line += ' ++ REALDATA: '+inputInfo.dataSet
385  if inputInfo.run!=[]: line += ', RUN:'+'|'.join(map(str,inputInfo.run))
386  line += ', FILES: ' +str(inputInfo.files)
387  line += ', EVENTS: '+str(inputInfo.events)
388  if inputInfo.label!='':
389  line += ', LABEL: ' +inputInfo.label
390  line += ', LOCATION:'+inputInfo.location
391  line += ' @@@'
392  else:
393  line += ' @@@ '+commands[0]
394  if self.revertDqmio=='yes':
395  line=line.replace('DQMIO','DQM')
396  writtenWF+=1
397  outFile.write(line+'\n')
398 
399 
400  outFile.write('\n'+'\n')
401  if step1Only: continue
402 
403  for (index,s) in enumerate(indexAndSteps):
404  for (stepName,cmd) in s:
405  stepIndex=index+1
406  if 'dasquery.log' in cmd: continue
407  line = 'STEP%d ++ '%(stepIndex,) +stepName + ' @@@ '+cmd
408  if self.revertDqmio=='yes':
409  line=line.replace('DQMIO','DQM')
410  outFile.write(line+'\n')
411  outFile.write('\n'+'\n')
412  outFile.close()
413  print("wrote ",writtenWF, ' workflow'+('s' if (writtenWF!=1) else ''),' to ', outFile.name)
414  return
415 
bool any(const std::vector< T > &v, const T &what)
Definition: ECalSD.cc:37
def showRaw(opt)
Definition: runTheMatrix.py:11
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
void add(std::map< std::string, TH1 *> &h, TH1 *hist)
#define str(s)

◆ showWorkFlows()

def MatrixReader.MatrixReader.showWorkFlows (   self,
  selected = None,
  extended = True,
  cafVeto = True 
)

Definition at line 431 of file MatrixReader.py.

References nano_mu_digi_cff.float, join(), genParticles_cff.map, print(), FastTimerService_cff.range, str, and MatrixReader.MatrixReader.workFlowsByLocation().

Referenced by MatrixReader.MatrixReader.show().

431  def showWorkFlows(self, selected=None, extended=True, cafVeto=True):
432  if selected: selected = list(map(float,selected))
433  wfs = self.workFlowsByLocation(cafVeto)
434  maxLen = 100 # for summary, limit width of output
435  fmt1 = "%-6s %-35s [1]: %s ..."
436  fmt2 = " %35s [%d]: %s ..."
437  print("\nfound a total of ", len(wfs), ' workflows:')
438  if selected:
439  print(" of which the following", len(selected), 'were selected:')
440  #-ap for now:
441  maxLen = -1 # for individual listing, no limit on width
442  fmt1 = "%-6s %-35s [1]: %s "
443  fmt2 = " %35s [%d]: %s"
444 
445  N=[]
446  for wf in wfs:
447  if selected and float(wf.numId) not in selected: continue
448  if extended: print('')
449  #pad with zeros
450  for i in range(len(N),len(wf.cmds)): N.append(0)
451  N[len(wf.cmds)-1]+=1
452  wfName = wf.nameId
453  stepNames = '+'.join(wf.stepList)
454  for i,s in enumerate(wf.cmds):
455  if extended:
456  if i==0:
457  print(fmt1 % (wf.numId, stepNames, (str(s)+' ')[:maxLen]))
458  else:
459  print(fmt2 % ( ' ', i+1, (str(s)+' ')[:maxLen]))
460  else:
461  print("%-6s %-35s "% (wf.numId, stepNames))
462  break
463  print('')
464  for i,n in enumerate(N):
465  if n: print(n,'workflows with',i+1,'steps')
466 
467  return
468 
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
#define str(s)

◆ updateDB()

def MatrixReader.MatrixReader.updateDB (   self)

Definition at line 525 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

525  def updateDB(self):
526 
527  import pickle
528  pickle.dump(self.workFlows, open('theMatrix.pkl', 'w') )
529 
530  return
531 
532 

◆ workFlowsByLocation()

def MatrixReader.MatrixReader.workFlowsByLocation (   self,
  cafVeto = True 
)

Definition at line 416 of file MatrixReader.py.

References MatrixReader.MatrixReader.workFlows.

Referenced by MatrixReader.MatrixReader.showWorkFlows().

416  def workFlowsByLocation(self, cafVeto=True):
417  # Check if we are on CAF
418  onCAF = False
419  if 'cms/caf/cms' in os.environ['CMS_PATH']:
420  onCAF = True
421 
422  workflows = []
423  for workflow in self.workFlows:
424  if isinstance(workflow.cmds[0], InputInfo):
425  if cafVeto and (workflow.cmds[0].location == 'CAF' and not onCAF):
426  continue
427  workflows.append(workflow)
428 
429  return workflows
430 

Member Data Documentation

◆ addCommand

MatrixReader.MatrixReader.addCommand

Definition at line 25 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

◆ apply

◆ commandLineWf

MatrixReader.MatrixReader.commandLineWf

Definition at line 27 of file MatrixReader.py.

◆ files

◆ filesDefault

MatrixReader.MatrixReader.filesDefault

◆ filesPrefMap

MatrixReader.MatrixReader.filesPrefMap

◆ nameList

MatrixReader.MatrixReader.nameList

Definition at line 41 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.createWorkFlows().

◆ noRun

MatrixReader.MatrixReader.noRun

Definition at line 30 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

◆ overWrite

MatrixReader.MatrixReader.overWrite

Definition at line 28 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

◆ relvalModule

MatrixReader.MatrixReader.relvalModule

Definition at line 96 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

◆ revertDqmio

MatrixReader.MatrixReader.revertDqmio

maybe we want too level deep input

Definition at line 24 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix(), and MatrixReader.MatrixReader.showRaw().

◆ what

MatrixReader.MatrixReader.what

◆ wm

MatrixReader.MatrixReader.wm

Definition at line 23 of file MatrixReader.py.

Referenced by MatrixReader.MatrixReader.readMatrix().

◆ workFlows

◆ workFlowSteps

MatrixReader.MatrixReader.workFlowSteps