CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
Classes | Functions | Variables
edmStreamStallGrapher Namespace Reference

Classes

class  Point
 
class  Stack
 
class  StreamInfoElement
 

Functions

def adjacentDiff
 
def chooseParser
 
def consolidateContiguousBlocks
 
def createAsciiImage
 
def createPDFImage
 
def findStalledModules
 
def getTime
 
def mergeContiguousBlocks
 
def parseStallMonitorOutput
 
def parseTracerOutput
 
def printHelp
 
def printStalledModulesInOrder
 
def readLogFile
 
def reduceSortedPoints
 

Variables

string action = 'store_true'
 
tuple args = parser.parse_args()
 
string const = 'stall.pdf'
 
string dest = 'graph'
 
 doGraphic = False
 
tuple extension = pdfFile.split('.')
 
string help = 'log file to process'
 
 inputFile = args.filename
 
string kSourceDelayedRead = "sourceDelayedRead"
 
string kSourceFindEvent = "sourceFindEvent"
 
string metavar = "'stall.pdf'"
 
string nargs = '?'
 
tuple parser
 
 pdfFile = args.graph
 
 shownStacks = args.stack
 
string stackType = 'stack'
 
tuple stalledModules = findStalledModules(processingSteps, numStreams)
 
tuple supported_filetypes = plt.figure()
 
tuple type = argparse.FileType('r')
 

Function Documentation

def edmStreamStallGrapher.adjacentDiff (   pairLists)

Definition at line 341 of file edmStreamStallGrapher.py.

Referenced by createPDFImage().

341 
342 def adjacentDiff(*pairLists):
343  points = []
344  for pairList in pairLists:
345  points += [Point(x[0], 1) for x in pairList if x[1] != 0]
346  points += [Point(sum(x),-1) for x in pairList if x[1] != 0]
347  points.sort(key=attrgetter('x'))
348  return points
def edmStreamStallGrapher.chooseParser (   inputFile)

Definition at line 187 of file edmStreamStallGrapher.py.

References cmsRelvalreport.exit.

Referenced by readLogFile().

187 
188 def chooseParser(inputFile):
189  firstLine = inputFile.readline().rstrip()
190  inputFile.seek(0) # Rewind back to beginning
191 
192  if firstLine.find("# Step") != -1:
193  print "> ... Parsing StallMonitor output."
194  return parseStallMonitorOutput
195  elif firstLine.find("++") != -1:
196  global kTracerInput
197  kTracerInput = True
198  print "> ... Parsing Tracer output."
199  return parseTracerOutput
200  else:
201  inputFile.close()
202  print "Unknown input format."
203  exit(1)
204 
#----------------------------------------------
def edmStreamStallGrapher.consolidateContiguousBlocks (   numStreams,
  streamInfo 
)

Definition at line 379 of file edmStreamStallGrapher.py.

References bitset_utilities.append(), and timeUnitHelper.unpack().

Referenced by createPDFImage().

379 
380 def consolidateContiguousBlocks(numStreams, streamInfo):
381  oldStreamInfo = streamInfo
382  streamInfo = [[] for x in xrange(numStreams)]
383 
384  for s in xrange(numStreams):
385  lastStartTime,lastTimeLength,lastColor = oldStreamInfo[s][0].unpack()
386  for info in oldStreamInfo[s][1:]:
387  start,length,color = info.unpack()
388  if color == lastColor and lastStartTime+lastTimeLength == start:
389  lastTimeLength += length
390  else:
391  streamInfo[s].append(StreamInfoElement(lastStartTime,lastTimeLength,lastColor))
392  lastStartTime = start
393  lastTimeLength = length
394  lastColor = color
395  streamInfo[s].append(StreamInfoElement(lastStartTime,lastTimeLength,lastColor))
396 
397  return streamInfo
398 
399 #----------------------------------------------
400 # Consolidating contiguous blocks with the same color drastically
401 # reduces the size of the pdf file. Same functionality as the
# previous function, but with slightly different implementation.
bitset_utilities::append
boost::dynamic_bitset append(const boost::dynamic_bitset<> &bs1, const boost::dynamic_bitset<> &bs2)
this method takes two bitsets bs1 and bs2 and returns result of bs2 appended to the end of bs1 ...
Definition: bitset_append.cc:10
def edmStreamStallGrapher.createAsciiImage (   processingSteps,
  numStreams,
  maxNameSize 
)

Definition at line 246 of file edmStreamStallGrapher.py.

246 
247 def createAsciiImage(processingSteps, numStreams, maxNameSize):
248  streamTime = [0]*numStreams
249  streamState = [0]*numStreams
250  modulesActiveOnStreams = [{} for x in xrange(numStreams)]
251  for n,trans,s,time in processingSteps:
252  modulesActiveOnStream = modulesActiveOnStreams[s]
253  waitTime = None
254  if trans == kPrefetchEnd:
255  modulesActiveOnStream[n] = time
256  continue
257  if trans == kStarted:
258  if n != kSourceFindEvent:
259  streamState[s] +=1
260  if n in modulesActiveOnStream:
261  waitTime = time - modulesActiveOnStream[n]
262  if n == kSourceDelayedRead:
263  if streamState[s] == 0:
264  waitTime = time-streamTime[s]
265  if trans == kFinished:
266  if n != kSourceDelayedRead and n!=kSourceFindEvent:
267  del modulesActiveOnStream[n]
268  if n != kSourceFindEvent:
269  streamState[s] -=1
270  streamTime[s] = time
271  states = "%-*s: " % (maxNameSize,n)
272  if trans == kStarted:
273  states +="+ "
274  if trans == kFinished:
275  states +="- "
276  for index, state in enumerate(streamState):
277  if n==kSourceFindEvent and index == s:
278  states +="* "
279  else:
280  states +=str(state)+" "
281  if waitTime is not None:
282  states += " %.2f"% (waitTime/1000.)
283  if waitTime > kStallThreshold:
284  states += " STALLED "+str(time/1000.)+" "+str(s)
285 
286  print states
287  return stalledModules
288 
#----------------------------------------------
def edmStreamStallGrapher.createPDFImage (   pdfFile,
  shownStacks,
  processingSteps,
  numStreams,
  stalledModuleInfo 
)

Definition at line 420 of file edmStreamStallGrapher.py.

References Clusterizer1DCommons.add(), adjacentDiff(), bitset_utilities.append(), consolidateContiguousBlocks(), bookConverter.max, mergeContiguousBlocks(), min(), reduceSortedPoints(), MatrixUtil.remove(), and ComparisonHelper.zip().

420 
421 def createPDFImage(pdfFile, shownStacks, processingSteps, numStreams, stalledModuleInfo):
422 
423  stalledModuleNames = set([x for x in stalledModuleInfo.iterkeys()])
424  streamInfo = [[] for x in xrange(numStreams)]
425  modulesActiveOnStreams = [{} for x in xrange(numStreams)]
426  streamLastEventEndTimes = [None]*numStreams
427  streamRunningTimes = [[] for x in xrange(numStreams)]
428  maxNumberOfConcurrentModulesOnAStream = 1
429  streamInvertedMessageFromModule = [set() for x in xrange(numStreams)]
430 
431  for n,trans,s,time in processingSteps:
432  startTime = None
433  if streamLastEventEndTimes[s] is None:
434  streamLastEventEndTimes[s]=time
435  if trans == kStarted:
436  if n == kSourceFindEvent:
437  # We assume the time from the end of the last event
438  # for a stream until the start of a new event for that
439  # stream is taken up by the source.
440  startTime = streamLastEventEndTimes[s]
441  moduleNames = set(n)
442  else:
443  activeModules = modulesActiveOnStreams[s]
444  moduleNames = set(activeModules.iterkeys())
445  if n in streamInvertedMessageFromModule[s] and kTracerInput:
446  # This is the rare case where a finished message
447  # is issued before the corresponding started.
448  streamInvertedMessageFromModule[s].remove(n)
449  continue
450  activeModules[n] = time
451  nModulesRunning = len(activeModules)
452  streamRunningTimes[s].append(Point(time,1))
453  maxNumberOfConcurrentModulesOnAStream = max(maxNumberOfConcurrentModulesOnAStream, nModulesRunning)
454  if nModulesRunning > 1:
455  # Need to create a new time span to avoid overlaps in graph.
456  startTime = min(activeModules.itervalues())
457  for k in activeModules.iterkeys():
458  activeModules[k]=time
459 
460  if trans == kFinished:
461  if n == kSourceFindEvent:
462  streamLastEventEndTimes[s]=time
463  else:
464  activeModules = modulesActiveOnStreams[s]
465  if n not in activeModules and kTracerInput:
466  # This is the rare case where a finished message
467  # is issued before the corresponding started.
468  streamInvertedMessageFromModule[s].add(n)
469  continue
470  streamRunningTimes[s].append(Point(time,-1))
471  startTime = activeModules[n]
472  moduleNames = set(activeModules.iterkeys())
473  del activeModules[n]
474  nModulesRunning = len(activeModules)
475  if nModulesRunning > 0:
476  # Reset start time for remaining modules to this time
477  # to avoid overlapping time ranges when making the plot.
478  for k in activeModules.iterkeys():
479  activeModules[k] = time
480  if startTime is not None:
481  c="green"
482  if (kSourceDelayedRead in moduleNames) or (kSourceFindEvent in moduleNames):
483  c = "orange"
484  for n in moduleNames:
485  if n in stalledModuleNames:
486  c="red"
487  break
488  streamInfo[s].append(StreamInfoElement(startTime, time-startTime, c))
489 
490  streamInfo = consolidateContiguousBlocks(numStreams, streamInfo)
491 
492  nr = 1
493  if shownStacks:
494  nr += 1
495  fig, ax = plt.subplots(nrows=nr, squeeze=True)
496  axStack = None
497  if shownStacks:
498  [xH,yH] = fig.get_size_inches()
499  fig.set_size_inches(xH,yH*4/3)
500  ax = plt.subplot2grid((4,1),(0,0), rowspan=3)
501  axStack = plt.subplot2grid((4,1),(3,0))
502 
503  ax.set_xlabel("Time (sec)")
504  ax.set_ylabel("Stream ID")
505  ax.set_ylim(-0.5,numStreams-0.5)
506  ax.yaxis.set_ticks(xrange(numStreams))
507 
508  height = 0.8/maxNumberOfConcurrentModulesOnAStream
509  allStackTimes={'green': [], 'red': [], 'blue': [], 'orange': []}
510  for i,perStreamInfo in enumerate(streamInfo):
511  times=[(x.begin/1000., x.delta/1000.) for x in perStreamInfo] # Scale from msec to sec.
512  colors=[x.color for x in perStreamInfo]
513  ax.broken_barh(times,(i-0.4,height),facecolors=colors,edgecolors=colors,linewidth=0)
514  for info in perStreamInfo:
515  allStackTimes[info.color].append((info.begin, info.delta))
516 
517  # Now superimpose the number of concurrently running modules on to the graph.
518  if maxNumberOfConcurrentModulesOnAStream > 1:
519 
520  for i,perStreamRunningTimes in enumerate(streamRunningTimes):
521  perStreamTimes = sorted(perStreamRunningTimes, key=attrgetter('x'))
522  perStreamTimes = reduceSortedPoints(perStreamTimes)
523  streamHeight = 0
524  preparedTimes = []
525  for t1,t2 in zip(perStreamTimes, perStreamTimes[1:]):
526  streamHeight += t1.y
527  if streamHeight < 2:
528  continue
529  preparedTimes.append((t1.x,t2.x-t1.x, streamHeight))
530  preparedTimes.sort(key=itemgetter(2))
531  preparedTimes = mergeContiguousBlocks(preparedTimes)
532  for nthreads, ts in groupby(preparedTimes, itemgetter(2)):
533  theTS = [(t[0],t[1]) for t in ts]
534  theTimes = [(t[0]/1000.,t[1]/1000.) for t in theTS]
535  yspan = (i-0.4+height,height*(nthreads-1))
536  ax.broken_barh(theTimes, yspan, facecolors='blue', edgecolors='blue', linewidth=0)
537  allStackTimes['blue'].extend(theTS*(nthreads-1))
538 
539  if shownStacks:
540  print "> ... Generating stack"
541  stack = Stack()
542  for color in ['green','blue','red','orange']:
543  tmp = allStackTimes[color]
544  tmp = reduceSortedPoints(adjacentDiff(tmp))
545  stack.update(color, tmp)
546 
547  for stk in reversed(stack.data):
548  color = stk[0]
549 
550  # Now arrange list in a manner that it can be grouped by the height of the block
551  height = 0
552  xs = []
553  for p1,p2 in zip(stk[1], stk[1][1:]):
554  height += p1.y
555  xs.append((p1.x, p2.x-p1.x, height))
556  xs.sort(key = itemgetter(2))
557  xs = mergeContiguousBlocks(xs)
558 
559  for height, xpairs in groupby(xs, itemgetter(2)):
560  finalxs = [(e[0]/1000.,e[1]/1000.) for e in xpairs]
561  axStack.broken_barh(finalxs, (0, height), facecolors=color, edgecolors=color, linewidth=0)
562 
563  axStack.set_xlabel("Time (sec)");
564  axStack.set_ylabel("# threads");
565  axStack.set_xlim(ax.get_xlim())
566  axStack.tick_params(top='off')
567 
568  fig.text(0.1, 0.95, "modules running", color = "green", horizontalalignment = 'left')
569  fig.text(0.5, 0.95, "stalled module running", color = "red", horizontalalignment = 'center')
570  fig.text(0.9, 0.95, "read from input", color = "orange", horizontalalignment = 'right')
571  fig.text(0.5, 0.92, "multiple modules running", color = "blue", horizontalalignment = 'center')
572  print "> ... Saving to file: '{}'".format(pdfFile)
573  plt.savefig(pdfFile)
574 
#=======================================
bitset_utilities::append
boost::dynamic_bitset append(const boost::dynamic_bitset<> &bs1, const boost::dynamic_bitset<> &bs2)
this method takes two bitsets bs1 and bs2 and returns result of bs2 appended to the end of bs1 ...
Definition: bitset_append.cc:10
Clusterizer1DCommons::add
void add(const std::vector< const T * > &source, std::vector< const T * > &dest)
Definition: Clusterizer1DCommons.h:26
ComparisonHelper::zip
OutputIterator zip(InputIterator1 first1, InputIterator1 last1, InputIterator2 first2, InputIterator2 last2, OutputIterator result, Compare comp)
Definition: L1TStage2CaloLayer1.h:38
min
T min(T a, T b)
Definition: MathUtil.h:58
def edmStreamStallGrapher.findStalledModules (   processingSteps,
  numStreams 
)

Definition at line 219 of file edmStreamStallGrapher.py.

219 
220 def findStalledModules(processingSteps, numStreams):
221  streamTime = [0]*numStreams
222  stalledModules = {}
223  modulesActiveOnStream = [{} for x in xrange(numStreams)]
224  for n,trans,s,time in processingSteps:
225  waitTime = None
226  modulesOnStream = modulesActiveOnStream[s]
227  if trans == kPrefetchEnd:
228  modulesOnStream[n] = time
229  if trans == kStarted:
230  if n in modulesOnStream:
231  waitTime = time - modulesOnStream[n]
232  if n == kSourceDelayedRead:
233  if 0 == len(modulesOnStream):
234  waitTime = time - streamTime[s]
235  if trans == kFinished:
236  if n != kSourceDelayedRead and n!=kSourceFindEvent:
237  del modulesOnStream[n]
238  streamTime[s] = time
239  if waitTime is not None:
240  if waitTime > kStallThreshold:
241  t = stalledModules.setdefault(n,[])
242  t.append(waitTime)
243  return stalledModules
244 
245 
#----------------------------------------------
def edmStreamStallGrapher.getTime (   line)

Definition at line 115 of file edmStreamStallGrapher.py.

Referenced by EcalPerEvtLaserAnalyzer.endJob(), EcalLaserAnalyzer2.endJob(), EcalLaserAnalyzer.endJob(), parseTracerOutput(), and pos::PixelFEDTestDAC.writeXMLHeader().

115 
116 def getTime(line):
117  time = line.split(" ")[1]
118  time = time.split(":")
119  time = int(time[0])*60*60+int(time[1])*60+float(time[2])
120  time = int(1000*time) # convert to milliseconds
121  return time
122 
#----------------------------------------------
def edmStreamStallGrapher.mergeContiguousBlocks (   blocks)

Definition at line 402 of file edmStreamStallGrapher.py.

Referenced by createPDFImage().

402 
403 def mergeContiguousBlocks(blocks):
404  oldBlocks = blocks
405 
406  blocks = []
407  lastStartTime,lastTimeLength,lastHeight = oldBlocks[0]
408  for start,length,height in oldBlocks[1:]:
409  if height == lastHeight and lastStartTime+lastTimeLength == start:
410  lastTimeLength += length
411  else:
412  blocks.append((lastStartTime,lastTimeLength,lastHeight))
413  lastStartTime = start
414  lastTimeLength = length
415  lastHeight = height
416  blocks.append((lastStartTime,lastTimeLength,lastHeight))
417 
418  return blocks
419 
#----------------------------------------------
def edmStreamStallGrapher.parseStallMonitorOutput (   f)

Definition at line 52 of file edmStreamStallGrapher.py.

References bookConverter.max, and split.

52 
53 def parseStallMonitorOutput(f):
54  processingSteps = []
55  numStreams = 0
56  maxNameSize = 0
57  foundEventToStartFrom = False
58  moduleNames = {}
59  for rawl in f:
60  l = rawl.strip()
61  if not l or l[0] == '#':
62  if len(l) > 5 and l[0:2] == "#M":
63  (id,name)=tuple(l[2:].split())
64  moduleNames[id] = name
65  continue
66  (step,payload) = tuple(l.split(None,1))
67  payload=payload.split()
68 
69  # Payload format is:
70  # <stream id> <..other fields..> <time since begin job>
71  stream = int(payload[0])
72  time = int(payload[-1])
73  trans = None
74 
75  numStreams = max(numStreams, stream+1)
76 
77  # 'S' = begin of event creation in source
78  # 's' = end of event creation in source
79  if step == 'S' or step == 's':
80  name = kSourceFindEvent
81  trans = kStarted
82  # The start of an event is the end of the framework part
83  if step == 's':
84  trans = kFinished
85  else:
86  # moduleID is the second payload argument for all steps below
87  moduleID = payload[1]
88 
89  # 'p' = end of module prefetching
90  # 'M' = begin of module processing
91  # 'm' = end of module processing
92  if step == 'p' or step == 'M' or step == 'm':
93  trans = kStarted
94  if step == 'p':
95  trans = kPrefetchEnd
96  elif step == 'm':
97  trans = kFinished
98  name = moduleNames[moduleID]
99 
100  # Delayed read from source
101  # 'R' = begin of delayed read from source
102  # 'r' = end of delayed read from source
103  if step == 'R' or step == 'r':
104  trans = kStarted
105  if step == 'r':
106  trans = kFinished
107  name = kSourceDelayedRead
108 
109  maxNameSize = max(maxNameSize, len(name))
110  processingSteps.append((name,trans,stream,time))
111 
112  f.close()
113  return (processingSteps,numStreams,maxNameSize)
114 
#----------------------------------------------
split
double split
Definition: MVATrainer.cc:139
def edmStreamStallGrapher.parseTracerOutput (   f)

Definition at line 123 of file edmStreamStallGrapher.py.

References getTime(), and bookConverter.max.

123 
124 def parseTracerOutput(f):
125  processingSteps = []
126  numStreams = 0
127  maxNameSize = 0
128  startTime = 0
129  for l in f:
130  if l.find("processing event :") != -1:
131  time = getTime(l)
132  if startTime == 0:
133  startTime = time
134  time = time - startTime
135  streamIndex = l.find("stream = ")
136  stream = int(l[streamIndex+9:l.find(" ",streamIndex+10)])
137  name = kSourceFindEvent
138  trans = kFinished
139  #the start of an event is the end of the framework part
140  if l.find("starting:") != -1:
141  trans = kStarted
142  processingSteps.append((name,trans,stream,time))
143  numStreams = max(numStreams, stream+1)
144  if l.find("processing event for module") != -1:
145  time = getTime(l)
146  if startTime == 0:
147  startTime = time
148  time = time - startTime
149  trans = kStarted
150  stream = 0
151  delayed = False
152  if l.find("finished:") != -1:
153  if l.find("prefetching") != -1:
154  trans = kPrefetchEnd
155  else:
156  trans = kFinished
157  else:
158  if l.find("prefetching") != -1:
159  #skip this since we don't care about prefetch starts
160  continue
161  streamIndex = l.find("stream = ")
162  stream = int( l[streamIndex+9:l.find(" ",streamIndex+10)])
163  name = l.split("'")[1]
164  maxNameSize = max(maxNameSize, len(name))
165  processingSteps.append((name,trans,stream,time))
166  numStreams = max(numStreams, stream+1)
167  if l.find("event delayed read from source") != -1:
168  time = getTime(l)
169  if startTime == 0:
170  startTime = time
171  time = time - startTime
172  trans = kStarted
173  stream = 0
174  delayed = False
175  if l.find("finished:") != -1:
176  trans = kFinished
177  streamIndex = l.find("stream = ")
178  stream = int(l[streamIndex+9:l.find(" ",streamIndex+10)])
179  name = kSourceDelayedRead
180  maxNameSize = max(maxNameSize, len(name))
181  processingSteps.append((name,trans,stream,time))
182  numStreams = max(numStreams, stream+1)
183  f.close()
184  return (processingSteps,numStreams,maxNameSize)
185 
186 
#----------------------------------------------
def edmStreamStallGrapher.printHelp ( )

Definition at line 7 of file edmStreamStallGrapher.py.

7 
8 def printHelp():
9  s = '''
10 To Use: Add the Tracer Service to the cmsRun job you want to check for
11  stream stalls. Make sure to use the 'printTimstamps' option
12  cms.Service("Tracer", printTimestamps = cms.untracked.bool(True))
13  After running the job, execute this script and pass the name of the
14  log file to the script as the only command line argument.
15 
16 To Read: The script will then print an 'ASCII art' stall graph which
17  consists of the name of the module which either started or stopped
18  running on a stream, and the number of modules running on each
19  stream at that the moment in time. If the module just started, you
20  will also see the amount of time the module spent between finishing
21  its prefetching and starting. The state of a module is represented
22  by a symbol:
23 
24  plus ("+") the stream has just finished waiting and is starting a module
25  minus ("-") the stream just finished running a module
26 
27  If a module had to wait more than 0.1 seconds, the end of the line
28  will have "STALLED". Once the first 4 events have finished
29  processing, the program prints "FINISH INIT". This is useful if one
30  wants to ignore stalled caused by startup actions, e.g. reading
31  conditions.
32 
33  Once the graph is completed, the program outputs the list of modules
34  which had the greatest total stall times. The list is sorted by
35  total stall time and written in descending order. In addition, the
36  list of all stall times for the module is given.'''
37  return s
38 
39 
40 kStallThreshold=100 #in milliseconds
41 kTracerInput=False
42 
43 #Stream states
44 kStarted=0
45 kFinished=1
46 kPrefetchEnd=2
47 
#Special names
def edmStreamStallGrapher.printStalledModulesInOrder (   stalledModules)

Definition at line 289 of file edmStreamStallGrapher.py.

References join(), and bookConverter.max.

289 
290 def printStalledModulesInOrder(stalledModules):
291  priorities = []
292  maxNameSize = 0
293  for name,t in stalledModules.iteritems():
294  maxNameSize = max(maxNameSize, len(name))
295  t.sort(reverse=True)
296  priorities.append((name,sum(t),t))
297 
298  def sumSort(i,j):
299  return cmp(i[1],j[1])
300  priorities.sort(cmp=sumSort, reverse=True)
301 
302  nameColumn = "Stalled Module"
303  maxNameSize = max(maxNameSize, len(nameColumn))
304 
305  stallColumn = "Tot Stall Time"
306  stallColumnLength = len(stallColumn)
307 
308  print "%-*s" % (maxNameSize, nameColumn), "%-*s"%(stallColumnLength,stallColumn), " Stall Times"
309  for n,s,t in priorities:
310  paddedName = "%-*s:" % (maxNameSize,n)
311  print paddedName, "%-*.2f"%(stallColumnLength,s/1000.), ", ".join([ "%.2f"%(x/1000.) for x in t])
312 
#--------------------------------------------------------
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
def edmStreamStallGrapher.readLogFile (   inputFile)

Definition at line 205 of file edmStreamStallGrapher.py.

References chooseParser(), and tools.parseInput().

205 
206 def readLogFile(inputFile):
207  parseInput = chooseParser(inputFile)
208  return parseInput(inputFile)
209 
210 #----------------------------------------------
211 # Patterns:
212 #
213 # source: The source just records how long it was spent doing work,
214 # not how long it was stalled. We can get a lower bound on the stall
215 # time by measuring the time the stream was doing no work up till
216 # the source was run.
217 # modules: The time between prefetch finished and 'start processing' is
218 # the time it took to acquire any resources
#
tools.parseInput
def parseInput
Definition: tools.py:122
def edmStreamStallGrapher.reduceSortedPoints (   ps)

Definition at line 325 of file edmStreamStallGrapher.py.

Referenced by createPDFImage(), and edmStreamStallGrapher.Stack.update().

325 
326 def reduceSortedPoints(ps):
327  if len(ps) < 2:
328  return ps
329  reducedPoints = []
330  tmp = ps[0]
331  for p in ps[1:]:
332  if tmp.x == p.x:
333  tmp.y += p.y
334  else:
335  reducedPoints.append(tmp)
336  tmp = p
337  reducedPoints.append(tmp)
338  reducedPoints = [p for p in reducedPoints if p.y != 0]
339  return reducedPoints
340 
# -------------------------------------------

Variable Documentation

string edmStreamStallGrapher.action = 'store_true'

Definition at line 596 of file edmStreamStallGrapher.py.

tuple edmStreamStallGrapher.args = parser.parse_args()

Definition at line 599 of file edmStreamStallGrapher.py.

string edmStreamStallGrapher.const = 'stall.pdf'

Definition at line 590 of file edmStreamStallGrapher.py.

Referenced by TkGluedMeasurementDet::HitCollectorForSimpleHits.addProjected(), TkGluedMeasurementDet::HitCollectorForFastMeasurements.addProjected(), cms.Adler32(), SiStripGainFromData.algoBeginJob(), ecaldqm::RecoSummaryTask.analyze(), ecaldqm::ClusterTask.analyze(), PixelLumiDQM.analyze(), B2GDQM.analyzeJets(), PFElecTkProducer.applySelection(), edm::SoATuple< edm::EDConsumerBase::TokenLookupInfo, bool, edm::EDConsumerBase::LabelPlacement, edm::KindOfType >.begin(), RPCConeBuilder.buildCones(), TrackProducerAlgorithm< reco::Track >.buildTrack(), GroupedCkfTrajectoryBuilder.buildTrajectories(), edm.checkClassDictionaries(), edm.convert_handle(), PFCTRecHitProducer.createHcalRecHit(), BareRootProductGetter.createNewBuffer(), edm::ProductResolverBase::Resolution.data(), edm::reftobase::RefVectorHolder< REFV >::const_iterator_imp_specific.dc(), PATTauDiscriminationAgainstElectronMVA6.discriminate(), edm::SoATuple< edm::EDConsumerBase::TokenLookupInfo, bool, edm::EDConsumerBase::LabelPlacement, edm::KindOfType >.end(), edm::EventForOutput.eventPrincipal(), edm::Event.eventPrincipal(), SiStripDigitizer.finalizeEvent(), PFElecTkProducer.FindPfRef(), edm::SoATuple< edm::EDConsumerBase::TokenLookupInfo, bool, edm::EDConsumerBase::LabelPlacement, edm::KindOfType >.get(), fwlite::LuminosityBlockBase.getByLabelImpl(), fwlite::RunBase.getByLabelImpl(), fwlite::EventBase.getByLabelImpl(), LTCDigi.GetEventNumberFromBuffer(), edm::eventsetup::EventSetupRecord.getImplementation(), BareRootProductGetter.getIt(), QuickTrackAssociatorByHitsImpl.getMatchedClusters(), edm.getProduct(), edm::refcore.getProduct_(), edm::refcore.getProductWithCoreFromRef_(), edm::reftobase::RefHolderBase.getPtr(), edm::refitem::GetRefPtrImpl< C, T, F, KEY >.getRefPtr_(), edm::refitem::GetRefPtrImpl< C, T, F, unsigned int >.getRefPtr_(), LTCDigi.GetRunNumberFromBuffer(), fwlite::DataGetterHelper.getThinnedAssociation(), BareRootProductGetter.getThinnedAssociation(), edm::EventPrincipal.getThinnedAssociation(), edm::refcore.getThinnedProduct_(), CosmicHitPairGeneratorFromLayerPair.hitPairs(), RectangularEtaPhiTrackingRegion.hits(), PFHFRecHitCreator.importRecHits(), CosmicLayerTriplets.init(), SiStripDetVOffFakeBuilder.initialize(), SiStripDigitizer.initializeEvent(), edm::DataMixingSiStripMCDigiWorker.initializeEvent(), edm::reftobase::Holder< T, REF >.isEqualTo(), edm::reftobase::IndirectHolder< T >.isEqualTo(), edm::Wrapper< T >.isProductEqual_(), PFElecTkProducer.isSameEgSC(), PFElecTkProducer.isSharingEcalEnergyWithEgSC(), LTCDigi.LTCDigi(), edm::LuminosityBlockForOutput.luminosityBlockPrincipal(), edm::LuminosityBlock.luminosityBlockPrincipal(), SiStripHashedDetIdESModule.make(), edm::WorkerMaker< T >.makeWorker(), edm::Wrapper< T >.mergeProduct_(), AntiElectronIDMVA6.MVAValue(), edm::Ptr< T >.operator*(), edm::Ptr< T >.operator->(), edm::Guid.operator=(), edm::Guid.operator==(), edm.pointerToBase(), edm::TypeWithDict.pointerToBaseType(), edm::UnscheduledProductResolver.prefetchAsync_(), edm::ProductSelector.print(), DeDxDiscriminatorLearner.processHit(), StandaloneTrackMonitor.processHit(), DeDxHitInfoProducer.processHit(), HSCPDeDxInfoProducer.processHit(), DeDxEstimatorProducer.processHit(), DigiSimLinkProducer.produce(), PFTrackProducer.produce(), HSCPDeDxInfoProducer.produce(), PFElecTkProducer.produce(), edm::OrphanHandle< T >.product(), edm::TestHandle< T >.product(), edm::Handle< T >.product(), edm::refcoreimpl.productGetter(), edm::refcoreimpl.productPtr(), edm::reftobase::RefVectorHolder< REFV >.push_back(), edm::PtrVector< reco::FFTJPTJet >.push_back(), HepMCFileReader.rdstate(), edm::Ref< C, T, F >.Ref(), PFElecTkProducer.resolveGsfTracks(), edm::UnscheduledProductResolver.resolveProduct_(), SimpleDAFHitCollector.rightdimension(), ecaldqm::LaserWriter.run(), ecaldqm::PedestalWriter.run(), ecaldqm::TestPulseWriter.run(), ecaldqm::LedWriter.run(), run_script(), edm::RunForOutput.runPrincipal(), edm::Run.runPrincipal(), edm::refcoreimpl.setCacheIsProductGetter(), RPCSimSimple.simulateNoise(), RPCSimParam.simulateNoise(), RPCSimModelTiming.simulateNoise(), RPCSimAverageNoise.simulateNoise(), RPCSimAverageNoiseEff.simulateNoise(), RPCSimAverage.simulateNoise(), RPCSimAverageNoiseEffCls.simulateNoise(), RPCSimAsymmetricCls.simulateNoise(), TkStackMeasurementDet.specificGeomDet(), TkGluedMeasurementDet.specificGeomDet(), Traj2TrackHits.split(), reco::PFCandidate.storeRefInfo(), edm::refcore.tryToGetProductWithCoreFromRef_(), L1GtfeExtWord.unpack(), L1GtfeWord.unpack(), L1TcsWord.unpack(), L1GtFdlWord.unpack(), and fwlite::DataGetterHelper.wrapperBasePtr().

string edmStreamStallGrapher.dest = 'graph'

Definition at line 591 of file edmStreamStallGrapher.py.

edmStreamStallGrapher.doGraphic = False

Definition at line 606 of file edmStreamStallGrapher.py.

tuple edmStreamStallGrapher.extension = pdfFile.split('.')

Definition at line 619 of file edmStreamStallGrapher.py.

Referenced by dataFileExtension(), ExternalLHEAsciiDumper.endRun(), and Vispa.Main.Application.Application.recentFilesFromPlugin().

string edmStreamStallGrapher.help = 'log file to process'

Definition at line 586 of file edmStreamStallGrapher.py.

edmStreamStallGrapher.inputFile = args.filename

Definition at line 602 of file edmStreamStallGrapher.py.

Referenced by GBRForestWriter.analyze(), edm::JobReport::JobReportImpl.associateInputLumiSection(), edm::JobReport::JobReportImpl.associateInputRun(), RecoTauDiscriminantCutMultiplexer.beginEvent(), PATTauDiscriminantCutMultiplexer.beginEvent(), TauDQMFileLoader::cfgEntryFileSet.cfgEntryFileSet(), LaserSorter.closeOutStream(), cond::auth::DecodingKey.createFromInputFile(), EcalDQMStatusWriter.EcalDQMStatusWriter(), EffectiveAreas.EffectiveAreas(), TauDQMFileLoader.endRun(), edm::JobReport::JobReportImpl.flushFiles(), matrixSaver.getConcreteMatrixVector(), WatcherStreamFileReader.getHeader(), edm::JobReport::JobReportImpl.getInputFileForToken(), matrixSaver.getMatrix(), matrixSaver.getMatrixVector(), WatcherStreamFileReader.getNextEvent(), HCovarianceVSParts.HCovarianceVSParts(), PFMETAlgorithmMVA.loadMVAfromFile(), main(), makePlots(), MaterialEffects.MaterialEffects(), WatcherStreamFileReader.newHeader(), L1GtVhdlTemplateFile.open(), DCCTBDataParser.parseFile(), cond::XMLAuthenticationService::XMLAuthenticationService.processFile(), cond::FileReader.read(), StoreEcalCondition.readEcalADCToGeVConstantFromFile(), StoreEcalCondition.readEcalGainRatiosFromFile(), StoreEcalCondition.readEcalIntercalibConstantsFromFile(), StoreEcalCondition.readEcalIntercalibConstantsMCFromFile(), StoreEcalCondition.readEcalTBWeightsFromFile(), StoreEcalCondition.readEcalWeightXtalGroupsFromFile(), popcon::EcalTPGWeightIdMapHandler.readFromFile(), popcon::EcalTPGLutIdMapHandler.readFromFile(), popcon::EcalTPGFineGrainEBIdMapHandler.readFromFile(), popcon::EcalTPGLinConstHandler.readFromFile(), popcon::EcalTPGPhysicsConstHandler.readFromFile(), popcon::EcalTPGBadXTHandler.readFromFile(), popcon::EcalTPGBadStripHandler.readFromFile(), popcon::EcalTPGBadTTHandler.readFromFile(), popcon::EcalTPGFineGrainEBGroupHandler.readFromFile(), popcon::EcalTPGWeightGroupHandler.readFromFile(), popcon::EcalTPGFineGrainTowerEEHandler.readFromFile(), popcon::EcalTPGFineGrainStripEEHandler.readFromFile(), popcon::EcalTPGSlidingWindowHandler.readFromFile(), popcon::EcalTPGPedestalsHandler.readFromFile(), popcon::EcalTPGLutGroupHandler.readFromFile(), popcon::EcalTPGSpikeThresholdHandler.readFromFile(), jsoncollector::FileIO.readStringFromFile(), RPCRecHitProducer.RPCRecHitProducer(), and matrixSaver.touch().

string edmStreamStallGrapher.kSourceDelayedRead = "sourceDelayedRead"

Definition at line 49 of file edmStreamStallGrapher.py.

string edmStreamStallGrapher.kSourceFindEvent = "sourceFindEvent"

Definition at line 48 of file edmStreamStallGrapher.py.

string edmStreamStallGrapher.metavar = "'stall.pdf'"

Definition at line 589 of file edmStreamStallGrapher.py.

string edmStreamStallGrapher.nargs = '?'

Definition at line 588 of file edmStreamStallGrapher.py.

tuple edmStreamStallGrapher.parser
Initial value:
1 = argparse.ArgumentParser(description='Convert a cmsRun log with Tracer info into a stream stall graph.',
2  formatter_class=argparse.RawDescriptionHelpFormatter,
3  epilog=printHelp())

Definition at line 581 of file edmStreamStallGrapher.py.

edmStreamStallGrapher.pdfFile = args.graph

Definition at line 603 of file edmStreamStallGrapher.py.

edmStreamStallGrapher.shownStacks = args.stack

Definition at line 604 of file edmStreamStallGrapher.py.

string edmStreamStallGrapher.stackType = 'stack'

Definition at line 349 of file edmStreamStallGrapher.py.

tuple edmStreamStallGrapher.stalledModules = findStalledModules(processingSteps, numStreams)

Definition at line 635 of file edmStreamStallGrapher.py.

tuple edmStreamStallGrapher.supported_filetypes = plt.figure()

Definition at line 620 of file edmStreamStallGrapher.py.

tuple edmStreamStallGrapher.type = argparse.FileType('r')

Definition at line 585 of file edmStreamStallGrapher.py.

Generated for CMSSW Reference Manual by doxygen 1.8.5