CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
Functions | Variables
cmsPerfStripChart Namespace Reference

Functions

def get_yaxis_range
 
def operate
 

Variables

string action = 'store'
 
string default = ''
 
string dest = 'timelog'
 
tuple dict = json.load(json_db)
 
string format = "\n { \"strips\" :\n"
 Some error handling for the usage. More...
 
string help = 'input file TIMELOG, the output of cmsTiming_parser.py'
 
tuple json_db = open(options.json_f, "r+")
 
string metavar = 'TIMELOG'
 
tuple parser = optparse.OptionParser(usage)
 
tuple script_name = os.path.basename(__file__)
 Definition of command usage. More...
 
string usage = script_name+' <options> -t TIMELOG -m MEMLOG'
 

Function Documentation

def cmsPerfStripChart.get_yaxis_range (   list) 
Given a list of dictionaries, where each dict holds the information
about an IB, this function returns a tuple (low, high) with the lowest
and the highest value of y-axis, respectively.

Definition at line 8 of file cmsPerfStripChart.py.

References bookConverter.max, and min().

Referenced by operate().

8 
9 def get_yaxis_range(list):
10  """
11  Given a list of dictionaries, where each dict holds the information
12  about an IB, this function returns a tuple (low, high) with the lowest
13  and the highest value of y-axis, respectively.
14  """
15  low, high = sys.maxint, -1
16  for node in list:
17  low = min((node['average'] - node['error']), low)
18  high = max((node['average'] + node['error']), high)
19  return (low, high)
20 
# Main operation function
min
T min(T a, T b)
Definition: MathUtil.h:58
def cmsPerfStripChart.operate (   timelog,
  memlog,
  json_f,
  num 
) 
Main operation of the script (i.e. json db update, histograms' creation)
with respect to the specifications of all the files & formats concerned.

Definition at line 21 of file cmsPerfStripChart.py.

References python.multivaluedict.append(), get_yaxis_range(), and split.

Referenced by edm::EventProcessor.beginJob(), edm::SubProcess.beginJob(), edm::SubProcess.clearCounters(), edm::SubProcess.closeOutputFiles(), edm::serviceregistry::ServicesManager.createServices(), edm::SubProcess.doBeginLuminosityBlock(), edm::SubProcess.doBeginRun(), edm::SubProcess.doBeginStream(), edm::SubProcess.doEndLuminosityBlock(), edm::SubProcess.doEndRun(), edm::SubProcess.doEndStream(), edm::SubProcess.doEvent(), edm::SubProcess.doStreamBeginLuminosityBlock(), edm::SubProcess.doStreamBeginRun(), edm::SubProcess.doStreamEndLuminosityBlock(), edm::SubProcess.doStreamEndRun(), evf::EvFBuildingThrottle.dowork(), edm::SubProcess.enableEndPaths(), edm::EventProcessor.endJob(), edm::SubProcess.endJob(), edm::SubProcess.endPathsEnabled(), spf::SherpackFetcher.FnFileGet(), edm::EventProcessor.forkProcess(), edm::SubProcess.getTriggerReport(), edm::EventProcessor.init(), main(), edm::SubProcess.openNewOutputFilesIfNeeded(), edm::SubProcess.openOutputFiles(), edm::SubProcess.postForkReacquireResources(), edm::SubProcess.preForkReleaseResources(), edm::EventProcessor.processEventsForStreamAsync(), edm::SubProcess.respondToCloseInputFile(), edm::SubProcess.respondToOpenInputFile(), cond::Utilities.run(), edm::EventProcessor.runToCompletion(), edm::SubProcess.shouldWeCloseOutput(), edm::SubProcess.SubProcess(), edm::SubProcess.terminate(), edm::SubProcess.totalEventsFailed(), edm::SubProcess.totalEventsPassed(), edm::SubProcess.writeLumi(), and edm::SubProcess.writeRun().

21 
22 def operate(timelog, memlog, json_f, num):
23  """
24  Main operation of the script (i.e. json db update, histograms' creation)
25  with respect to the specifications of all the files & formats concerned.
26  """
27  import re
28  import commands
29  import ROOT
30  from datetime import datetime
31 
32  script_name=os.path.basename(__file__)
33 
34  # Open files and store the lines.
35  timefile=open(timelog, 'r')
36  timelog_lines=timefile.readlines()
37  timefile.close()
38 
39  memfile=open(memlog, 'r')
40  memlog_lines=memfile.readlines()
41  memfile.close()
42 
43  # Get average, uncertainty of average and maximum rss.
44  max_rss=average=error=' '
45  i=0
46  while i<len(timelog_lines):
47  line=timelog_lines[i]
48  if 'Uncertainty of Average Time' in line:
49  line=line[:-1]
50  line_list=line.split(' ')
51  average=line_list[5]
52  error=line_list[7]
53  i+=1
54  i=0
55  while i<len(memlog_lines):
56  line=memlog_lines[i]
57  if 'Maximum rss' in line:
58  line=line[:-1]
59  line_list=line.split(' ')
60  max_rss=line_list[3]
61  break
62  i+=1
63 
64  # Get Integration Build's identifier
65  IB=os.path.basename(commands.getoutput("echo $CMSSW_BASE"))
66 
67  # Check if log files were parsed properly...
68  # and if the IB is valid using regular expressions.
69  try:
70  # regex for a float
71  regex="^\d+\.?\d*$"
72  if average == ' ' or re.match(regex, average) is None:
73  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
74  'Check if Average Time is defined correctly.')
75  if error == ' ' or re.match(regex, error) is None:
76  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
77  'Check if Uncertainty of Average Time is defined correctly.')
78  if max_rss == ' ' or re.match(regex, max_rss) is None:
79  raise RuntimeError('Could not parse \"' + memlog + '\" properly. ' +\
80  ' Check if Maximum rss is defined correct.')
81 
82  # regex for dates 'YYYY-MM-DD-HHMM'
83  regex = '(19|20|21)\d\d-(0[1-9]|1[012])-(0[1-9]|[12]'+\
84  '[0-9]|3[01])-([01][0-9]|2[0-4])([0-5][0-9])$'
85  if re.search(regex, IB) is None:
86  raise RuntimeError('Not a valid IB. Valid IB: ' +\
87  '[CMSSW_X_X_X_YYYY-MM-DD-HHMM]')
88  except Exception, err:
89  sys.stderr.write(script_name + ': Error: ' + str(err) + '\n')
90  return 2
91 
92  # Open for getting the data.
93  json_db=open(json_f, "r")
94  dict=json.load(json_db)
95  json_db.close()
96 
97  # Get the data to be stored and check if already exists.
98  ib_list=IB.split('_')
99  cmsrelease=ib_list[0] + '_' + ib_list[1] +\
100  '_' + ib_list[2] + '_' + ib_list[3]
101  data={"IB" : ib_list[4], "average" : float(average), "error" : float(error), "max_rss" : float(max_rss)}
102 
103  if data in dict["strips"]:
104  sys.stderr.write(script_name + ": Warning: Entry already exists " +\
105  "in json file and will not be stored! " +\
106  "Only the strip charts will be created.\n")
107  else:
108  dict["strips"].append(data)
109  print 'Storing entry to \"' + json_f +\
110  '\" file with attribute values:\n' +\
111  'IB=' + IB + '\naverage=' + average +\
112  '\nUncertainty of average=' + error +'\nmax_rss=' + max_rss
113  # Store the data in json file.
114  json_db = open(json_f, "w+")
115  json.dump(dict, json_db, indent=2)
116  json_db.close()
117  print 'File "' + json_f + '" was updated successfully!'
118 
119  # Change to datetime type (helpful for sorting).
120  for record in dict["strips"]:
121  time_list = record['IB'].split('-')
122  d = datetime(int(time_list[0]), int(time_list[1]),
123  int(time_list[2]), int(time_list[3][0:2]),
124  int(time_list[3][2:]))
125  record['IB'] = d
126 
127  # Sort the list.
128  list = sorted(dict["strips"], key=lambda k : k['IB'], reverse=True)
129 
130  # Check if there are NUM entries.
131  if num > len(list):
132  new_num = len(list)
133  sys.stderr.write(script_name + ': Warning: There are less than ' +\
134  str(num) + ' entries in json file. Changed number to ' +\
135  str(new_num) + '.\n')
136  num = new_num
137 
138  # The histograms.
139  ROOT.gROOT.SetStyle("Plain")
140  outdir='.'
141 
142  # Save in file
143  rootfilename=outdir + '/histograms.root'
144  myfile=ROOT.TFile(rootfilename, 'RECREATE')
145 
146  # Average time histogram.
147  histo1=ROOT.TH1F("AveCPU per IB", "Ave CPU per IB", num, 0., num)
148  histo1.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
149  histo1.SetName('avecpu_histo')
150 
151  # Maximum rss histogram.
152  histo2=ROOT.TH1F("Max rrs per IB", "Max rss per IB", num, 0., num)
153  histo2.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
154  histo2.SetName('maxrss_histo')
155 
156  # Fill in the histograms
157  for i in range(num):
158  datime = list[i]['IB'].__format__('%Y-%b-%d %H:%M')
159  average = list[i]['average']
160  max_rss = list[i]['max_rss']
161  error = list[i]['error']
162 
163  histo1.GetXaxis().SetBinLabel(num-i, datime)
164  histo1.SetBinContent(num-i, average)
165  histo1.SetBinError(num-i, error)
166  histo2.GetXaxis().SetBinLabel(num-i, datime)
167  histo2.SetBinContent(num-i, max_rss)
168 
169  histo1.SetStats(0)
170  histo1.GetYaxis().SetTitle("Average CPU time")
171  histo1.GetYaxis().SetTitleOffset(1.8)
172  histo1.GetXaxis().SetTitle("Integration Build")
173  histo1.GetXaxis().SetTitleOffset(4.)
174  histo1.GetXaxis().CenterTitle()
175  histo1.GetXaxis().LabelsOption('v')
176  # Histo1 - Set limits on the Y-axis
177  min, max = get_yaxis_range(list)
178  interval = max - min
179  # ...get a bit more space
180  min = min-interval*0.1
181  max = max+interval*0.1
182  histo1.GetYaxis().SetRangeUser(min, max)
183 
184  histo2.SetStats(0)
185  histo2.GetYaxis().SetTitle("Maximum rss")
186  histo2.GetYaxis().SetTitleOffset(1.8)
187  histo2.GetXaxis().SetTitle("Integration Build")
188  histo2.GetXaxis().SetTitleOffset(4.)
189  histo2.GetXaxis().CenterTitle()
190  histo2.GetXaxis().LabelsOption('v')
191 
192  # Draw and save!
193 
194  ave_canvas = ROOT.TCanvas(cmsrelease + '_average_canvas')
195  ave_canvas.SetGridy()
196  ave_canvas.SetBottomMargin(0.28)
197  ave_canvas.SetLeftMargin(0.18)
198  ave_canvas.cd()
199  # Histo1 - draw line
200  histo1.SetLineColor(2)
201  histo1.SetLineWidth(2)
202  histo1.DrawCopy("HISTO L")
203  # Histo1 - draw errors and markers
204  histo1.SetLineColor(1)
205  histo1.SetLineStyle(2)
206  histo1.SetLineWidth(1)
207  histo1.SetMarkerStyle(8)
208  histo1.SetMarkerSize(.6)
209  histo1.SetMarkerColor(1)
210  histo1.Draw("E1P SAME")
211  ROOT.gStyle.SetErrorX(0)
212  ave_canvas.Print(outdir + "/average_cpu_histo.png","png")
213 
214  rss_canvas = ROOT.TCanvas(cmsrelease + '_maxrss_canvas')
215  rss_canvas.SetGridy()
216  rss_canvas.SetBottomMargin(0.28)
217  rss_canvas.SetLeftMargin(0.18)
218  rss_canvas.cd()
219  # Histo2 - draw line
220  histo2.SetLineColor(2)
221  histo2.SetLineWidth(2)
222  histo2.DrawCopy("L")
223  # Histo2 - draw markers
224  histo2.SetMarkerStyle(8)
225  histo2.SetMarkerSize(.6)
226  histo2.SetMarkerColor(1)
227  histo2.Draw("P SAME")
228  rss_canvas.Print(outdir + "/maximum_rss_histo.png","png")
229 
230  # write them on file
231  histo1.Write()
232  ave_canvas.Write()
233  histo2.Write()
234  rss_canvas.Write()
235 
split
double split
Definition: MVATrainer.cc:139

Variable Documentation

string cmsPerfStripChart.action = 'store'

Definition at line 249 of file cmsPerfStripChart.py.

string cmsPerfStripChart.default = ''

Definition at line 251 of file cmsPerfStripChart.py.

string cmsPerfStripChart.dest = 'timelog'

Definition at line 250 of file cmsPerfStripChart.py.

tuple cmsPerfStripChart.dict = json.load(json_db)

Definition at line 307 of file cmsPerfStripChart.py.

string cmsPerfStripChart.format = "\n { \"strips\" :\n"

Some error handling for the usage.

# Validity of .json file-database. #

Definition at line 290 of file cmsPerfStripChart.py.

Referenced by argparse.HelpFormatter._metavar_formatter(), python.rootplot.argparse.HelpFormatter._metavar_formatter(), dqm::RamdiskMonitor.bookHistograms(), TriggerRatesMonitor.bookHistograms(), sistrip::MeasureLA.calibration_key(), ComphepSingletopFilter.endJob(), SummaryPlotFactory< CommissioningAnalysis * >.fill(), EmDQMReco::FourVectorMonitorElements.FourVectorMonitorElements(), MatacqProducer.getMatacqFile(), dqm::RamdiskMonitor.getRunTimestamp(), DQMFileSaver.globalEndRun(), edm::service::IgProfService.makeDump(), IgProfModule.makeDump(), dqmservices::DQMFileIterator.monUpdateLumi(), dqmservices::DQMMonitoringService.outputLumiUpdate(), ListGroups.overlayEtaReferences(), dqmservices::JsonWritingTimeoutPoolOutputModule.physicalAndLogicalNameForNewFile(), FastTimerService.postGlobalEndRun(), FastTimerService.preallocate(), ThroughputService.preStreamBeginRun(), FastTimerService.preStreamBeginRun(), dqmservices::DQMFileIterator.reset(), MatacqProducer.runSubDir(), DQMFileSaver.saveForFilterUnit(), FWTriggerTableView.saveImageTo(), FWTableView.saveImageTo(), dqm::DQMFileSaverPB.saveLumi(), TrackingMaterialAnalyser.saveParameters(), sistrip::EnsembleCalibrationLA.write_ensembles_text(), edm::writeParameterValue.writeValue(), edm::writeParameterValue.writeValueInVector(), edm::writeParameterValue.writeValueInVectorWithSpace(), and edm::writeParameterValue.writeVector().

string cmsPerfStripChart.help = 'input file TIMELOG, the output of cmsTiming_parser.py'

Definition at line 253 of file cmsPerfStripChart.py.

tuple cmsPerfStripChart.json_db = open(options.json_f, "r+")

Definition at line 297 of file cmsPerfStripChart.py.

string cmsPerfStripChart.metavar = 'TIMELOG'

Definition at line 252 of file cmsPerfStripChart.py.

tuple cmsPerfStripChart.parser = optparse.OptionParser(usage)

Definition at line 247 of file cmsPerfStripChart.py.

tuple cmsPerfStripChart.script_name = os.path.basename(__file__)

Definition of command usage.

#

Definition at line 245 of file cmsPerfStripChart.py.

string cmsPerfStripChart.usage = script_name+' <options> -t TIMELOG -m MEMLOG'

Definition at line 246 of file cmsPerfStripChart.py.

Generated for CMSSW Reference Manual by doxygen 1.8.5