CMS 3D CMS Logo
Functions | Variables
cmsPerfStripChart Namespace Reference

Functions

def get_yaxis_range (list)
 
def operate (timelog, memlog, json_f, num)
 

Variables

 action
 
 args
 
 default
 
 dest
 
 dict
 
 format
 Some error handling for the usage. More...
 
 help
 
 json_db
 
 metavar
 
 options
 
 parser
 
 script_name
 Definition of command usage. More...
 
 type
 
 usage
 

Function Documentation

def cmsPerfStripChart.get_yaxis_range (   list) 
Given a list of dictionaries, where each dict holds the information
about an IB, this function returns a tuple (low, high) with the lowest
and the highest value of y-axis, respectively.

Definition at line 9 of file cmsPerfStripChart.py.

References SiStripPI.max, and min().

Referenced by operate().

9 def get_yaxis_range(list):
10  """
11  Given a list of dictionaries, where each dict holds the information
12  about an IB, this function returns a tuple (low, high) with the lowest
13  and the highest value of y-axis, respectively.
14  """
15  low, high = sys.maxsize, -1
16  for node in list:
17  low = min((node['average'] - node['error']), low)
18  high = max((node['average'] + node['error']), high)
19  return (low, high)
20 
21 # Main operation function
min
T min(T a, T b)
Definition: MathUtil.h:58
cmsPerfStripChart.get_yaxis_range
def get_yaxis_range(list)
Definition: cmsPerfStripChart.py:9
def cmsPerfStripChart.operate (   timelog,
  memlog,
  json_f,
  num 
) 
Main operation of the script (i.e. json db update, histograms' creation)
with respect to the specifications of all the files & formats concerned.

Definition at line 22 of file cmsPerfStripChart.py.

References mps_setup.append, objects.autophobj.float, get_yaxis_range(), createfilelist.int, edm.print(), split, and str.

Referenced by edm::EventProcessor.beginJob(), edm::SubProcess.beginJob(), edm::test::TestProcessor.beginJob(), edm::EventProcessor.beginLumiAsync(), edm::SubProcess.clearCounters(), edm::SubProcess.closeOutputFiles(), edm::serviceregistry::ServicesManager.createServices(), edm::SubProcess.doBeginLuminosityBlockAsync(), edm::SubProcess.doBeginRunAsync(), edm::SubProcess.doBeginStream(), edm::SubProcess.doEndStream(), edm::SubProcess.doEventAsync(), evf::EvFBuildingThrottle.dowork(), edm::SubProcess.enableEndPaths(), edm::EventProcessor.endJob(), edm::SubProcess.endJob(), edm::test::TestProcessor.endJob(), edm::SubProcess.endPathsEnabled(), edm::test::TestProcessor.event(), edm::SubProcess.getTriggerReport(), edm::EventProcessor.globalEndLumiAsync(), edm::EventProcessor.handleNextEventForStreamAsync(), edm::EventProcessor.init(), main(), edm::SubProcess.openOutputFiles(), cond::payloadInspector::PlotBase.process(), edm::EventProcessor.processEventAsyncImpl(), edm::StreamSchedule.processOneEventAsync(), edm::EventProcessor.readNextEventForStream(), edm::ReducedProvenanceReader.ReducedProvenanceReader(), edm::SubProcess.respondToCloseInputFile(), edm::SubProcess.respondToOpenInputFile(), cond::Utilities.run(), edm::EventProcessor.runToCompletion(), edm::SubProcess.shouldWeCloseOutput(), edm::SubProcess.SubProcess(), edm::SubProcess.terminate(), edm::test::TestProcessor.TestProcessor(), topLevelPSet(), edm::SubProcess.totalEventsFailed(), edm::SubProcess.totalEventsPassed(), edm::SubProcess.writeLumiAsync(), and edm::SubProcess.writeRunAsync().

22 def operate(timelog, memlog, json_f, num):
23  """
24  Main operation of the script (i.e. json db update, histograms' creation)
25  with respect to the specifications of all the files & formats concerned.
26  """
27  import re
28  import commands
29  import ROOT
30  from datetime import datetime
31 
32  script_name=os.path.basename(__file__)
33 
34  # Open files and store the lines.
35  timefile=open(timelog, 'r')
36  timelog_lines=timefile.readlines()
37  timefile.close()
38 
39  memfile=open(memlog, 'r')
40  memlog_lines=memfile.readlines()
41  memfile.close()
42 
43  # Get average, uncertainty of average and maximum rss.
44  max_rss=average=error=' '
45  i=0
46  while i<len(timelog_lines):
47  line=timelog_lines[i]
48  if 'Uncertainty of Average Time' in line:
49  line=line[:-1]
50  line_list=line.split(' ')
51  average=line_list[5]
52  error=line_list[7]
53  i+=1
54  i=0
55  while i<len(memlog_lines):
56  line=memlog_lines[i]
57  if 'Maximum rss' in line:
58  line=line[:-1]
59  line_list=line.split(' ')
60  max_rss=line_list[3]
61  break
62  i+=1
63 
64  # Get Integration Build's identifier
65  IB=os.path.basename(commands.getoutput("echo $CMSSW_BASE"))
66 
67  # Check if log files were parsed properly...
68  # and if the IB is valid using regular expressions.
69  try:
70  # regex for a float
71  regex="^\d+\.?\d*$"
72  if average == ' ' or re.match(regex, average) is None:
73  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
74  'Check if Average Time is defined correctly.')
75  if error == ' ' or re.match(regex, error) is None:
76  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
77  'Check if Uncertainty of Average Time is defined correctly.')
78  if max_rss == ' ' or re.match(regex, max_rss) is None:
79  raise RuntimeError('Could not parse \"' + memlog + '\" properly. ' +\
80  ' Check if Maximum rss is defined correct.')
81 
82  # regex for dates 'YYYY-MM-DD-HHMM'
83  regex = '(19|20|21)\d\d-(0[1-9]|1[012])-(0[1-9]|[12]'+\
84  '[0-9]|3[01])-([01][0-9]|2[0-4])([0-5][0-9])$'
85  if re.search(regex, IB) is None:
86  raise RuntimeError('Not a valid IB. Valid IB: ' +\
87  '[CMSSW_X_X_X_YYYY-MM-DD-HHMM]')
88  except Exception as err:
89  sys.stderr.write(script_name + ': Error: ' + str(err) + '\n')
90  return 2
91 
92  # Open for getting the data.
93  json_db=open(json_f, "r")
94  dict=json.load(json_db)
95  json_db.close()
96 
97  # Get the data to be stored and check if already exists.
98  ib_list=IB.split('_')
99  cmsrelease=ib_list[0] + '_' + ib_list[1] +\
100  '_' + ib_list[2] + '_' + ib_list[3]
101  data={"IB" : ib_list[4], "average" : float(average), "error" : float(error), "max_rss" : float(max_rss)}
102 
103  if data in dict["strips"]:
104  sys.stderr.write(script_name + ": Warning: Entry already exists " +\
105  "in json file and will not be stored! " +\
106  "Only the strip charts will be created.\n")
107  else:
108  dict["strips"].append(data)
109  print('Storing entry to \"' + json_f +\
110  '\" file with attribute values:\n' +\
111  'IB=' + IB + '\naverage=' + average +\
112  '\nUncertainty of average=' + error +'\nmax_rss=' + max_rss)
113  # Store the data in json file.
114  json_db = open(json_f, "w+")
115  json.dump(dict, json_db, indent=2)
116  json_db.close()
117  print('File "' + json_f + '" was updated successfully!')
118 
119  # Change to datetime type (helpful for sorting).
120  for record in dict["strips"]:
121  time_list = record['IB'].split('-')
122  d = datetime(int(time_list[0]), int(time_list[1]),
123  int(time_list[2]), int(time_list[3][0:2]),
124  int(time_list[3][2:]))
125  record['IB'] = d
126 
127  # Sort the list.
128  list = sorted(dict["strips"], key=lambda k : k['IB'], reverse=True)
129 
130  # Check if there are NUM entries.
131  if num > len(list):
132  new_num = len(list)
133  sys.stderr.write(script_name + ': Warning: There are less than ' +\
134  str(num) + ' entries in json file. Changed number to ' +\
135  str(new_num) + '.\n')
136  num = new_num
137 
138  # The histograms.
139  ROOT.gROOT.SetStyle("Plain")
140  outdir='.'
141 
142  # Save in file
143  rootfilename=outdir + '/histograms.root'
144  myfile=ROOT.TFile(rootfilename, 'RECREATE')
145 
146  # Average time histogram.
147  histo1=ROOT.TH1F("AveCPU per IB", "Ave CPU per IB", num, 0., num)
148  histo1.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
149  histo1.SetName('avecpu_histo')
150 
151  # Maximum rss histogram.
152  histo2=ROOT.TH1F("Max rrs per IB", "Max rss per IB", num, 0., num)
153  histo2.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
154  histo2.SetName('maxrss_histo')
155 
156  # Fill in the histograms
157  for i in range(num):
158  datime = list[i]['IB'].__format__('%Y-%b-%d %H:%M')
159  average = list[i]['average']
160  max_rss = list[i]['max_rss']
161  error = list[i]['error']
162 
163  histo1.GetXaxis().SetBinLabel(num-i, datime)
164  histo1.SetBinContent(num-i, average)
165  histo1.SetBinError(num-i, error)
166  histo2.GetXaxis().SetBinLabel(num-i, datime)
167  histo2.SetBinContent(num-i, max_rss)
168 
169  histo1.SetStats(0)
170  histo1.GetYaxis().SetTitle("Average CPU time")
171  histo1.GetYaxis().SetTitleOffset(1.8)
172  histo1.GetXaxis().SetTitle("Integration Build")
173  histo1.GetXaxis().SetTitleOffset(4.)
174  histo1.GetXaxis().CenterTitle()
175  histo1.GetXaxis().LabelsOption('v')
176  # Histo1 - Set limits on the Y-axis
177  min, max = get_yaxis_range(list)
178  interval = max - min
179  # ...get a bit more space
180  min = min-interval*0.1
181  max = max+interval*0.1
182  histo1.GetYaxis().SetRangeUser(min, max)
183 
184  histo2.SetStats(0)
185  histo2.GetYaxis().SetTitle("Maximum rss")
186  histo2.GetYaxis().SetTitleOffset(1.8)
187  histo2.GetXaxis().SetTitle("Integration Build")
188  histo2.GetXaxis().SetTitleOffset(4.)
189  histo2.GetXaxis().CenterTitle()
190  histo2.GetXaxis().LabelsOption('v')
191 
192  # Draw and save!
193 
194  ave_canvas = ROOT.TCanvas(cmsrelease + '_average_canvas')
195  ave_canvas.SetGridy()
196  ave_canvas.SetBottomMargin(0.28)
197  ave_canvas.SetLeftMargin(0.18)
198  ave_canvas.cd()
199  # Histo1 - draw line
200  histo1.SetLineColor(2)
201  histo1.SetLineWidth(2)
202  histo1.DrawCopy("HISTO L")
203  # Histo1 - draw errors and markers
204  histo1.SetLineColor(1)
205  histo1.SetLineStyle(2)
206  histo1.SetLineWidth(1)
207  histo1.SetMarkerStyle(8)
208  histo1.SetMarkerSize(.6)
209  histo1.SetMarkerColor(1)
210  histo1.Draw("E1P SAME")
211  ROOT.gStyle.SetErrorX(0)
212  ave_canvas.Print(outdir + "/average_cpu_histo.png","png")
213 
214  rss_canvas = ROOT.TCanvas(cmsrelease + '_maxrss_canvas')
215  rss_canvas.SetGridy()
216  rss_canvas.SetBottomMargin(0.28)
217  rss_canvas.SetLeftMargin(0.18)
218  rss_canvas.cd()
219  # Histo2 - draw line
220  histo2.SetLineColor(2)
221  histo2.SetLineWidth(2)
222  histo2.DrawCopy("L")
223  # Histo2 - draw markers
224  histo2.SetMarkerStyle(8)
225  histo2.SetMarkerSize(.6)
226  histo2.SetMarkerColor(1)
227  histo2.Draw("P SAME")
228  rss_canvas.Print(outdir + "/maximum_rss_histo.png","png")
229 
230  # write them on file
231  histo1.Write()
232  ave_canvas.Write()
233  histo2.Write()
234  rss_canvas.Write()
235 
236 
edm::print
S & print(S &os, JobReport::InputFile const &f)
Definition: JobReport.cc:65
mps_setup.append
append
Definition: mps_setup.py:84
cmsPerfStripChart.get_yaxis_range
def get_yaxis_range(list)
Definition: cmsPerfStripChart.py:9
str
#define str(s)
Definition: TestProcessor.cc:47
split
double split
Definition: MVATrainer.cc:139
cmsPerfStripChart.operate
def operate(timelog, memlog, json_f, num)
Definition: cmsPerfStripChart.py:22

Variable Documentation

cmsPerfStripChart.action

Definition at line 250 of file cmsPerfStripChart.py.

cmsPerfStripChart.args

Definition at line 273 of file cmsPerfStripChart.py.

cmsPerfStripChart.default

Definition at line 252 of file cmsPerfStripChart.py.

cmsPerfStripChart.dest

Definition at line 251 of file cmsPerfStripChart.py.

cmsPerfStripChart.dict

Definition at line 308 of file cmsPerfStripChart.py.

Referenced by Mixins._TypedParameterizable.__findDefaultsFor(), svgfig.CurveAxis.__init__(), trackingPlots._appendTrackingPlots(), checkBTagCalibrationConsistency.BtagCalibConsistencyChecker._check_sys_side(), trackingPlots._constructSummary(), querying._get_netrc_data(), Config.Process._itemsInDependencyOrder(), trackingPlots._makeDistPlots(), trackingPlots._makeDistSimPlots(), trackingPlots._makeEffFakeDupPlots(), trackingPlots._makeMVAPlots(), Mixins._modifyParametersFromDict(), data_formats._to_array_of_dicts(), svgfig.canvas(), Utilities.cleanUnscheduled(), iterativeTkConfig.clusterRemoverForIter(), relval_machine.convert_keys_to_string(), geometry.Alignables.create_children_list(), alignmentValidation.createCanvasToIDList(), customisePixelPairStaticMitigation.customisePixelPairStaticMitigation(), Vispa.Gui.VispaWidget.VispaWidget.defineDistances(), dqmMemoryStats.HistogramAnalyzer.difference(), Config.Process.extend(), edmConvertToStreamModule.find_all_module_classes(), helper.findEDAnalyzers(), helper.findEDFilters(), helper.findEDProducers(), helper.findModules(), cms.generate(), models.generate(), generateFlavCfromFlavB.generate_flav_c(), web.app_utils.get_directory_summary_stats(), web.app_utils.get_folders(), HTMLExport.get_jquery(), web.app_utils.get_release_summary_stats(), cmssw_das_client.get_value(), das.get_value(), specificLumi.getFillFromDB(), specificLumi.getSpecificLumi(), RecoLocalCalo_Cosmics_cff.hbheCosmic(), mpl_axes_hist_fix.hist(), combineBTagCalibrationData.main(), printTrackingNtuple.main(), timingPdfMaker.maininfo(), utils_v2.make_file_pairs(), utils.make_files_pairs(), Types.makeCppPSet(), DeltaR.matchObjectCollection(), deltar.matchObjectCollection(), DeltaR.matchObjectCollection2(), deltar.matchObjectCollection2(), DeltaR.matchObjectCollection3(), deltar.matchObjectCollection3(), Mixins._Parameterizable.parameters_(), python.rootplot.core.parse_arguments(), iniparser.ConfigData.parseConfig(), svgfig.pathtoPath(), matplotRender.matplotRender.plotInst_RunLS(), matplotRender.matplotRender.plotPeakPerday_Time(), matplotRender.matplotRender.plotSumX_Fill(), matplotRender.matplotRender.plotSumX_Run(), matplotRender.matplotRender.plotSumX_Time(), cmsPerfSuiteHarvest.process_igprof_dir(), cmsPerfSuiteHarvest.process_memcheck_dir(), cmsPerfSuiteHarvest.process_timesize_dir(), HTMLExport.HTMLExport.produce(), utils_v2.recursive_search_online(), python.rootplot.core.rootplot(), fullMixCustomize_cff.setCrossingFrameOn(), geometryXMLtoCSV.ContentHandler.startElement(), geometryXMLparser.MuonGeometry.startElement(), Config.TestModuleCommand.testModifier(), Types.testTypes.testPSet(), SequenceTypes.TestModuleCommand.testResolve(), and dqm-mbProfile.Profile.to_dict().

cmsPerfStripChart.format

Some error handling for the usage.

# Validity of .json file-database. #

Definition at line 291 of file cmsPerfStripChart.py.

Referenced by NVProfilerService::Domains.allocate_streams(), L1uGTTreeProducer.analyze(), FastTimerService::PlotsPerElement.book(), dqm::RamdiskMonitor.bookHistograms(), TriggerBxMonitor.bookHistograms(), TriggerRatesMonitor.bookHistograms(), sistrip::MeasureLA.calibration_key(), sistrip::RawToDigiUnpacker.createDigis(), sistrip::DigiToRaw.createFedBuffers_(), ComphepSingletopFilter.endJob(), ComphepSingletopFilterPy8.endJob(), SummaryPlotFactory< CommissioningAnalysis * >.fill(), L1TGlobalPrescaler.filter(), EmDQMReco::FourVectorMonitorElements.FourVectorMonitorElements(), MatacqProducer.getMatacqFile(), dqm::RamdiskMonitor.getRunTimestamp(), L1TriggerJSONMonitoring.globalBeginRun(), HLTriggerJSONMonitoring.globalBeginRun(), L1TriggerJSONMonitoring.globalEndLuminosityBlockSummary(), HLTriggerJSONMonitoring.globalEndLuminosityBlockSummary(), DQMFileSaver.globalEndRun(), CovarianceParameterization.load(), edm::service::IgProfService.makeDump(), IgProfModule.makeDump(), dqmservices::DQMFileIterator.monUpdateLumi(), dqmservices::DQMMonitoringService.outputLumiUpdate(), ListGroups.overlayEtaReferences(), dqmservices::JsonWritingTimeoutPoolOutputModule.physicalAndLogicalNameForNewFile(), FastTimerService.postGlobalEndLumi(), FastTimerService.postGlobalEndRun(), ThroughputService.preallocate(), FastTimerService.preallocate(), ThroughputService.preGlobalBeginRun(), FastTimerService.printEventLine(), FastTimerService.printPathSummaryLine(), FastTimerService.printSummaryLine(), dqmservices::DQMFileIterator.reset(), MatacqProducer.runSubDir(), DQMFileSaver.saveForFilterUnit(), FWTriggerTableView.saveImageTo(), FWTableView.saveImageTo(), dqm::DQMFileSaverPB.saveLumi(), TrackingMaterialAnalyser.saveParameters(), sistrip::EnsembleCalibrationLA.write_ensembles_text(), HLTriggerJSONMonitoring.writeIniFile(), L1TriggerJSONMonitoring.writeIniFile(), edm::writeParameterValue.writeValue(), edm::writeParameterValue.writeValueInVector(), edm::writeParameterValue.writeValueInVectorWithSpace(), and edm::writeParameterValue.writeVector().

cmsPerfStripChart.help

Definition at line 254 of file cmsPerfStripChart.py.

cmsPerfStripChart.json_db

Definition at line 298 of file cmsPerfStripChart.py.

cmsPerfStripChart.metavar

Definition at line 253 of file cmsPerfStripChart.py.

cmsPerfStripChart.options

Definition at line 273 of file cmsPerfStripChart.py.

cmsPerfStripChart.parser

Definition at line 248 of file cmsPerfStripChart.py.

cmsPerfStripChart.script_name

Definition of command usage.

#

Definition at line 246 of file cmsPerfStripChart.py.

cmsPerfStripChart.type

Definition at line 267 of file cmsPerfStripChart.py.

cmsPerfStripChart.usage

Definition at line 247 of file cmsPerfStripChart.py.

Generated for CMSSW Reference Manual by doxygen 1.8.11