CMS 3D CMS Logo
Functions | Variables
cmsPerfStripChart Namespace Reference

Functions

def get_yaxis_range (list)
 
def operate (timelog, memlog, json_f, num)
 

Variables

 action
 
 args
 
 default
 
 dest
 
 dict
 
 format
 Some error handling for the usage. More...
 
 help
 
 json_db
 
 metavar
 
 options
 
 parser
 
 script_name
 Definition of command usage. More...
 
 type
 
 usage
 

Function Documentation

def cmsPerfStripChart.get_yaxis_range (   list) 
Given a list of dictionaries, where each dict holds the information
about an IB, this function returns a tuple (low, high) with the lowest
and the highest value of y-axis, respectively.

Definition at line 10 of file cmsPerfStripChart.py.

References SiStripPI.max, and min().

Referenced by operate().

10 def get_yaxis_range(list):
11  """
12  Given a list of dictionaries, where each dict holds the information
13  about an IB, this function returns a tuple (low, high) with the lowest
14  and the highest value of y-axis, respectively.
15  """
16  low, high = sys.maxsize, -1
17  for node in list:
18  low = min((node['average'] - node['error']), low)
19  high = max((node['average'] + node['error']), high)
20  return (low, high)
21 
22 # Main operation function
min
T min(T a, T b)
Definition: MathUtil.h:58
def cmsPerfStripChart.operate (   timelog,
  memlog,
  json_f,
  num 
) 
Main operation of the script (i.e. json db update, histograms' creation)
with respect to the specifications of all the files & formats concerned.

Definition at line 23 of file cmsPerfStripChart.py.

References mps_setup.append, objects.autophobj.float, get_yaxis_range(), createfilelist.int, edm.print(), split, and str.

Referenced by edm::EventProcessor.beginJob(), edm::SubProcess.beginJob(), edm::test::TestProcessor.beginJob(), edm::EventProcessor.beginLumiAsync(), edm::SubProcess.clearCounters(), edm::SubProcess.closeOutputFiles(), edm::serviceregistry::ServicesManager.createServices(), edm::SubProcess.doBeginLuminosityBlockAsync(), edm::SubProcess.doBeginRunAsync(), edm::SubProcess.doBeginStream(), edm::SubProcess.doEndStream(), edm::SubProcess.doEventAsync(), evf::EvFBuildingThrottle.dowork(), edm::SubProcess.enableEndPaths(), edm::EventProcessor.endJob(), edm::SubProcess.endJob(), edm::test::TestProcessor.endJob(), edm::SubProcess.endPathsEnabled(), edm::test::TestProcessor.event(), edm::SubProcess.getTriggerReport(), edm::EventProcessor.globalEndLumiAsync(), edm::EventProcessor.handleNextEventForStreamAsync(), edm::EventProcessor.init(), main(), edm::SubProcess.openOutputFiles(), cond::payloadInspector::PlotBase.process(), edm::EventProcessor.processEventAsyncImpl(), edm::StreamSchedule.processOneEventAsync(), cond::payloadInspector::PlotBase.processTwoTags(), edm::EventProcessor.readNextEventForStream(), edm::ReducedProvenanceReader.ReducedProvenanceReader(), edm::SubProcess.respondToCloseInputFile(), edm::SubProcess.respondToOpenInputFile(), cond::Utilities.run(), edm::EventProcessor.runToCompletion(), edm::SubProcess.shouldWeCloseOutput(), edm::SubProcess.SubProcess(), edm::SubProcess.terminate(), edm::test::TestProcessor.TestProcessor(), topLevelPSet(), edm::SubProcess.totalEventsFailed(), edm::SubProcess.totalEventsPassed(), edm::SubProcess.writeLumiAsync(), and edm::SubProcess.writeRunAsync().

23 def operate(timelog, memlog, json_f, num):
24  """
25  Main operation of the script (i.e. json db update, histograms' creation)
26  with respect to the specifications of all the files & formats concerned.
27  """
28  import re
29  import commands
30  import ROOT
31  from datetime import datetime
32 
33  script_name=os.path.basename(__file__)
34 
35  # Open files and store the lines.
36  timefile=open(timelog, 'r')
37  timelog_lines=timefile.readlines()
38  timefile.close()
39 
40  memfile=open(memlog, 'r')
41  memlog_lines=memfile.readlines()
42  memfile.close()
43 
44  # Get average, uncertainty of average and maximum rss.
45  max_rss=average=error=' '
46  i=0
47  while i<len(timelog_lines):
48  line=timelog_lines[i]
49  if 'Uncertainty of Average Time' in line:
50  line=line[:-1]
51  line_list=line.split(' ')
52  average=line_list[5]
53  error=line_list[7]
54  i+=1
55  i=0
56  while i<len(memlog_lines):
57  line=memlog_lines[i]
58  if 'Maximum rss' in line:
59  line=line[:-1]
60  line_list=line.split(' ')
61  max_rss=line_list[3]
62  break
63  i+=1
64 
65  # Get Integration Build's identifier
66  IB=os.path.basename(commands.getoutput("echo $CMSSW_BASE"))
67 
68  # Check if log files were parsed properly...
69  # and if the IB is valid using regular expressions.
70  try:
71  # regex for a float
72  regex="^\d+\.?\d*$"
73  if average == ' ' or re.match(regex, average) is None:
74  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
75  'Check if Average Time is defined correctly.')
76  if error == ' ' or re.match(regex, error) is None:
77  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
78  'Check if Uncertainty of Average Time is defined correctly.')
79  if max_rss == ' ' or re.match(regex, max_rss) is None:
80  raise RuntimeError('Could not parse \"' + memlog + '\" properly. ' +\
81  ' Check if Maximum rss is defined correct.')
82 
83  # regex for dates 'YYYY-MM-DD-HHMM'
84  regex = '(19|20|21)\d\d-(0[1-9]|1[012])-(0[1-9]|[12]'+\
85  '[0-9]|3[01])-([01][0-9]|2[0-4])([0-5][0-9])$'
86  if re.search(regex, IB) is None:
87  raise RuntimeError('Not a valid IB. Valid IB: ' +\
88  '[CMSSW_X_X_X_YYYY-MM-DD-HHMM]')
89  except Exception as err:
90  sys.stderr.write(script_name + ': Error: ' + str(err) + '\n')
91  return 2
92 
93  # Open for getting the data.
94  json_db=open(json_f, "r")
95  dict=json.load(json_db)
96  json_db.close()
97 
98  # Get the data to be stored and check if already exists.
99  ib_list=IB.split('_')
100  cmsrelease=ib_list[0] + '_' + ib_list[1] +\
101  '_' + ib_list[2] + '_' + ib_list[3]
102  data={"IB" : ib_list[4], "average" : float(average), "error" : float(error), "max_rss" : float(max_rss)}
103 
104  if data in dict["strips"]:
105  sys.stderr.write(script_name + ": Warning: Entry already exists " +\
106  "in json file and will not be stored! " +\
107  "Only the strip charts will be created.\n")
108  else:
109  dict["strips"].append(data)
110  print('Storing entry to \"' + json_f +\
111  '\" file with attribute values:\n' +\
112  'IB=' + IB + '\naverage=' + average +\
113  '\nUncertainty of average=' + error +'\nmax_rss=' + max_rss)
114  # Store the data in json file.
115  json_db = open(json_f, "w+")
116  json.dump(dict, json_db, indent=2)
117  json_db.close()
118  print('File "' + json_f + '" was updated successfully!')
119 
120  # Change to datetime type (helpful for sorting).
121  for record in dict["strips"]:
122  time_list = record['IB'].split('-')
123  d = datetime(int(time_list[0]), int(time_list[1]),
124  int(time_list[2]), int(time_list[3][0:2]),
125  int(time_list[3][2:]))
126  record['IB'] = d
127 
128  # Sort the list.
129  list = sorted(dict["strips"], key=lambda k : k['IB'], reverse=True)
130 
131  # Check if there are NUM entries.
132  if num > len(list):
133  new_num = len(list)
134  sys.stderr.write(script_name + ': Warning: There are less than ' +\
135  str(num) + ' entries in json file. Changed number to ' +\
136  str(new_num) + '.\n')
137  num = new_num
138 
139  # The histograms.
140  ROOT.gROOT.SetStyle("Plain")
141  outdir='.'
142 
143  # Save in file
144  rootfilename=outdir + '/histograms.root'
145  myfile=ROOT.TFile(rootfilename, 'RECREATE')
146 
147  # Average time histogram.
148  histo1=ROOT.TH1F("AveCPU per IB", "Ave CPU per IB", num, 0., num)
149  histo1.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
150  histo1.SetName('avecpu_histo')
151 
152  # Maximum rss histogram.
153  histo2=ROOT.TH1F("Max rrs per IB", "Max rss per IB", num, 0., num)
154  histo2.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
155  histo2.SetName('maxrss_histo')
156 
157  # Fill in the histograms
158  for i in range(num):
159  datime = list[i]['IB'].__format__('%Y-%b-%d %H:%M')
160  average = list[i]['average']
161  max_rss = list[i]['max_rss']
162  error = list[i]['error']
163 
164  histo1.GetXaxis().SetBinLabel(num-i, datime)
165  histo1.SetBinContent(num-i, average)
166  histo1.SetBinError(num-i, error)
167  histo2.GetXaxis().SetBinLabel(num-i, datime)
168  histo2.SetBinContent(num-i, max_rss)
169 
170  histo1.SetStats(0)
171  histo1.GetYaxis().SetTitle("Average CPU time")
172  histo1.GetYaxis().SetTitleOffset(1.8)
173  histo1.GetXaxis().SetTitle("Integration Build")
174  histo1.GetXaxis().SetTitleOffset(4.)
175  histo1.GetXaxis().CenterTitle()
176  histo1.GetXaxis().LabelsOption('v')
177  # Histo1 - Set limits on the Y-axis
178  min, max = get_yaxis_range(list)
179  interval = max - min
180  # ...get a bit more space
181  min = min-interval*0.1
182  max = max+interval*0.1
183  histo1.GetYaxis().SetRangeUser(min, max)
184 
185  histo2.SetStats(0)
186  histo2.GetYaxis().SetTitle("Maximum rss")
187  histo2.GetYaxis().SetTitleOffset(1.8)
188  histo2.GetXaxis().SetTitle("Integration Build")
189  histo2.GetXaxis().SetTitleOffset(4.)
190  histo2.GetXaxis().CenterTitle()
191  histo2.GetXaxis().LabelsOption('v')
192 
193  # Draw and save!
194 
195  ave_canvas = ROOT.TCanvas(cmsrelease + '_average_canvas')
196  ave_canvas.SetGridy()
197  ave_canvas.SetBottomMargin(0.28)
198  ave_canvas.SetLeftMargin(0.18)
199  ave_canvas.cd()
200  # Histo1 - draw line
201  histo1.SetLineColor(2)
202  histo1.SetLineWidth(2)
203  histo1.DrawCopy("HISTO L")
204  # Histo1 - draw errors and markers
205  histo1.SetLineColor(1)
206  histo1.SetLineStyle(2)
207  histo1.SetLineWidth(1)
208  histo1.SetMarkerStyle(8)
209  histo1.SetMarkerSize(.6)
210  histo1.SetMarkerColor(1)
211  histo1.Draw("E1P SAME")
212  ROOT.gStyle.SetErrorX(0)
213  ave_canvas.Print(outdir + "/average_cpu_histo.png","png")
214 
215  rss_canvas = ROOT.TCanvas(cmsrelease + '_maxrss_canvas')
216  rss_canvas.SetGridy()
217  rss_canvas.SetBottomMargin(0.28)
218  rss_canvas.SetLeftMargin(0.18)
219  rss_canvas.cd()
220  # Histo2 - draw line
221  histo2.SetLineColor(2)
222  histo2.SetLineWidth(2)
223  histo2.DrawCopy("L")
224  # Histo2 - draw markers
225  histo2.SetMarkerStyle(8)
226  histo2.SetMarkerSize(.6)
227  histo2.SetMarkerColor(1)
228  histo2.Draw("P SAME")
229  rss_canvas.Print(outdir + "/maximum_rss_histo.png","png")
230 
231  # write them on file
232  histo1.Write()
233  ave_canvas.Write()
234  histo2.Write()
235  rss_canvas.Write()
236 
237 
edm::print
S & print(S &os, JobReport::InputFile const &f)
Definition: JobReport.cc:66
mps_setup.append
append
Definition: mps_setup.py:85
str
#define str(s)
Definition: TestProcessor.cc:48
split
double split
Definition: MVATrainer.cc:139
cmsPerfStripChart.operate
def operate(timelog, memlog, json_f, num)
Definition: cmsPerfStripChart.py:23

Variable Documentation

cmsPerfStripChart.action

Definition at line 251 of file cmsPerfStripChart.py.

cmsPerfStripChart.args

Definition at line 274 of file cmsPerfStripChart.py.

cmsPerfStripChart.default

Definition at line 253 of file cmsPerfStripChart.py.

cmsPerfStripChart.dest

Definition at line 252 of file cmsPerfStripChart.py.

cmsPerfStripChart.dict

Definition at line 309 of file cmsPerfStripChart.py.

Referenced by Mixins._TypedParameterizable.__findDefaultsFor(), Modules.SwitchProducerTest.__init__(), Modules.SwitchProducerTest1Dis.__init__(), Modules.SwitchProducerTest2Dis.__init__(), Modules.SwitchProducerPickleable.__init__(), Config.SwitchProducerTest.__init__(), svgfig.CurveAxis.__init__(), trackingPlots._appendTrackingPlots(), checkBTagCalibrationConsistency.BtagCalibConsistencyChecker._check_sys_side(), trackingPlots._constructSummary(), querying._get_netrc_data(), Config.Process._itemsInDependencyOrder(), trackingPlots._makeDistPlots(), trackingPlots._makeDistSimPlots(), trackingPlots._makeEffFakeDupPlots(), trackingPlots._makeMVAPlots(), Mixins._modifyParametersFromDict(), FWLite.WorkingPoints._reformat_cut_definitions(), data_formats._to_array_of_dicts(), ctppsProtons_cff.apply2017Settings(), ctppsProtons_cff.apply2018Settings(), ctppsProtons_cff.applyDefaultSettings(), svgfig.canvas(), Utilities.cleanUnscheduled(), Modules.SwitchProducer.clone(), iterativeTkConfig.clusterRemoverForIter(), relval_machine.convert_keys_to_string(), geometry.Alignables.create_children_list(), alignmentValidation.createCanvasToIDList(), customisePixelPairStaticMitigation.customisePixelPairStaticMitigation(), Vispa.Gui.VispaWidget.VispaWidget.defineDistances(), dqmMemoryStats.HistogramAnalyzer.difference(), Config.Process.extend(), edmConvertToStreamModule.find_all_module_classes(), helper.findEDAnalyzers(), helper.findEDFilters(), helper.findEDProducers(), helper.findModules(), cms.generate(), models.generate(), generateFlavCfromFlavB.generate_flav_c(), web.app_utils.get_directory_summary_stats(), web.app_utils.get_folders(), HTMLExport.get_jquery(), web.app_utils.get_release_summary_stats(), cmssw_das_client.get_value(), das.get_value(), specificLumi.getFillFromDB(), mixture.getMixtures(), specificLumi.getSpecificLumi(), RecoLocalCalo_Cosmics_cff.hbheCosmic(), mpl_axes_hist_fix.hist(), combineBTagCalibrationData.main(), printTrackingNtuple.main(), timingPdfMaker.maininfo(), utils_v2.make_file_pairs(), utils.make_files_pairs(), Types.makeCppPSet(), DeltaR.matchObjectCollection(), deltar.matchObjectCollection(), DeltaR.matchObjectCollection2(), deltar.matchObjectCollection2(), DeltaR.matchObjectCollection3(), deltar.matchObjectCollection3(), Mixins._Parameterizable.parameters_(), python.rootplot.core.parse_arguments(), iniparser.ConfigData.parseConfig(), svgfig.pathtoPath(), matplotRender.matplotRender.plotInst_RunLS(), matplotRender.matplotRender.plotPeakPerday_Time(), matplotRender.matplotRender.plotSumX_Fill(), matplotRender.matplotRender.plotSumX_Run(), matplotRender.matplotRender.plotSumX_Time(), cmsPerfSuiteHarvest.process_igprof_dir(), cmsPerfSuiteHarvest.process_memcheck_dir(), cmsPerfSuiteHarvest.process_timesize_dir(), HTMLExport.HTMLExport.produce(), utils_v2.recursive_search_online(), python.rootplot.core.rootplot(), fullMixCustomize_cff.setCrossingFrameOn(), geometryXMLtoCSV.ContentHandler.startElement(), geometryXMLparser.MuonGeometry.startElement(), Types.testTypes.testEDAlias(), Config.TestModuleCommand.testModifier(), Types.testTypes.testPSet(), SequenceTypes.TestModuleCommand.testResolve(), Modules.TestModules.testSwitchProducer(), Modules.TestModules.testSwithProducerWithAlias(), and dqm-mbProfile.Profile.to_dict().

cmsPerfStripChart.format

Some error handling for the usage.

# Validity of .json file-database. #

Definition at line 292 of file cmsPerfStripChart.py.

Referenced by NVProfilerService::Domains.allocate_streams(), L1uGTTreeProducer.analyze(), FastTimerService::PlotsPerElement.book(), TriggerBxMonitor.bookHistograms(), TriggerRatesMonitor.bookHistograms(), sistrip::MeasureLA.calibration_key(), sistrip::RawToDigiUnpacker.createDigis(), sistrip::DigiToRaw.createFedBuffers_(), ComphepSingletopFilter.endJob(), ComphepSingletopFilterPy8.endJob(), CalibrationSummaryFactory.extract(), CalibrationScanSummaryFactory.extract(), SummaryPlotFactory< CommissioningAnalysis * >.fill(), L1TGlobalPrescaler.filter(), EmDQMReco::FourVectorMonitorElements.FourVectorMonitorElements(), MatacqProducer.getMatacqFile(), dqm::RamdiskMonitor.getRunTimestamp(), L1TriggerJSONMonitoring.globalBeginRun(), HLTriggerJSONMonitoring.globalBeginRun(), L1TriggerJSONMonitoring.globalEndLuminosityBlockSummary(), HLTriggerJSONMonitoring.globalEndLuminosityBlockSummary(), DQMFileSaver.globalEndRun(), CovarianceParameterization.load(), edm::service::IgProfService.makeDump(), IgProfModule.makeDump(), dqmservices::DQMFileIterator.monUpdateLumi(), dqmservices::DQMMonitoringService.outputLumiUpdate(), ListGroups.overlayEtaReferences(), dqmservices::JsonWritingTimeoutPoolOutputModule.physicalAndLogicalNameForNewFile(), FastTimerService.postGlobalEndLumi(), FastTimerService.postGlobalEndRun(), ThroughputService.preallocate(), FastTimerService.preallocate(), ThroughputService.preGlobalBeginRun(), FastTimerService.printEventLine(), FastTimerService.printPathSummaryLine(), FastTimerService.printSummaryLine(), dqm::RamdiskMonitor.RamdiskMonitor(), dqmservices::DQMFileIterator.reset(), MatacqProducer.runSubDir(), DQMFileSaver.saveForFilterUnit(), FWTriggerTableView.saveImageTo(), FWTableView.saveImageTo(), dqm::DQMFileSaverPB.saveLumi(), TrackingMaterialAnalyser.saveParameters(), sistrip::EnsembleCalibrationLA.write_ensembles_text(), HLTriggerJSONMonitoring.writeIniFile(), L1TriggerJSONMonitoring.writeIniFile(), cmsswFiletrace.writeoutput(), edm::writeParameterValue.writeValue(), edm::writeParameterValue.writeValueInVector(), edm::writeParameterValue.writeValueInVectorWithSpace(), and edm::writeParameterValue.writeVector().

cmsPerfStripChart.help

Definition at line 255 of file cmsPerfStripChart.py.

cmsPerfStripChart.json_db

Definition at line 299 of file cmsPerfStripChart.py.

cmsPerfStripChart.metavar

Definition at line 254 of file cmsPerfStripChart.py.

cmsPerfStripChart.options

Definition at line 274 of file cmsPerfStripChart.py.

cmsPerfStripChart.parser

Definition at line 249 of file cmsPerfStripChart.py.

cmsPerfStripChart.script_name

Definition of command usage.

#

Definition at line 247 of file cmsPerfStripChart.py.

cmsPerfStripChart.type

Definition at line 268 of file cmsPerfStripChart.py.

cmsPerfStripChart.usage

Definition at line 248 of file cmsPerfStripChart.py.

Generated for CMSSW Reference Manual by doxygen 1.8.11