CMS 3D CMS Logo
Functions | Variables
cmsPerfStripChart Namespace Reference

Functions

def get_yaxis_range (list)
 
def operate (timelog, memlog, json_f, num)
 

Variables

 action
 
 args
 
 default
 
 dest
 
 dict
 
 format
 Some error handling for the usage. More...
 
 help
 
 json_db
 
 metavar
 
 options
 
 parser
 
 script_name
 Definition of command usage. More...
 
 type
 
 usage
 

Function Documentation

def cmsPerfStripChart.get_yaxis_range (   list) 
Given a list of dictionaries, where each dict holds the information
about an IB, this function returns a tuple (low, high) with the lowest
and the highest value of y-axis, respectively.

Definition at line 8 of file cmsPerfStripChart.py.

References hpstanc_transforms.max, and min().

Referenced by operate().

8 def get_yaxis_range(list):
9  """
10  Given a list of dictionaries, where each dict holds the information
11  about an IB, this function returns a tuple (low, high) with the lowest
12  and the highest value of y-axis, respectively.
13  """
14  low, high = sys.maxsize, -1
15  for node in list:
16  low = min((node['average'] - node['error']), low)
17  high = max((node['average'] + node['error']), high)
18  return (low, high)
19 
20 # Main operation function
hpstanc_transforms.max
max
Definition: hpstanc_transforms.py:5
min
T min(T a, T b)
Definition: MathUtil.h:58
cmsPerfStripChart.get_yaxis_range
def get_yaxis_range(list)
Definition: cmsPerfStripChart.py:8
def cmsPerfStripChart.operate (   timelog,
  memlog,
  json_f,
  num 
) 
Main operation of the script (i.e. json db update, histograms' creation)
with respect to the specifications of all the files & formats concerned.

Definition at line 21 of file cmsPerfStripChart.py.

References mps_alisetup.append, objects.autophobj.float, get_yaxis_range(), createfilelist.int, split, and harvestTrackValidationPlots.str.

Referenced by edm::EventProcessor.beginJob(), edm::SubProcess.beginJob(), edm::SubProcess.clearCounters(), edm::SubProcess.closeOutputFiles(), edm::serviceregistry::ServicesManager.createServices(), edm::SubProcess.doBeginLuminosityBlock(), edm::SubProcess.doBeginLuminosityBlockAsync(), edm::SubProcess.doBeginRun(), edm::SubProcess.doBeginRunAsync(), edm::SubProcess.doBeginStream(), edm::SubProcess.doEndLuminosityBlock(), edm::SubProcess.doEndRun(), edm::SubProcess.doEndStream(), edm::SubProcess.doEventAsync(), edm::SubProcess.doStreamBeginLuminosityBlock(), edm::SubProcess.doStreamBeginLuminosityBlockAsync(), edm::SubProcess.doStreamBeginRun(), edm::SubProcess.doStreamBeginRunAsync(), edm::SubProcess.doStreamEndLuminosityBlock(), edm::SubProcess.doStreamEndLuminosityBlockAsync(), edm::SubProcess.doStreamEndRun(), edm::SubProcess.doStreamEndRunAsync(), evf::EvFBuildingThrottle.dowork(), edm::SubProcess.enableEndPaths(), edm::EventProcessor.endJob(), edm::SubProcess.endJob(), edm::SubProcess.endPathsEnabled(), edm::StreamSchedule.finishedPaths(), edm::SubProcess.getTriggerReport(), edm::EventProcessor.handleNextEventForStreamAsync(), edm::EventProcessor.init(), main(), edm::SubProcess.openOutputFiles(), edm::EventProcessor.processEventAsync(), edm::StreamSchedule.processOneEventAsync(), edm::EventProcessor.readNextEventForStream(), edm::ReducedProvenanceReader.ReducedProvenanceReader(), edm::SubProcess.respondToCloseInputFile(), edm::SubProcess.respondToOpenInputFile(), cond::Utilities.run(), edm::EventProcessor.runToCompletion(), edm::SubProcess.shouldWeCloseOutput(), edm::SubProcess.SubProcess(), edm::SubProcess.terminate(), topLevelPSet(), edm::SubProcess.totalEventsFailed(), edm::SubProcess.totalEventsPassed(), edm::SubProcess.writeLumi(), and edm::SubProcess.writeRun().

21 def operate(timelog, memlog, json_f, num):
22  """
23  Main operation of the script (i.e. json db update, histograms' creation)
24  with respect to the specifications of all the files & formats concerned.
25  """
26  import re
27  import commands
28  import ROOT
29  from datetime import datetime
30 
31  script_name=os.path.basename(__file__)
32 
33  # Open files and store the lines.
34  timefile=open(timelog, 'r')
35  timelog_lines=timefile.readlines()
36  timefile.close()
37 
38  memfile=open(memlog, 'r')
39  memlog_lines=memfile.readlines()
40  memfile.close()
41 
42  # Get average, uncertainty of average and maximum rss.
43  max_rss=average=error=' '
44  i=0
45  while i<len(timelog_lines):
46  line=timelog_lines[i]
47  if 'Uncertainty of Average Time' in line:
48  line=line[:-1]
49  line_list=line.split(' ')
50  average=line_list[5]
51  error=line_list[7]
52  i+=1
53  i=0
54  while i<len(memlog_lines):
55  line=memlog_lines[i]
56  if 'Maximum rss' in line:
57  line=line[:-1]
58  line_list=line.split(' ')
59  max_rss=line_list[3]
60  break
61  i+=1
62 
63  # Get Integration Build's identifier
64  IB=os.path.basename(commands.getoutput("echo $CMSSW_BASE"))
65 
66  # Check if log files were parsed properly...
67  # and if the IB is valid using regular expressions.
68  try:
69  # regex for a float
70  regex="^\d+\.?\d*$"
71  if average == ' ' or re.match(regex, average) is None:
72  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
73  'Check if Average Time is defined correctly.')
74  if error == ' ' or re.match(regex, error) is None:
75  raise RuntimeError('Could not parse \"' + timelog + '\" properly. ' +\
76  'Check if Uncertainty of Average Time is defined correctly.')
77  if max_rss == ' ' or re.match(regex, max_rss) is None:
78  raise RuntimeError('Could not parse \"' + memlog + '\" properly. ' +\
79  ' Check if Maximum rss is defined correct.')
80 
81  # regex for dates 'YYYY-MM-DD-HHMM'
82  regex = '(19|20|21)\d\d-(0[1-9]|1[012])-(0[1-9]|[12]'+\
83  '[0-9]|3[01])-([01][0-9]|2[0-4])([0-5][0-9])$'
84  if re.search(regex, IB) is None:
85  raise RuntimeError('Not a valid IB. Valid IB: ' +\
86  '[CMSSW_X_X_X_YYYY-MM-DD-HHMM]')
87  except Exception as err:
88  sys.stderr.write(script_name + ': Error: ' + str(err) + '\n')
89  return 2
90 
91  # Open for getting the data.
92  json_db=open(json_f, "r")
93  dict=json.load(json_db)
94  json_db.close()
95 
96  # Get the data to be stored and check if already exists.
97  ib_list=IB.split('_')
98  cmsrelease=ib_list[0] + '_' + ib_list[1] +\
99  '_' + ib_list[2] + '_' + ib_list[3]
100  data={"IB" : ib_list[4], "average" : float(average), "error" : float(error), "max_rss" : float(max_rss)}
101 
102  if data in dict["strips"]:
103  sys.stderr.write(script_name + ": Warning: Entry already exists " +\
104  "in json file and will not be stored! " +\
105  "Only the strip charts will be created.\n")
106  else:
107  dict["strips"].append(data)
108  print 'Storing entry to \"' + json_f +\
109  '\" file with attribute values:\n' +\
110  'IB=' + IB + '\naverage=' + average +\
111  '\nUncertainty of average=' + error +'\nmax_rss=' + max_rss
112  # Store the data in json file.
113  json_db = open(json_f, "w+")
114  json.dump(dict, json_db, indent=2)
115  json_db.close()
116  print 'File "' + json_f + '" was updated successfully!'
117 
118  # Change to datetime type (helpful for sorting).
119  for record in dict["strips"]:
120  time_list = record['IB'].split('-')
121  d = datetime(int(time_list[0]), int(time_list[1]),
122  int(time_list[2]), int(time_list[3][0:2]),
123  int(time_list[3][2:]))
124  record['IB'] = d
125 
126  # Sort the list.
127  list = sorted(dict["strips"], key=lambda k : k['IB'], reverse=True)
128 
129  # Check if there are NUM entries.
130  if num > len(list):
131  new_num = len(list)
132  sys.stderr.write(script_name + ': Warning: There are less than ' +\
133  str(num) + ' entries in json file. Changed number to ' +\
134  str(new_num) + '.\n')
135  num = new_num
136 
137  # The histograms.
138  ROOT.gROOT.SetStyle("Plain")
139  outdir='.'
140 
141  # Save in file
142  rootfilename=outdir + '/histograms.root'
143  myfile=ROOT.TFile(rootfilename, 'RECREATE')
144 
145  # Average time histogram.
146  histo1=ROOT.TH1F("AveCPU per IB", "Ave CPU per IB", num, 0., num)
147  histo1.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
148  histo1.SetName('avecpu_histo')
149 
150  # Maximum rss histogram.
151  histo2=ROOT.TH1F("Max rrs per IB", "Max rss per IB", num, 0., num)
152  histo2.SetTitle(cmsrelease + ": Showing last " + str(num) + " IBs")
153  histo2.SetName('maxrss_histo')
154 
155  # Fill in the histograms
156  for i in range(num):
157  datime = list[i]['IB'].__format__('%Y-%b-%d %H:%M')
158  average = list[i]['average']
159  max_rss = list[i]['max_rss']
160  error = list[i]['error']
161 
162  histo1.GetXaxis().SetBinLabel(num-i, datime)
163  histo1.SetBinContent(num-i, average)
164  histo1.SetBinError(num-i, error)
165  histo2.GetXaxis().SetBinLabel(num-i, datime)
166  histo2.SetBinContent(num-i, max_rss)
167 
168  histo1.SetStats(0)
169  histo1.GetYaxis().SetTitle("Average CPU time")
170  histo1.GetYaxis().SetTitleOffset(1.8)
171  histo1.GetXaxis().SetTitle("Integration Build")
172  histo1.GetXaxis().SetTitleOffset(4.)
173  histo1.GetXaxis().CenterTitle()
174  histo1.GetXaxis().LabelsOption('v')
175  # Histo1 - Set limits on the Y-axis
176  min, max = get_yaxis_range(list)
177  interval = max - min
178  # ...get a bit more space
179  min = min-interval*0.1
180  max = max+interval*0.1
181  histo1.GetYaxis().SetRangeUser(min, max)
182 
183  histo2.SetStats(0)
184  histo2.GetYaxis().SetTitle("Maximum rss")
185  histo2.GetYaxis().SetTitleOffset(1.8)
186  histo2.GetXaxis().SetTitle("Integration Build")
187  histo2.GetXaxis().SetTitleOffset(4.)
188  histo2.GetXaxis().CenterTitle()
189  histo2.GetXaxis().LabelsOption('v')
190 
191  # Draw and save!
192 
193  ave_canvas = ROOT.TCanvas(cmsrelease + '_average_canvas')
194  ave_canvas.SetGridy()
195  ave_canvas.SetBottomMargin(0.28)
196  ave_canvas.SetLeftMargin(0.18)
197  ave_canvas.cd()
198  # Histo1 - draw line
199  histo1.SetLineColor(2)
200  histo1.SetLineWidth(2)
201  histo1.DrawCopy("HISTO L")
202  # Histo1 - draw errors and markers
203  histo1.SetLineColor(1)
204  histo1.SetLineStyle(2)
205  histo1.SetLineWidth(1)
206  histo1.SetMarkerStyle(8)
207  histo1.SetMarkerSize(.6)
208  histo1.SetMarkerColor(1)
209  histo1.Draw("E1P SAME")
210  ROOT.gStyle.SetErrorX(0)
211  ave_canvas.Print(outdir + "/average_cpu_histo.png","png")
212 
213  rss_canvas = ROOT.TCanvas(cmsrelease + '_maxrss_canvas')
214  rss_canvas.SetGridy()
215  rss_canvas.SetBottomMargin(0.28)
216  rss_canvas.SetLeftMargin(0.18)
217  rss_canvas.cd()
218  # Histo2 - draw line
219  histo2.SetLineColor(2)
220  histo2.SetLineWidth(2)
221  histo2.DrawCopy("L")
222  # Histo2 - draw markers
223  histo2.SetMarkerStyle(8)
224  histo2.SetMarkerSize(.6)
225  histo2.SetMarkerColor(1)
226  histo2.Draw("P SAME")
227  rss_canvas.Print(outdir + "/maximum_rss_histo.png","png")
228 
229  # write them on file
230  histo1.Write()
231  ave_canvas.Write()
232  histo2.Write()
233  rss_canvas.Write()
234 
235 
cmsPerfStripChart.get_yaxis_range
def get_yaxis_range(list)
Definition: cmsPerfStripChart.py:8
mps_alisetup.append
string append
Definition: mps_alisetup.py:651
split
double split
Definition: MVATrainer.cc:139
cmsPerfStripChart.operate
def operate(timelog, memlog, json_f, num)
Definition: cmsPerfStripChart.py:21

Variable Documentation

cmsPerfStripChart.action

Definition at line 249 of file cmsPerfStripChart.py.

cmsPerfStripChart.args

Definition at line 272 of file cmsPerfStripChart.py.

cmsPerfStripChart.default

Definition at line 251 of file cmsPerfStripChart.py.

cmsPerfStripChart.dest

Definition at line 250 of file cmsPerfStripChart.py.

cmsPerfStripChart.dict

Definition at line 307 of file cmsPerfStripChart.py.

Referenced by Mixins._TypedParameterizable.__findDefaultsFor(), svgfig.CurveAxis.__init__(), argparse.ArgumentError.__str__(), python.rootplot.argparse.ArgumentError.__str__(), trackingPlots._appendTrackingPlots(), checkBTagCalibrationConsistency.BtagCalibConsistencyChecker._check_sys_side(), trackingPlots._constructSummary(), argparse.HelpFormatter._expand_help(), python.rootplot.argparse.HelpFormatter._expand_help(), python.rootplot.argparse.HelpFormatter._format_text(), argparse.HelpFormatter._format_usage(), python.rootplot.argparse.HelpFormatter._format_usage(), querying._get_netrc_data(), argparse._ActionsContainer._get_optional_kwargs(), python.rootplot.argparse._ActionsContainer._get_optional_kwargs(), argparse._ActionsContainer._get_positional_kwargs(), python.rootplot.argparse._ActionsContainer._get_positional_kwargs(), Config.Process._itemsInDependencyOrder(), trackingPlots._makeDistPlots(), trackingPlots._makeDistSimPlots(), trackingPlots._makeEffFakeDupPlots(), trackingPlots._makeMVAPlots(), Mixins._modifyParametersFromDict(), data_formats._to_array_of_dicts(), svgfig.canvas(), Utilities.cleanUnscheduled(), iterativeTkConfig.clusterRemoverForIter(), relval_machine.convert_keys_to_string(), geometry.Alignables.create_children_list(), alignmentValidation.createCanvasToIDList(), Vispa.Gui.VispaWidget.VispaWidget.defineDistances(), Config.Process.extend(), edmConvertToStreamModule.find_all_module_classes(), helper.findEDAnalyzers(), helper.findEDFilters(), helper.findEDProducers(), helper.findModules(), cms.generate(), models.generate(), generateFlavCfromFlavB.generate_flav_c(), web.app_utils.get_directory_summary_stats(), web.app_utils.get_folders(), HTMLExport.get_jquery(), web.app_utils.get_release_summary_stats(), cmssw_das_client.get_value(), das.get_value(), specificLumi.getFillFromDB(), specificLumi.getSpecificLumi(), RecoLocalCalo_Cosmics_cff.hbheCosmic(), mpl_axes_hist_fix.hist(), combineBTagCalibrationData.main(), printTrackingNtuple.main(), timingPdfMaker.maininfo(), utils_v2.make_file_pairs(), utils.make_files_pairs(), Types.makeCppPSet(), DeltaR.matchObjectCollection(), deltar.matchObjectCollection(), DeltaR.matchObjectCollection2(), deltar.matchObjectCollection2(), DeltaR.matchObjectCollection3(), deltar.matchObjectCollection3(), Mixins._Parameterizable.parameters_(), python.rootplot.core.parse_arguments(), iniparser.ConfigData.parseConfig(), svgfig.pathtoPath(), matplotRender.matplotRender.plotInst_RunLS(), matplotRender.matplotRender.plotPeakPerday_Time(), matplotRender.matplotRender.plotSumX_Fill(), matplotRender.matplotRender.plotSumX_Run(), matplotRender.matplotRender.plotSumX_Time(), cmsPerfSuiteHarvest.process_igprof_dir(), cmsPerfSuiteHarvest.process_memcheck_dir(), cmsPerfSuiteHarvest.process_timesize_dir(), HTMLExport.HTMLExport.produce(), utils_v2.recursive_search_online(), python.rootplot.core.rootplot(), fullMixCustomize_cff.setCrossingFrameOn(), geometryXMLtoCSV.ContentHandler.startElement(), geometryXMLparser.MuonGeometry.startElement(), Config.TestModuleCommand.testModifier(), Types.testTypes.testPSet(), SequenceTypes.TestModuleCommand.testResolve(), and dqm-mbProfile.Profile.to_dict().

cmsPerfStripChart.format

Some error handling for the usage.

# Validity of .json file-database. #

Definition at line 290 of file cmsPerfStripChart.py.

Referenced by argparse.HelpFormatter._metavar_formatter(), python.rootplot.argparse.HelpFormatter._metavar_formatter(), L1uGTTreeProducer.analyze(), FastTimerService::PlotsPerElement.book(), dqm::RamdiskMonitor.bookHistograms(), TriggerBxMonitor.bookHistograms(), TriggerRatesMonitor.bookHistograms(), sistrip::MeasureLA.calibration_key(), ComphepSingletopFilter.endJob(), ComphepSingletopFilterPy8.endJob(), SummaryPlotFactory< CommissioningAnalysis * >.fill(), L1TGlobalPrescaler.filter(), EmDQMReco::FourVectorMonitorElements.FourVectorMonitorElements(), MatacqProducer.getMatacqFile(), dqm::RamdiskMonitor.getRunTimestamp(), L1TriggerJSONMonitoring.globalBeginRun(), HLTriggerJSONMonitoring.globalBeginRun(), L1TriggerJSONMonitoring.globalEndLuminosityBlockSummary(), HLTriggerJSONMonitoring.globalEndLuminosityBlockSummary(), DQMFileSaver.globalEndRun(), CovarianceParameterization.load(), edm::service::IgProfService.makeDump(), IgProfModule.makeDump(), dqmservices::DQMFileIterator.monUpdateLumi(), dqmservices::DQMMonitoringService.outputLumiUpdate(), ListGroups.overlayEtaReferences(), dqmservices::JsonWritingTimeoutPoolOutputModule.physicalAndLogicalNameForNewFile(), FastTimerService.preallocate(), ThroughputService.preStreamBeginRun(), FastTimerService.printEventLine(), FastTimerService.printSummaryLine(), dqmservices::DQMFileIterator.reset(), MatacqProducer.runSubDir(), DQMFileSaver.saveForFilterUnit(), FWTriggerTableView.saveImageTo(), FWTableView.saveImageTo(), dqm::DQMFileSaverPB.saveLumi(), TrackingMaterialAnalyser.saveParameters(), sistrip::EnsembleCalibrationLA.write_ensembles_text(), HLTriggerJSONMonitoring.writeIniFile(), L1TriggerJSONMonitoring.writeIniFile(), edm::writeParameterValue.writeValue(), edm::writeParameterValue.writeValueInVector(), edm::writeParameterValue.writeValueInVectorWithSpace(), and edm::writeParameterValue.writeVector().

cmsPerfStripChart.help

Definition at line 253 of file cmsPerfStripChart.py.

cmsPerfStripChart.json_db

Definition at line 297 of file cmsPerfStripChart.py.

cmsPerfStripChart.metavar

Definition at line 252 of file cmsPerfStripChart.py.

cmsPerfStripChart.options

Definition at line 272 of file cmsPerfStripChart.py.

cmsPerfStripChart.parser

Definition at line 247 of file cmsPerfStripChart.py.

cmsPerfStripChart.script_name

Definition of command usage.

#

Definition at line 245 of file cmsPerfStripChart.py.

cmsPerfStripChart.type

Definition at line 266 of file cmsPerfStripChart.py.

cmsPerfStripChart.usage

Definition at line 246 of file cmsPerfStripChart.py.

Generated for CMSSW Reference Manual by doxygen 1.8.11