|
|
|
Classes | |
| class | PerfTest |
| class | simpleGenReportThread |
Public Member Functions | |
| def | __init__ |
| def | benchmarks |
| Run cmsScimark benchmarks a number of times. | |
| def | cprootfile |
| Copy root file from another candle's directory ! Again this is messy. | |
| def | displayErrors |
| Display G4 cerr errors and CMSExceptions in the logfile. | |
| def | getDate |
| def | mkCandleDir |
| Make directory for a particular candle and profiler. | |
| def | optionParse |
| def | printDate |
| def | printFlush |
| Print and flush a string (for output to a log file) | |
| def | runcmd |
| Run a command and return the exit status. | |
| def | runCmdSet |
| Run a list of commands using system ! We should rewrite this not to use system (most cases it is unnecessary) | |
| def | runCmsInput |
| Wrapper for cmsRelvalreportInput. | |
| def | runCmsReport |
| This function is a wrapper around cmsRelvalreport. | |
| def | runPerfSuite |
| Runs benchmarking, cpu spinlocks on spare cores and profiles selected candles. | |
| def | simpleGenReport |
| Prepares the profiling directory and runs all the selected profiles (if this is not a unit test) | |
| def | testCmsDriver |
| Test cmsDriver.py (parses the simcandles file, removing duplicate lines, and runs the cmsDriver part) | |
| def | valFilterReport |
| Filter lines in the valgrind report that match GEN,SIM. | |
Public Attributes | |
| AuxiliaryScripts | |
| cmssw_arch | |
| cmssw_version | |
| Commands | |
| ERRORS | |
| HEPSPEC06 | |
| host | |
| logh | |
| PerfTestTotalTimer | |
| FIXME: We may want to introduce a switch here or agree on a different default (currently 10 cmsScimark and 10 cmsScimarkLarge) | |
| Scripts | |
| user | |
Private Attributes | |
| _CASTOR_DIR | |
| _debug | |
| Developer options. | |
| _dryrun | |
| _noexec | |
| _unittest | |
| _verbose | |
| Check logfile option. | |
Definition at line 60 of file cmsPerfSuite.py.
| def cmsPerfSuite::PerfSuite::__init__ | ( | self | ) |
Definition at line 61 of file cmsPerfSuite.py.
00062 : 00063 00064 self.ERRORS = 0 00065 self._CASTOR_DIR = "/castor/cern.ch/cms/store/relval/performance/" 00066 self._dryrun = False 00067 self._debug = False 00068 self._unittest = False 00069 self._noexec = False 00070 self._verbose = True 00071 self.logh = sys.stdout 00072 00073 #Get some environment variables to use 00074 try: 00075 self.cmssw_arch = os.environ["SCRAM_ARCH"] 00076 self.cmssw_version= os.environ["CMSSW_VERSION"] 00077 self.host = os.environ["HOST"] 00078 self.user = os.environ["USER"] 00079 except KeyError: 00080 self.logh.write('Error: An environment variable either SCRAM_ARCH, CMSSW_VERSION, HOST or USER is not available.\n') 00081 self.logh.write(' Please run eval `scramv1 runtime -csh` to set your environment variables\n') 00082 self.logh.flush() 00083 sys.exit() 00084 00085 #Scripts used by the suite: 00086 self.Scripts =["cmsDriver.py","cmsRelvalreport.py","cmsRelvalreportInput.py","cmsScimark2"] 00087 self.AuxiliaryScripts=["cmsScimarkLaunch.csh","cmsScimarkParser.py","cmsScimarkStop.py"] 00088
| def cmsPerfSuite::PerfSuite::benchmarks | ( | self, | |
| cpu, | |||
| pfdir, | |||
| name, | |||
| bencher, | |||
large = False |
|||
| ) |
Run cmsScimark benchmarks a number of times.
Definition at line 678 of file cmsPerfSuite.py.
00679 : 00680 cmd = self.Commands[cpu][3] 00681 redirect = "" 00682 if large: 00683 redirect = " -large >>" 00684 else: 00685 redirect = " >>" 00686 00687 for i in range(bencher): 00688 #Check first for the existence of the file so that we can append: 00689 if not os.path.exists(os.path.join(pfdir,os.path.basename(name))): 00690 #Equivalent of touch to make sure the file exist to be able to append to it. 00691 open(os.path.join(pfdir,os.path.basename(name))) 00692 00693 command= cmd + redirect + os.path.join(pfdir,os.path.basename(name)) 00694 self.printFlush(command + " [%s/%s]" % (i+1,bencher)) 00695 self.runcmd(command) 00696 self.logh.flush()
| def cmsPerfSuite::PerfSuite::cprootfile | ( | self, | |
| dir, | |||
| candle, | |||
| NumOfEvents, | |||
cmsdriverOptions = "" |
|||
| ) |
Copy root file from another candle's directory ! Again this is messy.
Definition at line 599 of file cmsPerfSuite.py.
00600 : 00601 cmds = ("cd %s" % dir, 00602 "cp -pR ../%s_IgProf/%s_GEN,SIM.root ." % (candle,CandFname[candle])) 00603 00604 if self.runCmdSet(cmds): 00605 self.logh.write("Since there was no ../%s_IgProf/%s_GEN,SIM.root file it will be generated first\n"%(candle,CandFname[candle])) 00606 00607 cmd = "cd %s ; cmsDriver.py %s -s GEN,SIM -n %s --fileout %s_GEN,SIM.root %s>& %s_GEN_SIM_for_valgrind.log" % (dir,KeywordToCfi[candle],str(NumOfEvents),candle,cmsdriverOptions,candle) 00608 00609 self.printFlush(cmd) 00610 #Obsolete popen4-> subprocess.Popen 00611 #cmdout=os.popen3(cmd)[2].read() 00612 cmdout=subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read() 00613 if cmdout: 00614 self.printFlush(cmdout) 00615 return cmdout
| def cmsPerfSuite::PerfSuite::displayErrors | ( | self, | |
| file | |||
| ) |
Display G4 cerr errors and CMSExceptions in the logfile.
Definition at line 619 of file cmsPerfSuite.py.
00620 : 00621 try: 00622 for line in open(file,"r"): 00623 if "cerr" in line or "CMSException" in line: 00624 self.logh.write("ERROR: %s\n" % line) 00625 self.ERRORS += 1 00626 except OSError, detail: 00627 self.logh.write("WARNING: %s\n" % detail) 00628 self.ERRORS += 1 00629 except IOError, detail: 00630 self.logh.write("WARNING: %s\n" % detail) 00631 self.ERRORS += 1
| def cmsPerfSuite::PerfSuite::getDate | ( | self | ) |
Definition at line 579 of file cmsPerfSuite.py.
| def cmsPerfSuite::PerfSuite::mkCandleDir | ( | self, | |
| pfdir, | |||
| candle, | |||
| profiler | |||
| ) |
Make directory for a particular candle and profiler.
! This is really unnecessary code and should be replaced with a os.mkdir() call
Definition at line 588 of file cmsPerfSuite.py.
| def cmsPerfSuite::PerfSuite::optionParse | ( | self, | |
argslist = None |
|||
| ) |
Definition at line 146 of file cmsPerfSuite.py.
00147 : 00148 parser = opt.OptionParser(usage='''./cmsPerfSuite.py [options] 00149 00150 Examples: 00151 00152 cmsPerfSuite.py --step GEN-HLT -t 5 -i 2 -c 1 -m 5 --RunTimeSize MinBias,TTbar --RunIgProf TTbar --RunCallgrind TTbar --RunMemcheck TTbar --RunDigiPileUp TTbar --PUInputFile /store/relval/CMSSW_2_2_1/RelValMinBias/GEN-SIM-DIGI-RAW-HLTDEBUG/IDEAL_V9_v2/0001/101C84AF-56C4-DD11-A90D-001D09F24EC0.root --cmsdriver="--eventcontent FEVTDEBUGHLT --conditions FrontierConditions_GlobalTag,IDEAL_V9::All" 00153 (this will run the suite with 5 events for TimeSize tests on MinBias and TTbar, 2 for IgProf tests on TTbar only, 1 for Callgrind tests on TTbar only, 5 for Memcheck on MinBias and TTbar, it will also run DIGI PILEUP for all TTbar tests defined, i.e. 5 TimeSize, 2 IgProf, 1 Callgrind, 5 Memcheck. The file /store/relval/CMSSW_2_2_1/RelValMinBias/GEN-SIM-DIGI-RAW-HLTDEBUG/IDEAL_V9_v2/0001/101C84AF-56C4-DD11-A90D-001D09F24EC0.root will be copied locally as INPUT_PILEUP_EVENTS.root and it will be used as the input file for the MixingModule pile up events. All these tests will be done for the step GEN-HLT, i.e. GEN,SIM,DIGI,L1,DIGI2RAW,HLT at once) 00154 OR 00155 cmsPerfSuite.py --step GEN-HLT -t 5 -i 2 -c 1 -m 5 --RunTimeSize MinBias,TTbar --RunIgProf TTbar --RunCallgrind TTbar --RunMemcheck TTbar --RunTimeSizePU TTbar --PUInputFile /store/relval/CMSSW_2_2_1/RelValMinBias/GEN-SIM-DIGI-RAW-HLTDEBUG/IDEAL_V9_v2/0001/101C84AF-56C4-DD11-A90D-001D09F24EC0.root 00156 (this will run the suite with 5 events for TimeSize tests on MinBias and TTbar, 2 for IgProf tests on TTbar only, 1 for Callgrind tests on TTbar only, 5 for Memcheck on MinBias and TTbar, it will also run DIGI PILEUP on TTbar but only for 5 TimeSize events. All these tests will be done for the step GEN-HLT, i.e. GEN,SIM,DIGI,L1,DIGI2RAW,HLT at once) 00157 OR 00158 cmsPerfSuite.py --step GEN-HLT -t 5 -i 2 -c 1 -m 5 --RunTimeSize MinBias,TTbar --RunIgProf TTbar --RunCallgrind TTbar --RunMemcheck TTbar --RunTimeSizePU TTbar --PUInputFile /store/relval/CMSSW_2_2_1/RelValMinBias/GEN-SIM-DIGI-RAW-HLTDEBUG/IDEAL_V9_v2/0001/101C84AF-56C4-DD11-A90D-001D09F24EC0.root --cmsdriver="--eventcontent RAWSIM --conditions FrontierConditions_GlobalTag,IDEAL_V9::All" 00159 (this will run the suite with 5 events for TimeSize tests on MinBias and TTbar, 2 for IgProf tests on TTbar only, 1 for Callgrind tests on TTbar only, 5 for Memcheck on MinBias and TTbar, it will also run DIGI PILEUP on TTbar but only for 5 TimeSize events. All these tests will be done for the step GEN-HLT, i.e. GEN,SIM,DIGI,L1,DIGI2RAW,HLT at once. It will also add the options "--eventcontent RAWSIM --conditions FrontierConditions_GlobalTag,IDEAL_V9::All" to all cmsDriver.py commands executed by the suite. In addition it will run only 2 cmsDriver.py "steps": "GEN,SIM" and "DIGI". Note the syntax GEN-SIM for combined cmsDriver.py steps) 00160 00161 Legal entries for individual candles (--RunTimeSize, --RunIgProf, --RunCallgrind, --RunMemcheck options): 00162 %s 00163 ''' % ("\n".join(Candles))) 00164 00165 parser.set_defaults(TimeSizeEvents = 0 , 00166 IgProfEvents = 0 , 00167 CallgrindEvents = 0 , 00168 MemcheckEvents = 0 , 00169 cmsScimark = 10 , 00170 cmsScimarkLarge = 10 , 00171 cmsdriverOptions = "--eventcontent FEVTDEBUGHLT", # Decided to avoid using the automatic parsing of cmsDriver_highstats_hlt.txt: cmsRelValCmd.get_cmsDriverOptions(), #Get these options automatically now! 00172 #"Release Integrators" will create another file relative to the performance suite and the operators will fetch from that file the --cmsdriver option... for now just set the eventcontent since that is needed in order for things to run at all now... 00173 stepOptions = "" , 00174 profilers = "" , 00175 outputdir = "" , 00176 logfile = os.path.join(os.getcwd(),"cmsPerfSuite.log"), 00177 runonspare = True , 00178 bypasshlt = False , 00179 quicktest = False , 00180 unittest = False , 00181 noexec = False , 00182 dryrun = False , 00183 verbose = True , 00184 previousrel = "" , 00185 castordir = self._CASTOR_DIR, 00186 cores = cmsCpuInfo.get_NumOfCores(), #Get Number of cpu cores on the machine from /proc/cpuinfo 00187 cpu = "1" , #Cpu core on which the suite is run: 00188 RunTimeSize = "" , 00189 RunIgProf = "" , 00190 RunCallgrind = "" , 00191 RunMemcheck = "" , 00192 RunDigiPileUP = "" , 00193 RunTimeSizePU = "" , 00194 RunIgProfPU = "" , 00195 RunCallgrindPU = "" , 00196 RunMemcheckPU = "" , 00197 PUInputFile = "" , 00198 userInputFile = "" ) 00199 parser.add_option('-q', '--quiet' , action="store_false", dest='verbose' , 00200 help = 'Output less information' ) 00201 parser.add_option('-b', '--bypass-hlt' , action="store_true" , dest='bypasshlt' , 00202 help = 'Bypass HLT root file as input to RAW2DIGI') 00203 parser.add_option('-n', '--notrunspare', action="store_false", dest='runonspare', 00204 help = 'Do not run cmsScimark on spare cores') 00205 parser.add_option('-t', '--timesize' , type='int' , dest='TimeSizeEvents' , metavar='<#EVENTS>' , 00206 help = 'specify the number of events for the TimeSize tests' ) 00207 parser.add_option('-i', '--igprof' , type='int' , dest='IgProfEvents' , metavar='<#EVENTS>' , 00208 help = 'specify the number of events for the IgProf tests' ) 00209 parser.add_option('-c', '--callgrind' , type='int' , dest='CallgrindEvents' , metavar='<#EVENTS>' , 00210 help = 'specify the number of events for the Callgrind tests' ) 00211 parser.add_option('-m', '--memcheck' , type='int' , dest='MemcheckEvents' , metavar='<#EVENTS>' , 00212 help = 'specify the number of events for the Memcheck tests' ) 00213 parser.add_option('--cmsScimark' , type='int' , dest='cmsScimark' , metavar='' , 00214 help = 'specify the number of times the cmsScimark benchmark is run before and after the performance suite on cpu1') 00215 parser.add_option('--cmsScimarkLarge' , type='int' , dest='cmsScimarkLarge' , metavar='' , 00216 help = 'specify the number of times the cmsScimarkLarge benchmark is run before and after the performance suite on cpu1') 00217 parser.add_option('--cores' , type='int', dest='cores' , metavar='<CORES>' , 00218 help = 'specify the number of cores of the machine (can be used with 0 to stop cmsScimark from running on the other cores)') 00219 parser.add_option('--cmsdriver' , type='string', dest='cmsdriverOptions', metavar='<OPTION_STR>', 00220 help = 'specify special options to use with the cmsDriver.py commands (designed for integration build use') 00221 parser.add_option('-a', '--archive' , type='string', dest='castordir' , metavar='<DIR>' , 00222 help = 'specify the wanted CASTOR directory where to store the results tarball') 00223 parser.add_option('-L', '--logfile' , type='string', dest='logfile' , metavar='<FILE>' , 00224 help = 'file to store log output of the script') 00225 parser.add_option('-o', '--output' , type='string', dest='outputdir' , metavar='<DIR>' , 00226 help = 'specify the directory where to store the output of the script') 00227 parser.add_option('-r', '--prevrel' , type='string', dest='previousrel' , metavar='<DIR>' , 00228 help = 'Top level dir of previous release for regression analysis') 00229 parser.add_option('--step' , type='string', dest='stepOptions' , metavar='<STEPS>' , 00230 help = 'specify the processing steps intended (instead of the default ones)' ) 00231 parser.add_option('--cpu' , type='string', dest='cpu' , metavar='<CPU>' , 00232 help = 'specify the core on which to run the performance suite') 00233 00234 #Adding new options to put everything configurable at command line: 00235 parser.add_option('--RunTimeSize' , type='string', dest='RunTimeSize' , metavar='<CANDLES>' , 00236 help = 'specify on which candles to run the TimeSize tests') 00237 parser.add_option('--RunIgProf' , type='string', dest='RunIgProf' , metavar='<CANDLES>' , 00238 help = 'specify on which candles to run the IgProf tests') 00239 parser.add_option('--RunCallgrind' , type='string', dest='RunCallgrind' , metavar='<CANDLES>' , 00240 help = 'specify on which candles to run the Callgrind tests') 00241 parser.add_option('--RunMemcheck' , type='string', dest='RunMemcheck' , metavar='<CANDLES>' , 00242 help = 'specify on which candles to run the Memcheck tests') 00243 parser.add_option('--RunDigiPileUp' , type='string', dest='RunDigiPileUp' , metavar='<CANDLES>' , 00244 help = 'specify the candle on which to run DIGI PILE UP and repeat all the tests set to run on that candle with PILE UP') 00245 parser.add_option('--PUInputFile' , type='string', dest='PUInputFile' , metavar='<FILE>' , 00246 help = 'specify the root file to pick the pile-up events from') 00247 parser.add_option('--RunTimeSizePU' , type='string', dest='RunTimeSizePU' , metavar='<CANDLES>' , 00248 help = 'specify on which candles to run the TimeSize tests with PILE UP') 00249 parser.add_option('--RunIgProfPU' , type='string', dest='RunIgProfPU' , metavar='<CANDLES>' , 00250 help = 'specify on which candles to run the IgProf tests with PILE UP') 00251 parser.add_option('--RunCallgrindPU' , type='string', dest='RunCallgrindPU' , metavar='<CANDLES>' , 00252 help = 'specify on which candles to run the Callgrind tests with PILE UP') 00253 parser.add_option('--RunMemcheckPU' , type='string', dest='RunMemcheckPU' , metavar='<CANDLES>' , 00254 help = 'specify on which candles to run the Memcheck tests with PILE UP') 00255 00256 #Adding a filein option to use pre-processed RAW file for RECO and HLT: 00257 parser.add_option('--filein' , type='string', dest='userInputFile' , metavar='<FILE>', #default="", 00258 help = 'specify input RAW root file for HLT and RAW2DIGI-RECO (list the files in the same order as the candles for the tests)') 00259 00260 #Adding an option to handle additional (to the default user) email addresses to the email notification list (that sends the cmsPerfSuite.log once the performance suite is done running): 00261 parser.add_option('--mail', type='string', dest='MailLogRecipients', metavar='<EMAIL ADDRESS>', default=self.user, help='specify valid email address(es) name@domain in order to receive notification at the end of the performance suite running with the cmsPerfSuite.log file') 00262 00263 #Adding option to turn off tarball creation at the end of the execution of the performance suite: 00264 parser.add_option('--no_tarball', action="store_false", dest='tarball', default=True, help='Turn off automatic tarball creation at the end of the performance suite execution') 00265 00266 ##################### 00267 # 00268 # Developer options 00269 # 00270 00271 devel = opt.OptionGroup(parser, "Developer Options", 00272 "Caution: use these options at your own risk." 00273 "It is believed that some of them bite.\n") 00274 00275 devel.add_option('-p', '--profile' , type="str" , dest='profilers', metavar="<PROFILERS>" , 00276 help = 'Profile codes to use for cmsRelvalInput' ) 00277 devel.add_option('-f', '--false-run', action="store_true", dest='dryrun' , 00278 help = 'Dry run' ) 00279 devel.add_option('-d', '--debug' , action='store_true', dest='debug' , 00280 help = 'Debug' ) 00281 devel.add_option('--quicktest' , action="store_true", dest='quicktest', 00282 help = 'Quick overwrite all the defaults to small numbers so that we can run a quick test of our chosing.' ) 00283 devel.add_option('--test' , action="store_true", dest='unittest' , 00284 help = 'Perform a simple test, overrides other options. Overrides verbosity and sets it to false.' ) 00285 devel.add_option('--no_exec' , action="store_true", dest='noexec' , 00286 help = 'Run the suite without executing the cmsRelvalreport.py commands in the various directories. This is a useful debugging tool.' ) 00287 parser.add_option_group(devel) 00288 (options, args) = parser.parse_args(argslist) 00289 00290 00291 self._debug = options.debug 00292 self._unittest = options.unittest 00293 self._noexec = options.noexec 00294 self._verbose = options.verbose 00295 self._dryrun = options.dryrun 00296 castordir = options.castordir 00297 TimeSizeEvents = options.TimeSizeEvents 00298 IgProfEvents = options.IgProfEvents 00299 CallgrindEvents = options.CallgrindEvents 00300 MemcheckEvents = options.MemcheckEvents 00301 cmsScimark = options.cmsScimark 00302 cmsScimarkLarge = options.cmsScimarkLarge 00303 cmsdriverOptions = options.cmsdriverOptions 00304 stepOptions = options.stepOptions 00305 quicktest = options.quicktest 00306 #candleoption = options.candleOptions 00307 runonspare = options.runonspare 00308 profilers = options.profilers.strip() 00309 cpu = options.cpu.strip() 00310 bypasshlt = options.bypasshlt 00311 cores = options.cores 00312 logfile = options.logfile 00313 prevrel = options.previousrel 00314 outputdir = options.outputdir 00315 RunTimeSize = options.RunTimeSize 00316 RunIgProf = options.RunIgProf 00317 RunCallgrind = options.RunCallgrind 00318 RunMemcheck = options.RunMemcheck 00319 RunDigiPileUp = options.RunDigiPileUp 00320 RunTimeSizePU = options.RunTimeSizePU 00321 RunIgProfPU = options.RunIgProfPU 00322 RunCallgrindPU = options.RunCallgrindPU 00323 RunMemcheckPU = options.RunMemcheckPU 00324 PUInputFile = options.PUInputFile 00325 userInputFile = options.userInputFile 00326 if options.MailLogRecipients !="" and self.user not in options.MailLogRecipients: #To allow for the --mail "" case of suppressing the email and the default user case 00327 MailLogRecipients= self.user+","+options.MailLogRecipients #Add the user by default if there is a mail report 00328 else: 00329 MailLogRecipients=options.MailLogRecipients 00330 tarball = options.tarball 00331 00332 ################# 00333 # Check logfile option 00334 # 00335 if not logfile == None: 00336 logfile = os.path.abspath(logfile) 00337 logdir = os.path.dirname(logfile) 00338 if not os.path.exists(logdir): 00339 parser.error("Directory to output logfile does not exist") 00340 sys.exit() 00341 logfile = os.path.abspath(logfile) 00342 00343 ############# 00344 # Check step Options 00345 # 00346 if "GEN,SIM" in stepOptions: 00347 self.logh.write("WARNING: Please use GEN-SIM with a hypen not a \",\"!\n") 00348 #Using the step option as a switch between different dictionaries for: 00349 #RunTimeSize,RunIgProf,RunCallgrind,RunMemCheck,RunDigiPileUp: 00350 if stepOptions == "" or stepOptions == 'Default': 00351 pass 00352 else: 00353 stepOptions='--usersteps=%s' % (stepOptions) 00354 00355 ############### 00356 # Check profile option 00357 # 00358 isnumreg = re.compile("^-?[0-9]*$") 00359 found = isnumreg.search(profilers) 00360 if not found : 00361 parser.error("profile codes option contains non-numbers") 00362 sys.exit() 00363 00364 ############### 00365 # Check output directory option 00366 # 00367 if outputdir == "": 00368 outputdir = os.getcwd() 00369 else: 00370 outputdir = os.path.abspath(outputdir) 00371 00372 if not os.path.isdir(outputdir): 00373 parser.error("%s is not a valid output directory" % outputdir) 00374 sys.exit() 00375 00376 ################ 00377 # Check cpu option 00378 # 00379 numetcomreg = re.compile("^[0-9,]*") 00380 if not numetcomreg.search(cpu): 00381 parser.error("cpu option needs to be a comma separted list of ints or a single int") 00382 sys.exit() 00383 00384 cpustr = cpu 00385 cpu = [] 00386 if "," in cpustr: 00387 cpu = map(lambda x: int(x),cpustr.split(",")) 00388 else: 00389 cpu = [ int(cpustr) ] 00390 00391 ################ 00392 # Check previous release directory 00393 # 00394 if not prevrel == "": 00395 prevrel = os.path.abspath(prevrel) 00396 if not os.path.exists(prevrel): 00397 self.logh.write("ERROR: Previous release dir %s could not be found" % prevrel) 00398 sys.exit() 00399 00400 ############# 00401 # Setup quicktest option 00402 # 00403 if quicktest: 00404 TimeSizeEvents = 1 00405 IgProfEvents = 1 00406 CallgrindEvents = 0 00407 MemcheckEvents = 0 00408 cmsScimark = 1 00409 cmsScimarkLarge = 1 00410 00411 ############# 00412 # Setup unit test option 00413 # 00414 if self._unittest: 00415 self._verbose = False 00416 if stepOptions == "": 00417 stepOptions = "GEN-SIM,DIGI,L1,DIGI2RAW,HLT,RAW2DIGI-RECO" 00418 cmsScimark = 0 00419 cmsScimarkLarge = 0 00420 CallgrindEvents = 0 00421 MemcheckEvents = 0 00422 IgProfEvents = 0 00423 TimeSizeEvents = 1 00424 00425 #Split all the RunTimeSize etc candles in lists: 00426 TimeSizeCandles=[] 00427 IgProfCandles=[] 00428 CallgrindCandles=[] 00429 MemcheckCandles=[] 00430 TimeSizePUCandles=[] 00431 IgProfPUCandles=[] 00432 CallgrindPUCandles=[] 00433 MemcheckPUCandles=[] 00434 userInputRootFiles=[] 00435 if RunTimeSize: 00436 TimeSizeCandles = RunTimeSize.split(",") 00437 if RunIgProf: 00438 IgProfCandles = RunIgProf.split(",") 00439 if RunCallgrind: 00440 CallgrindCandles = RunCallgrind.split(",") 00441 if RunMemcheck: 00442 MemcheckCandles = RunMemcheck.split(",") 00443 if RunDigiPileUp: 00444 for candle in RunDigiPileUp.split(","): 00445 if candle in TimeSizeCandles: 00446 TimeSizePUCandles.append(candle) 00447 if candle in IgProfCandles: 00448 IgProfPUCandles.append(candle) 00449 if candle in CallgrindCandles: 00450 CallgrindPUCandles.append(candle) 00451 if candle in MemcheckCandles: 00452 MemcheckPUCandles.append(candle) 00453 if RunTimeSizePU: 00454 TimeSizePUCandles.extend(RunTimeSizePU.split(",")) 00455 #Some smart removal of duplicates from the list! 00456 temp=set(TimeSizePUCandles) 00457 TimeSizePUCandles=list(temp) #Doing it in 2 steps to avoid potential issues with type of arguments 00458 if RunIgProfPU: 00459 IgProfPUCandles.extend(RunIgProfPU.split(",")) 00460 #Some smart removal of duplicates from the list! 00461 temp=set(IgProfPUCandles) 00462 IgProfPUCandles=list(temp) #Doing it in 2 steps to avoid potential issues with type of arguments 00463 if RunCallgrindPU: 00464 CallgrindPUCandles.extend(RunCallgrindPU.split(",")) 00465 #Some smart removal of duplicates from the list! 00466 temp=set(CallgrindPUCandles) 00467 CallgrindPUCandles=list(temp) #Doing it in 2 steps to avoid potential issues with type of arguments 00468 if RunMemcheckPU: 00469 MemcheckPUCandles.extend(RunMemcheckPU.split(",")) 00470 #Some smart removal of duplicates from the list! 00471 temp=set(MemcheckPUCandles) 00472 MemcheckPUCandles=list(temp) #Doing it in 2 steps to avoid potential issues with type of arguments 00473 if userInputFile: 00474 userInputRootFiles=userInputFile.split(",") 00475 00476 00477 00478 ############# 00479 # Setup cmsdriver and eventual cmsdriverPUoption 00480 # 00481 cmsdriverPUOptions="" 00482 if cmsdriverOptions: 00483 #Set the eventual Pile Up cmsdriver options first: 00484 if TimeSizePUCandles or IgProfPUCandles or CallgrindPUCandles or MemcheckPUCandles: 00485 #Bug fixed: no space between --pileup= and LowLumiPileUp (otherwise could omit the =) 00486 cmsdriverPUOptions = '--cmsdriver="%s %s%s"'%(cmsdriverOptions," --pileup=",cmsDriverPileUpOption) 00487 #Set the regular ones too: 00488 cmsdriverOptions = '--cmsdriver="%s"'%cmsdriverOptions 00489 00490 return (castordir , 00491 TimeSizeEvents , 00492 IgProfEvents , 00493 CallgrindEvents , 00494 MemcheckEvents , 00495 cmsScimark , 00496 cmsScimarkLarge , 00497 cmsdriverOptions, 00498 cmsdriverPUOptions, 00499 stepOptions , 00500 quicktest , 00501 profilers , 00502 cpu , 00503 cores , 00504 prevrel , 00505 bypasshlt , 00506 runonspare , 00507 outputdir , 00508 logfile , 00509 TimeSizeCandles , 00510 IgProfCandles , 00511 CallgrindCandles, 00512 MemcheckCandles , 00513 TimeSizePUCandles , 00514 IgProfPUCandles , 00515 CallgrindPUCandles, 00516 MemcheckPUCandles , 00517 PUInputFile , 00518 userInputRootFiles, 00519 MailLogRecipients, 00520 tarball)
| def cmsPerfSuite::PerfSuite::printDate | ( | self | ) |
Definition at line 582 of file cmsPerfSuite.py.
| def cmsPerfSuite::PerfSuite::printFlush | ( | self, | |
| command | |||
| ) |
Print and flush a string (for output to a log file)
Definition at line 545 of file cmsPerfSuite.py.
| def cmsPerfSuite::PerfSuite::runcmd | ( | self, | |
| command | |||
| ) |
Run a command and return the exit status.
Definition at line 553 of file cmsPerfSuite.py.
00554 : 00555 #Substitute popen with subprocess.Popen! 00556 #Using try/except until Popen becomes thread safe (it seems that everytime it is called 00557 #all processes are checked to reap the ones that are done, this creates a race condition with the wait()... that 00558 #results into an error with "No child process". 00559 #os.popen(command) 00560 try: 00561 process = subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT) 00562 pid=process.pid 00563 exitstat= process.wait() 00564 cmdout = process.stdout.read() 00565 exitstat = process.returncode 00566 except OSError, detail: 00567 self.logh.write("Race condition in subprocess.Popen has robbed us of the exit code of the %s process (PID %s).Assume it failed!\n %s\n"%(command,pid,detail)) 00568 self.logh.flush() 00569 exitstat=999 00570 cmdout="Race condition in subprocess.Popen has robbed us of the exit code of the %s process (PID %s).Assume it failed!\n %s"%(command,pid,detail) 00571 if self._verbose: 00572 self.logh.write(cmdout)# + "\n") No need of extra \n! 00573 self.logh.flush() 00574 if exitstat == None: 00575 self.logh.write("Something strange is going on! Exit code was None for command %s: check if it really ran!"%command) 00576 self.logh.flush() 00577 exitstat=0 00578 return exitstat
| def cmsPerfSuite::PerfSuite::runCmdSet | ( | self, | |
| cmd | |||
| ) |
Run a list of commands using system ! We should rewrite this not to use system (most cases it is unnecessary)
Definition at line 527 of file cmsPerfSuite.py.
00528 : 00529 exitstat = 0 00530 if len(cmd) <= 1: 00531 exitstat = self.runcmd(cmd) 00532 if self._verbose: 00533 self.printFlush(cmd) 00534 else: 00535 for subcmd in cmd: 00536 if self._verbose: 00537 self.printFlush(subcmd) 00538 exitstat = self.runcmd(" && ".join(cmd)) 00539 if self._verbose: 00540 self.printFlush(self.getDate()) 00541 return exitstat
| def cmsPerfSuite::PerfSuite::runCmsInput | ( | self, | |
| cpu, | |||
| dir, | |||
| numevents, | |||
| candle, | |||
| cmsdrvopts, | |||
| stepopt, | |||
| profiles, | |||
| bypasshlt, | |||
| userInputFile | |||
| ) |
Wrapper for cmsRelvalreportInput.
Definition at line 750 of file cmsPerfSuite.py.
00751 : 00752 00753 #Crappy fix for optional options with special synthax (bypasshlt and userInputFile) 00754 bypass = "" 00755 if bypasshlt: 00756 bypass = "--bypass-hlt" 00757 userInputFileOption="" 00758 if userInputFile: 00759 userInputFileOption = "--filein %s"%userInputFile 00760 cmd = self.Commands[cpu][2] 00761 cmds=[] 00762 #print cmds 00763 cmds = ("cd %s" % (dir), 00764 "%s %s \"%s\" %s %s %s %s %s" % (cmd, 00765 numevents, 00766 candle, 00767 profiles, 00768 cmsdrvopts, 00769 stepopt, 00770 bypass,userInputFileOption)) 00771 exitstat=0 00772 exitstat = self.runCmdSet(cmds) 00773 if self._unittest and (not exitstat == 0): 00774 self.logh.write("ERROR: CMS Report Input returned a non-zero exit status \n" ) return exitstat
| def cmsPerfSuite::PerfSuite::runCmsReport | ( | self, | |
| cpu, | |||
| dir, | |||
| candle | |||
| ) |
This function is a wrapper around cmsRelvalreport.
Definition at line 700 of file cmsPerfSuite.py.
00701 : 00702 cmd = self.Commands[cpu][1] 00703 cmds = ("cd %s" % (dir), 00704 "%s -i SimulationCandles_%s.txt -t perfreport_tmp -R -P >& %s.log" % (cmd,self.cmssw_version,candle)) 00705 exitstat = 0 00706 if not self._debug: 00707 exitstat = self.runCmdSet(cmds) 00708 00709 if self._unittest and (not exitstat == 0): 00710 self.logh.write("ERROR: CMS Report returned a non-zero exit status \n") 00711 sys.exit(exitstat) 00712 else: 00713 return(exitstat) #To return the exit code of the cmsRelvalreport.py commands to the runPerfSuite function
| def cmsPerfSuite::PerfSuite::runPerfSuite | ( | self, | |
castordir = "/castor/cern.ch/cms/store/relval/performance/", |
|||
TimeSizeEvents = 100, |
|||
IgProfEvents = 5, |
|||
CallgrindEvents = 1, |
|||
MemcheckEvents = 5, |
|||
cmsScimark = 10, |
|||
cmsScimarkLarge = 10, |
|||
cmsdriverOptions = "", |
|||
cmsdriverPUOptions = "", |
|||
stepOptions = "", |
|||
quicktest = False, |
|||
profilers = "", |
|||
cpus = [1], |
|||
cores = 4, |
|||
prevrel = "", |
|||
bypasshlt = False, |
|||
runonspare = True, |
|||
perfsuitedir = os.getcwd(), |
|||
logfile = None, |
|||
TimeSizeCandles = "", |
|||
IgProfCandles = "", |
|||
CallgrindCandles = "", |
|||
MemcheckCandles = "", |
|||
TimeSizePUCandles = "", |
|||
IgProfPUCandles = "", |
|||
CallgrindPUCandles = "", |
|||
MemcheckPUCandles = "", |
|||
PUInputFile = "", |
|||
userInputFile = "", |
|||
MailLogRecipients = "", |
|||
tarball = "" |
|||
| ) |
Runs benchmarking, cpu spinlocks on spare cores and profiles selected candles.
FIXME: Could redesign interface of functions to use keyword arguments: def runPerfSuite(**opts): then instead of using castordir variable, would use opts['castordir'] etc
Definition at line 877 of file cmsPerfSuite.py.
00909 : 00910 00911 #Set up a variable for the FinalExitCode to be used as the sum of exit codes: 00912 FinalExitCode=0 00913 00914 #Set up the logfile first! 00915 if not logfile == None: 00916 try: 00917 self.logh = open(logfile,"a") 00918 except (OSError, IOError), detail: 00919 self.logh.write(detail + "\n") 00920 self.logh.flush() 00921 00922 #Adding HEPSPEC06 score if available in /build/HEPSPEC06.score file 00923 self.HEPSPEC06 = 0 #Set it to 0 by default (so it is easy to catch in the DB too) 00924 try: 00925 HEPSPEC06_file=open("/build/HEPSPEC06.score","r") 00926 for line in HEPSPEC06_file.readlines(): 00927 if not line.startswith("#") and "HEPSPEC06" in line: 00928 self.HEPSPEC06= line.split()[2] 00929 except IOError: 00930 self.logh.write("***Warning***: Could not find file /build/HEPSPEC06.score file on this machine!\n") 00931 self.logh.flush() 00932 00933 #Adding a copy of /proc/cpuinfo and /proc/meminfo in the working directory so it can be kept in the tarball on CASTOR: 00934 localcpuinfo=os.path.join(perfsuitedir,"cpuinfo") 00935 cpuinfo_exitcode=-1 00936 if os.path.exists(localcpuinfo): 00937 cpuinfo_exitcode=0 00938 else: 00939 self.logh.write("Copying /proc/cpuinfo in current working directory (%s)\n"%perfsuitedir) 00940 cpuinfo_exitcode=self.runcmd("cp /proc/cpuinfo %s"%perfsuitedir) 00941 localmeminfo=os.path.join(perfsuitedir,"meminfo") 00942 meminfo_exitcode=-1 00943 if os.path.exists(localmeminfo): 00944 meminfo_exitcode=0 00945 else: 00946 self.logh.write("Copying /proc/meminfo in current working directory (%s)\n"%perfsuitedir) 00947 meminfo_exitcode=self.runcmd("cp /proc/meminfo %s"%perfsuitedir) 00948 if cpuinfo_exitcode or meminfo_exitcode: 00949 self.logh.write("There was an issue copying the cpuinfo or meminfo files!\n") 00950 self.logh.flush() 00951 00952 try: 00953 if not prevrel == "": 00954 self.logh.write("Production of regression information has been requested with release directory %s\n" % prevrel) 00955 if not cmsdriverOptions == "": 00956 self.logh.write("Running cmsDriver.py with user defined options: %s\n" % cmsdriverOptions) 00957 #Attach the full option synthax for cmsRelvalreportInput.py: 00958 cmsdriverOptionsRelvalInput="--cmsdriver="+cmsdriverOptions 00959 #FIXME: should import cmsRelvalreportInput.py and avoid these issues... 00960 if not stepOptions == "": 00961 self.logh.write("Running user defined steps only: %s\n" % stepOptions) 00962 #Attach the full option synthax for cmsRelvalreportInput.py: 00963 setpOptionsRelvalInput="--usersteps="+stepOptions 00964 #FIXME: should import cmsRelvalreportInput.py and avoid these issues... 00965 if bypasshlt: 00966 #Attach the full option synthax for cmsRelvalreportInput.py: 00967 bypasshltRelvalInput="--bypass-hlt" 00968 #FIXME: should import cmsRelvalreportInput.py and avoid these issues... 00969 self.logh.write("Current Architecture is %s\n"%self.cmssw_arch) 00970 self.logh.write("Current CMSSW version is %s\n"%self.cmssw_version) 00971 self.logh.write("This machine ( %s ) is assumed to have %s cores, and the suite will be run on cpu %s\n" %(self.host,cores,cpus)) 00972 self.logh.write("This machine's HEPSPEC06 score is: %s \n"%self.HEPSPEC06) 00973 path=os.path.abspath(".") 00974 self.logh.write("Performance Suite started running at %s on %s in directory %s, run by user %s\n" % (self.getDate(),self.host,path,self.user)) 00975 #Start the timer for the total performance suite running time: 00976 TotalTime=PerfSuiteTimer(start=datetime.datetime.now()) 00977 #Also initialize the dictionary that will contain all the timing information: 00978 global TimerInfo 00979 TimerInfo={'TotalTime':{'TotalTime':TotalTime}} #Structure will be {'key':[PerfSuiteTimerInstance,...],...} 00980 #Obsolete popen4-> subprocess.Popen 00981 #showtags=os.popen4("showtags -r")[1].read() 00982 showtags=subprocess.Popen("showtags -r",shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read() 00983 self.logh.write(showtags) # + "\n") No need for extra \n! 00984 self.logh.flush() 00985 #For the log: 00986 if self._verbose: 00987 self.logh.write("The performance suite results tarball will be stored in CASTOR at %s\n" % self._CASTOR_DIR) 00988 self.logh.write("%s TimeSize events\n" % TimeSizeEvents) 00989 self.logh.write("%s IgProf events\n" % IgProfEvents) 00990 self.logh.write("%s Callgrind events\n" % CallgrindEvents) 00991 self.logh.write("%s Memcheck events\n" % MemcheckEvents) 00992 self.logh.write("%s cmsScimark benchmarks before starting the tests\n" % cmsScimark) 00993 self.logh.write("%s cmsScimarkLarge benchmarks before starting the tests\n" % cmsScimarkLarge) 00994 self.logh.flush() 00995 #Actual script actions! 00996 #Will have to fix the issue with the matplotlib pie-charts: 00997 #Used to source /afs/cern.ch/user/d/dpiparo/w0/perfreport2.1installation/share/perfreport/init_matplotlib.sh 00998 #Need an alternative in the release 00999 01000 #Code for the architecture benchmarking use-case 01001 if len(cpus) > 1: 01002 for cpu in cpus: 01003 cpupath = os.path.join(perfsuitedir,"cpu_%s" % cpu) 01004 if not os.path.exists(cpupath): 01005 os.mkdir(cpupath) 01006 01007 self.Commands = {} 01008 AllScripts = self.Scripts + self.AuxiliaryScripts 01009 01010 for cpu in range(cmsCpuInfo.get_NumOfCores()): #FIXME use the actual number of cores of the machine here! 01011 self.Commands[cpu] = [] 01012 01013 #Information for the log: 01014 self.logh.write("Full path of all the scripts used in this run of the Performance Suite:\n") 01015 for script in AllScripts: 01016 which="which " + script 01017 01018 #Logging the actual version of cmsDriver.py, cmsRelvalreport.py, cmsSimPyRelVal.pl 01019 #Obsolete popen4-> subprocess.Popen 01020 #whichstdout=os.popen4(which)[1].read() 01021 whichstdout=subprocess.Popen(which,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read() 01022 self.logh.write(whichstdout) # + "\n") No need of the extra \n! 01023 if script in self.Scripts: 01024 for cpu in range(cmsCpuInfo.get_NumOfCores()):#FIXME use the actual number of cores of the machine here! 01025 command="taskset -c %s %s" % (cpu,script) 01026 self.Commands[cpu].append(command) 01027 01028 #First submit the cmsScimark benchmarks on the unused cores: 01029 scimark = "" 01030 scimarklarge = "" 01031 if not (self._unittest or self._noexec): 01032 for core in range(cores): 01033 if (not core in cpus) and runonspare: 01034 self.logh.write("Submitting cmsScimarkLaunch.csh to run on core cpu "+str(core) + "\n") 01035 subcmd = "cd %s ; cmsScimarkLaunch.csh %s" % (perfsuitedir, str(core)) 01036 command="taskset -c %s sh -c \"%s\" &" % (str(core), subcmd) 01037 self.logh.write(command + "\n") 01038 01039 #cmsScimarkLaunch.csh is an infinite loop to spawn cmsScimark2 on the other 01040 #cpus so it makes no sense to try reading its stdout/err 01041 #Obsolete popen4-> subprocess.Popen 01042 #os.popen4(command) 01043 subprocess.Popen(command,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT) 01044 01045 self.logh.flush() 01046 01047 #Don't do benchmarking if in debug mode... saves time 01048 benching = not self._debug 01049 ##FIXME: 01050 #We may want to introduce a switch here or agree on a different default (currently 10 cmsScimark and 10 cmsScimarkLarge) 01051 if benching and not (self._unittest or self._noexec): 01052 #Submit the cmsScimark benchmarks on the cpu where the suite will be run: 01053 for cpu in cpus: 01054 scimark = open(os.path.join(perfsuitedir,"cmsScimark2.log") ,"w") 01055 scimarklarge = open(os.path.join(perfsuitedir,"cmsScimark2_large.log"),"w") 01056 if cmsScimark > 0: 01057 self.logh.write("Starting with %s cmsScimark on cpu%s\n" % (cmsScimark,cpu)) 01058 cmsScimarkInitialTime=PerfSuiteTimer(start=datetime.datetime.now()) #Create the cmsScimark PerfSuiteTimer 01059 TimerInfo.update({'cmsScimarkTime':{'cmsScimarkInitial':cmsScimarkInitialTime}}) #Add the cmsScimarkInitialTime information to the general TimerInfo dictionary 01060 self.benchmarks(cpu,perfsuitedir,scimark.name,cmsScimark) 01061 cmsScimarkInitialTime.set_end(datetime.datetime.now()) #Stop the cmsScimark initial timer 01062 01063 if cmsScimarkLarge > 0: 01064 self.logh.write("Following with %s cmsScimarkLarge on cpu%s\n" % (cmsScimarkLarge,cpu)) 01065 cmsScimarkLargeInitialTime=PerfSuiteTimer(start=datetime.datetime.now()) #Create the cmsScimarkLarge PerfSuiteTimer 01066 TimerInfo['cmsScimarkTime'].update({'cmsScimarkLargeInitial':cmsScimarkLargeInitialTime}) #Add the cmsScimarkLargeInitialTime information to the general TimerInfo dictionary 01067 self.benchmarks(cpu,perfsuitedir,scimarklarge.name,cmsScimarkLarge, large=True) 01068 cmsScimarkLargeInitialTime.set_end(datetime.datetime.now()) #Stop the cmsScimarkLarge Initial timer 01069 self.logh.flush() 01070 #Handling the Pile up input file here: 01071 if (TimeSizePUCandles or IgProfPUCandles or CallgrindPUCandles or MemcheckPUCandles) and not ("FASTSIM" in stepOptions): 01072 #Note the FASTSIM exclusion... since there is no need to copy the file for FASTSIM. 01073 PUInputName=os.path.join(perfsuitedir,"INPUT_PILEUP_EVENTS.root") 01074 if PUInputFile: 01075 #Define the actual command to copy the file locally: 01076 #Allow the file to be mounted locally (or accessible via AFS) 01077 copycmd="cp" 01078 #Allow the file to be on CASTOR (taking a full CASTOR path) 01079 if '/store/relval/' in PUInputFile: 01080 copycmd="rfcp" 01081 #Accept plain LFNs from DBS for RelVal CASTOR files: 01082 #Minor fix to allow the case of user using the full path /castor/cern.ch/cms... 01083 if PUInputFile.startswith('/store/relval/'): 01084 PUInputFile="/castor/cern.ch/cms"+PUInputFile 01085 #Copy the file locally 01086 self.logh.write("Copying the file %s locally to %s\n"%(PUInputFile,PUInputName)) 01087 self.logh.flush() 01088 GetPUInput=subprocess.Popen("%s %s %s"%(copycmd,PUInputFile,PUInputName), shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) 01089 GetPUInputExitCode=GetPUInput.wait() 01090 #Allow even the potential copy of a local file (even one already named INPUT_PILEUP_EVENTS.root!) 01091 if GetPUInputExitCode: 01092 self.logh.write("The copying of the pile-up input file returned a non-zero exit code: %s \nThis is the stdout+stderr if the command:\n%s\n"%(GetPUInputExitCode,GetPUInput.stdout)) 01093 #Ultimately accept the case of the file being already there and not being specified in the --PUInputFile option 01094 if not os.path.exists(PUInputName): 01095 self.logh.write("The necessary INPUT_PILEUP_EVENTS.root file was not found in the working directory %s\nExiting now!"%perfsuitedir) 01096 self.logh.flush() 01097 sys.exit(1) 01098 else: 01099 #Set up here the DIGI PILE UP options 01100 self.printFlush("Some PILE UP tests will be run!") 01101 #Actually setting them earlier... when handling options... May not need this else after all... or just as a log entry. 01102 self.printFlush("cmsdriverPUOptions is %s"%cmsdriverPUOptions) 01103 pass 01104 01105 #TimeSize tests: 01106 if TimeSizeEvents > 0: 01107 TimeSizeTime=PerfSuiteTimer(start=datetime.datetime.now()) #Start the TimeSize timer 01108 TimerInfo.update({'TimeSize':{'TotalTime':TimeSizeTime}}) #Add the TimeSize timer to the dictionary 01109 if TimeSizeCandles: 01110 self.logh.write("Launching the TimeSize tests (TimingReport, TimeReport, SimpleMemoryCheck, EdmSize) with %s events each\n" % TimeSizeEvents) 01111 NoPileUpTime=PerfSuiteTimer(start=datetime.datetime.now()) #Start the TimeSize timer 01112 TimerInfo['TimeSize'].update({'NoPileUpTime':NoPileUpTime}) #Add the TimeSize No Pile Up tests timer to the list 01113 self.printDate() 01114 self.logh.flush() 01115 ReportExit=self.simpleGenReport(cpus,perfsuitedir,TimeSizeEvents,TimeSizeCandles,cmsdriverOptions,stepOptions,"TimeSize",profilers,bypasshlt,userInputFile) 01116 FinalExitCode=FinalExitCode+ReportExit 01117 #Adding a time stamp here to parse for performance suite running time data 01118 self.printFlush("Regular TimeSize tests were finished at %s"%(self.getDate())) 01119 NoPileUpTime.set_end(datetime.datetime.now()) #Stop TimeSize timer 01120 01121 #Launch eventual Digi Pile Up TimeSize too: 01122 if TimeSizePUCandles: 01123 self.logh.write("Launching the PILE UP TimeSize tests (TimingReport, TimeReport, SimpleMemoryCheck, EdmSize) with %s events each\n" % TimeSizeEvents) 01124 PileUpTime=PerfSuiteTimer(start=datetime.datetime.now()) #Start the TimeSize timer 01125 TimerInfo['TimeSize'].update({'PileUpTime':PileUpTime}) #Add the TimeSize Pile Up tests timer to the list 01126 self.printDate() 01127 self.logh.flush() 01128 ReportExit=self.simpleGenReport(cpus,perfsuitedir,TimeSizeEvents,TimeSizePUCandles,cmsdriverPUOptions,stepOptions,"TimeSize",profilers,bypasshlt,userInputFile) 01129 FinalExitCode=FinalExitCode+ReportExit 01130 #Adding a time stamp here to parse for performance suite running time data 01131 self.printFlush("Pileup TimeSize tests were finished at %s"%(self.getDate())) 01132 PileUpTime.set_end(datetime.datetime.now()) #Stop TimeSize timer 01133 01134 #Check for issue with 01135 if not (TimeSizeCandles or TimeSizePUCandles): 01136 self.printFlush("A number of events (%s) for TimeSize tests was selected, but no candle for regular or pileup tests was selected!"%(TimeSizeEvents)) 01137 #Adding a time stamp here to parse for performance suite running time data 01138 self.printFlush("All TimeSize tests were finished at %s"%(self.getDate())) 01139 TimeSizeTime.set_end(datetime.datetime.now()) #Stop TimeSize timer 01140 01141 #Stopping all cmsScimark jobs and analysing automatically the logfiles 01142 #No need to waste CPU while the load does not affect Valgrind measurements! 01143 if not (self._unittest or self._noexec): 01144 self.logh.write("Stopping all cmsScimark jobs now\n") 01145 subcmd = "cd %s ; %s" % (perfsuitedir,self.AuxiliaryScripts[2]) 01146 stopcmd = "sh -c \"%s\"" % subcmd 01147 self.printFlush(stopcmd) 01148 #os.popen(stopcmd) 01149 #Obsolete popen4-> subprocess.Popen 01150 #self.printFlush(os.popen4(stopcmd)[1].read()) 01151 self.printFlush(subprocess.Popen(stopcmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read()) 01152 01153 #From here on we can use all available cores to speed up the performance suite remaining tests: 01154 if cores==0: #When specifying the cpu to run the suite on, one has to set cores to 0 to avoid threading of PerfSuite itself... 01155 #So we need to catch this case for the IB tests case where we assign the test to a specific cpu. 01156 AvailableCores=cpus 01157 else: 01158 AvailableCores=range(cores) 01159 #Initialize a list that will contain all the simpleGenReport keyword arguments (1 dictionary per test): 01160 TestsToDo=[] 01161 #IgProf tests: 01162 if IgProfEvents > 0: 01163 if IgProfCandles: 01164 self.printFlush("Preparing IgProf tests") 01165 #Special case for IgProf: user could pick with the option --profilers to run only IgProf perf or Mem (or Mem_Total alone etc) 01166 #So in general we want to be able to split the perf and mem tests... 01167 #For the case of --profiler option we will run only 1 test (i.e. it will get one core slot until it is done with whatever profiling choosen) 01168 if profilers: 01169 self.printFlush("Special profiler option for IgProf was indicated by the user: %s"%profilers) 01170 #Prepare the simpleGenReport arguments for this test: 01171 IgProfProfilerArgs={ 01172 'perfdir':perfsuitedir, 01173 'NumEvents':IgProfEvents, 01174 'candles':IgProfCandles, 01175 'cmsdriverOptions':cmsdriverOptions, 01176 'stepOptions':stepOptions, 01177 'Name':"IgProf", 01178 'profilers':profilers, 01179 'bypasshlt':bypasshlt, 01180 'userInputRootFiles':userInputFile 01181 } 01182 #Append the test to the TestsToDo list: 01183 TestsToDo.append(IgProfProfilerArgs) 01184 self.printFlush("Appended IgProf test with profiler option %s to the TestsToDo list"%profilers) 01185 #For the default case (4,5,6,7) we split the tests into 2 jobs since they naturally are 2 cmsRun jobs and for machines with many cores this will 01186 #make the performance suite run faster. 01187 else: 01188 self.printFlush("Splitting the IgProf tests into Perf and Mem to parallelize the cmsRun execution as much as possible:") 01189 ##PERF## 01190 #Prepare the simpleGenReport arguments for this test: 01191 IgProfPerfArgs={ 01192 'perfdir':perfsuitedir, 01193 'NumEvents':IgProfEvents, 01194 'candles':IgProfCandles, 01195 'cmsdriverOptions':cmsdriverOptions, 01196 'stepOptions':stepOptions, 01197 'Name':"IgProf_Perf", 01198 'profilers':profilers, 01199 'bypasshlt':bypasshlt, 01200 'userInputRootFiles':userInputFile 01201 } 01202 #Append the test to the TestsToDo list: 01203 TestsToDo.append(IgProfPerfArgs) 01204 self.printFlush("Appended IgProf PERF test to the TestsToDo list") 01205 ##MEM## 01206 #Prepare the simpleGenReport arguments for this test: 01207 IgProfMemArgs={ 01208 'perfdir':perfsuitedir, 01209 'NumEvents':IgProfEvents, 01210 'candles':IgProfCandles, 01211 'cmsdriverOptions':cmsdriverOptions, 01212 'stepOptions':stepOptions, 01213 'Name':"IgProf_Mem", 01214 'profilers':profilers, 01215 'bypasshlt':bypasshlt, 01216 'userInputRootFiles':userInputFile 01217 } 01218 #Append the test to the TestsToDo list: 01219 TestsToDo.append(IgProfMemArgs) 01220 self.printFlush("Appended IgProf MEM test to the TestsToDo list") 01221 #The following will be handled in the while loop that handles the starting of the threads: 01222 #ReportExit=self.simpleGenReport(cpus,perfsuitedir,IgProfEvents,IgProfCandles,cmsdriverOptions,stepOptions,"IgProf",profilers,bypasshlt,userInputFile) 01223 #FinalExitCode=FinalExitCode+ReportExit 01224 #Launch eventual Digi Pile Up IgProf too: 01225 if IgProfPUCandles: 01226 self.printFlush("Preparing IgProf PileUp tests") 01227 #Special case for IgProf: user could pick with the option --profilers to run only IgProf perf or Mem (or Mem_Total alone etc) 01228 #So in general we want to be able to split the perf and mem tests... 01229 #For the case of --profiler option we will run only 1 test (i.e. it will get one core slot until it is done with whatever profiling choosen) 01230 if profilers: 01231 self.printFlush("Special profiler option for IgProf was indicated by the user: %s"%profilers) 01232 #Prepare the simpleGenReport arguments for this test: 01233 IgProfProfilerPUArgs={ 01234 'perfdir':perfsuitedir, 01235 'NumEvents':IgProfEvents, 01236 'candles':IgProfPUCandles, 01237 'cmsdriverOptions':cmsdriverPUOptions, 01238 'stepOptions':stepOptions, 01239 'Name':"IgProf", 01240 'profilers':profilers, 01241 'bypasshlt':bypasshlt, 01242 'userInputRootFiles':userInputFile 01243 } 01244 #Append the test to the TestsToDo list: 01245 TestsToDo.append(IgProfProfilerPUArgs) 01246 self.printFlush("Appended IgProf PileUp test with profiler option %s to the TestsToDo list"%profilers) 01247 else: 01248 self.printFlush("Splitting the IgProf tests into Perf and Mem to parallelize the cmsRun execution as much as possible:") 01249 ##PERF## 01250 #Prepare the simpleGenReport arguments for this test: 01251 IgProfPerfPUArgs={ 01252 'perfdir':perfsuitedir, 01253 'NumEvents':IgProfEvents, 01254 'candles':IgProfPUCandles, 01255 'cmsdriverOptions':cmsdriverPUOptions, 01256 'stepOptions':stepOptions, 01257 'Name':"IgProf_Perf", 01258 'profilers':profilers, 01259 'bypasshlt':bypasshlt, 01260 'userInputRootFiles':userInputFile 01261 } 01262 #Append the test to the TestsToDo list: 01263 TestsToDo.append(IgProfPerfPUArgs) 01264 self.printFlush("Appended IgProf MEM PileUp test to the TestsToDo list") 01265 ##MEM## 01266 #Prepare the simpleGenReport arguments for this test: 01267 IgProfMemPUArgs={ 01268 'perfdir':perfsuitedir, 01269 'NumEvents':IgProfEvents, 01270 'candles':IgProfPUCandles, 01271 'cmsdriverOptions':cmsdriverPUOptions, 01272 'stepOptions':stepOptions, 01273 'Name':"IgProf_Mem", 01274 'profilers':profilers, 01275 'bypasshlt':bypasshlt, 01276 'userInputRootFiles':userInputFile 01277 } 01278 #Append the test to the TestsToDo list: 01279 TestsToDo.append(IgProfMemPUArgs) 01280 self.printFlush("Appended IgProf MEM PileUp test to the TestsToDo list") 01281 if not (IgProfCandles or IgProfPUCandles): 01282 self.printFlush("A number of events (%s) for IgProf tests was selected, but no candle for regular or pileup tests was selected!"%(IgProfEvents)) 01283 01284 01285 #Valgrind tests: 01286 if CallgrindEvents > 0: 01287 if CallgrindCandles: 01288 self.printFlush("Preparing Callgrind tests") 01289 CallgrindArgs={ 01290 'perfdir':perfsuitedir, 01291 'NumEvents':CallgrindEvents, 01292 'candles':CallgrindCandles, 01293 'cmsdriverOptions':cmsdriverOptions, 01294 'stepOptions':stepOptions, 01295 'Name':"Callgrind", 01296 'profilers':profilers, 01297 'bypasshlt':bypasshlt, 01298 'userInputRootFiles':userInputFile 01299 } 01300 #Append the test to the TestsToDo list: 01301 TestsToDo.append(CallgrindArgs) 01302 self.printFlush("Appended Callgrind test to the TestsToDo list") 01303 #Launch eventual Digi Pile Up Callgrind too: 01304 if CallgrindPUCandles: 01305 self.printFlush("Preparing Callgrind PileUp tests") 01306 CallgrindPUArgs={ 01307 'perfdir':perfsuitedir, 01308 'NumEvents':CallgrindEvents, 01309 'candles':CallgrindPUCandles, 01310 'cmsdriverOptions':cmsdriverPUOptions, 01311 'stepOptions':stepOptions, 01312 'Name':"Callgrind", 01313 'profilers':profilers, 01314 'bypasshlt':bypasshlt, 01315 'userInputRootFiles':userInputFile 01316 } 01317 #Append the test to the TestsToDo list: 01318 TestsToDo.append(CallgrindPUArgs) 01319 self.printFlush("Appended Callgrind PileUp test to the TestsToDo list") 01320 if not (CallgrindCandles or CallgrindPUCandles): 01321 self.printFlush("A number of events (%s) for Callgrind tests was selected, but no candle for regular or pileup tests was selected!"%(CallgrindEvents)) 01322 01323 if MemcheckEvents > 0: 01324 if MemcheckCandles: 01325 self.printFlush("Preparing Memcheck tests") 01326 MemcheckArgs={ 01327 'perfdir':perfsuitedir, 01328 'NumEvents':MemcheckEvents, 01329 'candles':MemcheckCandles, 01330 'cmsdriverOptions':cmsdriverOptions, 01331 'stepOptions':stepOptions, 01332 'Name':"Memcheck", 01333 'profilers':profilers, 01334 'bypasshlt':bypasshlt, 01335 'userInputRootFiles':userInputFile 01336 } 01337 #Append the test to the TestsToDo list: 01338 TestsToDo.append(MemcheckArgs) 01339 self.printFlush("Appended Memcheck test to the TestsToDo list") 01340 #Launch eventual Digi Pile Up Memcheck too: 01341 if MemcheckPUCandles: 01342 self.printFlush("Preparing Memcheck PileUp tests") 01343 MemcheckPUArgs={ 01344 'perfdir':perfsuitedir, 01345 'NumEvents':MemcheckEvents, 01346 'candles':MemcheckPUCandles, 01347 'cmsdriverOptions':cmsdriverPUOptions, 01348 'stepOptions':stepOptions, 01349 'Name':"Memcheck", 01350 'profilers':profilers, 01351 'bypasshlt':bypasshlt, 01352 'userInputRootFiles':userInputFile 01353 } 01354 #Append the test to the TestsToDo list: 01355 TestsToDo.append(MemcheckPUArgs) 01356 self.printFlush("Appended Memcheck PileUp test to the TestsToDo list") 01357 if not (MemcheckCandles or MemcheckPUCandles): 01358 self.printFlush("A number of events (%s) for Memcheck tests was selected, but no candle for regular or pileup tests was selected!"%(MemcheckEvents)) 01359 01360 #Here if there are any IgProf, Callgrind or MemcheckEvents to be run, 01361 #run the infinite loop that submits the PerfTest() threads on the available cores: 01362 if IgProfEvents or CallgrindEvents or MemcheckEvents: 01363 #FIXME:We should consider what behavior makes most sense in case we use the --cores option at this time only the cores=0 care is considered... 01364 self.printFlush("Threading all remaining tests on all %s available cores!"%len(AvailableCores)) 01365 self.printDate() 01366 self.logh.flush() 01367 #Save the original AvailableCores list to use it as a test to break the infinite loop: 01368 #While in the regular RelVal use-case it makes sense to use the actual number of cores of the machines, in 01369 #the IB case the AvailableCores will always consist of only 1 single core.. 01370 OriginalAvailableCores=list(AvailableCores) #Tricky list copy bug! without the list() OriginalAvalaibleCores would point to AvailableCores! 01371 #Print this out in the log for debugging reasons 01372 self.printFlush("Original available cores list: %s"%AvailableCores) 01373 01374 #Create a dictionary to keep track of running threads on the various cores: 01375 activePerfTestThreads={} 01376 #Flag for waiting messages: 01377 Waiting=False 01378 while 1: 01379 #Check if there are tests to run: 01380 if TestsToDo: 01381 #Using the Waiting flag to avoid writing this message every 5 seconds in the case 01382 #of having more tests to do than available cores... 01383 if not Waiting: 01384 self.printFlush("Currently %s tests are scheduled to be run:"%len(TestsToDo)) 01385 self.printFlush(TestsToDo) 01386 #Check the available cores: 01387 if AvailableCores: 01388 #Set waiting flag to False since we'll be doing something 01389 Waiting=False 01390 self.printFlush("There is/are %s core(s) available"%len(AvailableCores)) 01391 cpu=AvailableCores.pop() 01392 self.printFlush("Let's use cpu %s"%cpu) 01393 simpleGenReportArgs=TestsToDo.pop() 01394 self.printFlush("Let's submit %s test on core %s"%(simpleGenReportArgs['Name'],cpu)) 01395 #Adding a Total timer for each of the threaded tests: 01396 if simpleGenReportArgs['Name'] not in TimerInfo.keys(): 01397 #if 'TotalTime' not in TimerInfo[simpleGenReportArgs['Name']].keys(): 01398 self.PerfTestTotalTimer=PerfSuiteTimer(start=datetime.datetime.now()) #Start the TimeSize timer 01399 TimerInfo.update({simpleGenReportArgs['Name']:{'TotalTime':self.PerfTestTotalTimer}}) #Add the TimeSize timer to the dictionary 01400 threadToDo=self.simpleGenReportThread(cpu,self,**simpleGenReportArgs) #Need to send self too, so that the thread has access to the PerfSuite.simpleGenReport() function 01401 self.printFlush("Starting thread %s"%threadToDo) 01402 ReportExitCode=threadToDo.start() 01403 self.printFlush("Adding thread %s to the list of active threads"%threadToDo) 01404 activePerfTestThreads[cpu]=threadToDo 01405 #If there is no available core, pass, there will be some checking of activeThreads, a little sleep and then another check. 01406 else: 01407 pass 01408 #Test activePerfThreads: 01409 activeTestNames=[] 01410 activeTestNamesPU=[] 01411 for cpu in activePerfTestThreads.keys(): 01412 if activePerfTestThreads[cpu].isAlive(): 01413 #print "%% cpu %s activerPerfTestThreads[cpu] %s activePerfTestThreads[cpu].simpleGenReportArgs['cmsdriverOptions'] %s"%(cpu,activePerfTestThreads[cpu],activePerfTestThreads[cpu].simpleGenReportArgs['cmsdriverOptions']) 01414 if "--pileup" in activePerfTestThreads[cpu].simpleGenReportArgs['cmsdriverOptions']: 01415 activeTestNamesPU.append(activePerfTestThreads[cpu].simpleGenReportArgs['Name']) 01416 else: 01417 activeTestNames.append(activePerfTestThreads[cpu].simpleGenReportArgs['Name']) 01418 pass 01419 elif cpu not in AvailableCores: 01420 #Set waiting flag to False since we'll be doing something 01421 Waiting=False 01422 self.printFlush(time.ctime()) 01423 self.printFlush("%s test, in thread %s is done running on core %s"%(activePerfTestThreads[cpu].simpleGenReportArgs['Name'],activePerfTestThreads[cpu],cpu) ) 01424 self.printFlush("About to append cpu %s to AvailableCores list"%cpu) 01425 AvailableCores.append(cpu) 01426 #Eliminate from activeTestNames lists: 01427 #print activeTestNames 01428 #print activeTestNamesPU 01429 #print activePerfTestThreads[cpu].simpleGenReportArgs['Name'] 01430 if "--pileup" in activePerfTestThreads[cpu].simpleGenReportArgs['cmsdriverOptions']: 01431 try: 01432 activeTestNamesPU.remove(activePerfTestThreads[cpu].simpleGenReportArgs['Name']) 01433 except: 01434 pass 01435 else: 01436 try: 01437 activeTestNames.remove(activePerfTestThreads[cpu].simpleGenReportArgs['Name']) 01438 except: 01439 pass 01440 #Eliminate also from activePErfTestThreads dictionary: 01441 activePerfTestThreads.pop(cpu) 01442 #FIXME: 01443 #Delicate check to stop the timer on the individual threaded test! 01444 #Need to thik about it still... 01445 #FIXME: 01446 #Delicate check to stop the timers on the threaded tests: 01447 #Check activePerfTestThreads dictionary for "Name" if any name is missing, the total can be stopped for that name. 01448 #self.PerfTestTotalTimer 01449 for TestName in ["IgProf_Perf","IgProf_Mem","Memcheck","Valgrind"]: 01450 if (TestName not in activeTestNames) and (TestName not in activeTestNamesPU) : 01451 try: 01452 TimerInfo[TestName]['TotalTime'].set_end(datetime.datetime.now()) 01453 except: 01454 #print "No %s test was running"%TestName 01455 pass 01456 #Buggy if... it seems we don't wait for the running thread to be finished... 01457 #We should request: 01458 #-All OriginalAvailableCores should be actually available. 01459 if not AvailableCores==[] and (set(AvailableCores)==set(range(cmsCpuInfo.get_NumOfCores())) or set(AvailableCores)==set(OriginalAvailableCores)) and not TestsToDo: 01460 self.printFlush("PHEW! We're done... all TestsToDo are done... at %s "%(self.getDate())) 01461 #Debug printouts: 01462 #print "AvailableCores",AvailableCores 01463 #print "set(AvailableCores)",set(AvailableCores) 01464 #print "set(range(cmsCpuInfo.get_NumOfCores())",set(range(cmsCpuInfo.get_NumOfCores())) 01465 #print "OriginalAvailableCores",OriginalAvailableCores 01466 #print "set(OriginalAvailableCores)",set(OriginalAvailableCores) 01467 #print "TestsToDo",TestsToDo 01468 break 01469 else: 01470 #Putting the sleep statement first to avoid writing Waiting... before the output of the started thread reaches the log... 01471 time.sleep(5) 01472 #Use Waiting flag to writing 1 waiting message while waiting and avoid having 1 message every 5 seconds... 01473 if not Waiting: 01474 self.printFlush(time.ctime()) 01475 self.printFlush("Waiting for tests to be done...") 01476 sys.stdout.flush() 01477 Waiting=True 01478 #End of the if for IgProf, Callgrind, Memcheck tests 01479 01480 if benching and not (self._unittest or self._noexec): 01481 #Ending the performance suite with the cmsScimark benchmarks again: 01482 for cpu in cpus: 01483 if cmsScimark > 0: 01484 self.logh.write("Ending with %s cmsScimark on cpu%s\n" % (cmsScimark,cpu)) 01485 cmsScimarkFinalTime=PerfSuiteTimer(start=datetime.datetime.now()) #Create the cmsScimark PerfSuiteTimer 01486 TimerInfo['cmsScimarkTime'].update({'cmsScimarkFinal':cmsScimarkFinalTime}) #Add the cmsScimarkFinalTime information to the general TimerInfo dictionary 01487 01488 self.benchmarks(cpu,perfsuitedir,scimark.name,cmsScimark) 01489 cmsScimarkFinalTime.set_end(datetime.datetime.now()) #Stop the cmsScimarkLarge Initial timer 01490 if cmsScimarkLarge > 0: 01491 self.logh.write("Following with %s cmsScimarkLarge on cpu%s\n" % (cmsScimarkLarge,cpu)) 01492 cmsScimarkLargeFinalTime=PerfSuiteTimer(start=datetime.datetime.now()) #Create the cmsScimarkLargePerfSuiteTimer 01493 TimerInfo['cmsScimarkTime'].update({'cmsScimarkLargeFinal':cmsScimarkLargeFinalTime}) #Add the cmsScimarkLargeFinalTime information to the general TimerInfo dictionary 01494 self.benchmarks(cpu,perfsuitedir,scimarklarge.name,cmsScimarkLarge,large=True) 01495 cmsScimarkLargeFinalTime.set_end(datetime.datetime.now()) #Stop the cmsScimarkLarge Initial timer 01496 01497 if prevrel: 01498 self.logh.write("Running the regression analysis with respect to %s\n"%getVerFromLog(prevrel)) 01499 self.logh.write(time.ctime(time.time())) 01500 self.logh.flush() 01501 01502 crr.regressReports(prevrel,os.path.abspath(perfsuitedir),oldRelName = getVerFromLog(prevrel),newRelName=self.cmssw_version) 01503 01504 #Create a tarball of the work directory 01505 if tarball: 01506 tarballTime=PerfSuiteTimer(start=datetime.datetime.now()) #Create the tarball PerfSuiteTimer 01507 TimerInfo.update({'tarballTime':{'TotalTime':tarballTime}}) 01508 # Adding the str(stepOptions to distinguish the tarballs for 1 release 01509 # (GEN->DIGI, L1->RECO will be run in parallel) 01510 01511 # Cleaning the stepOptions from the --usersteps=: 01512 if "=" in str(stepOptions): 01513 fileStepOption=str(stepOptions).split("=")[1] 01514 else: 01515 fileStepOption=str(stepOptions) 01516 if fileStepOption=="": 01517 fileStepOption="UnknownStep" 01518 # Add the working directory used to avoid overwriting castor files (also put a check...) 01519 fileWorkingDir=os.path.basename(perfsuitedir) 01520 01521 # Also add the --conditions and --eventcontent options used in the --cmsdriver options since it 01522 # is possible that the same tests will be run with different conditions and/or event content: 01523 # Parse it out of --cmsdriver option: 01524 fileEventContentOption="UnknownEventContent" 01525 fileConditionsOption="UnknownConditions" 01526 for token in cmsdriverOptions.split("--"): 01527 if token!='' and 'cmsdriver' not in token: 01528 if "=" in token: 01529 fileOption=token.split("=")[0] 01530 fileOptionValue=token.split("=")[1].strip("'").strip('"') 01531 else: 01532 fileOption=token.split()[0] 01533 fileOptionValue=token.split()[1].strip("'").strip('"') 01534 if "eventcontent" or "conditions" in fileOption: 01535 if "eventcontent" in fileOption: 01536 fileEventContentOption=fileOptionValue 01537 elif "conditions" in fileOption: 01538 # check if we are using the autoCond style of flexible conditions 01539 # if so, expand the condition here so that the file names contain the real conditions 01540 if "auto:" in fileOptionValue: 01541 from Configuration.PyReleaseValidation.autoCond import autoCond 01542 fileConditionsOption = autoCond[ fileOptionValue.split(':')[1] ] 01543 else: 01544 # "old style" conditions, hardcoded values ... 01545 # FIXME: 01546 # Should put at least the convention in cmsPerfCommons to know how to parse it... 01547 # Potential weak point if the conditions tag convention changes... 01548 if "," in fileOptionValue: #Since 330, conditions don't have FrontierConditions_GlobalTag, in front of them anymore... 01549 fileConditionsOption=fileOptionValue.split("::")[0].split(",")[1] #"Backward" compatibility 01550 else: 01551 fileConditionsOption=fileOptionValue.split("::")[0] 01552 else: # empty token 01553 #print "Print this is the token: %s"%token 01554 pass 01555 01556 #self.printFlush("Conditions label to add to the tarball name is %s"%fileConditionsOption) 01557 #self.printFlush("Eventcontent label to add to the tarball name is %s"%fileEventContentOption) 01558 #FIXME: 01559 #Could add the allowed event contents in the cmsPerfCommons.py file and use those to match in the command line options... This assumes maintenance of cmsPerfCommons.py 01560 01561 01562 #Create a tarball with just the logfiles 01563 subprocess.Popen("ls -R | grep .root > rootFiles",shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read() 01564 LogFile = "%s_%s_%s_%s_%s_%s_%s_%s_log.tgz" % (self.cmssw_arch, self.cmssw_version, fileStepOption, fileConditionsOption, fileEventContentOption.split()[0], fileWorkingDir, self.host, self.user) 01565 AbsTarFileLOG = os.path.join(perfsuitedir,LogFile) 01566 tarcmd = "tar zcfX %s %s %s" %(AbsTarFileLOG, "rootFiles", os.path.join(perfsuitedir,"*")) 01567 self.printFlush("Creating a tarball for the logfiles") 01568 self.printFlush(tarcmd) 01569 self.printFlush(subprocess.Popen(tarcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read()) 01570 self.printFlush(subprocess.Popen("rm rootFiles",shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read()) 01571 01572 fullcastorpathlog=os.path.join(castordir,LogFile) 01573 01574 01575 #Create the tarball with the contents of the directory + md5 checksum 01576 TarFile = "%s_%s_%s_%s_%s_%s_%s_%s.tgz" % (self.cmssw_arch, self.cmssw_version, fileStepOption, fileConditionsOption, fileEventContentOption.split()[0], fileWorkingDir, self.host, self.user) 01577 AbsTarFile = os.path.join(perfsuitedir,TarFile) 01578 tarcmd = "tar -zcf %s %s" %(AbsTarFile, os.path.join(perfsuitedir,"*")) 01579 md5cmd = "md5sum %s" %(AbsTarFile) 01580 self.printFlush("Creating a tarball with the content of the directory") 01581 self.printFlush(tarcmd) 01582 self.printFlush(md5cmd) 01583 #FIXME: 01584 #Anything that will be logged after the tar command below will not enter the cmsPerfSuite.log in the tarball (by definition)... 01585 #To remain backward compatible the harvesting script needs to be based on the command above to identify the tarball location. 01586 #Obsolete popen4-> subprocess.Popen 01587 #self.printFlush(os.popen3(tarcmd)[2].read()) #Using popen3 to get only stderr we don't want the whole stdout of tar! 01588 self.printFlush(subprocess.Popen(tarcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read()) 01589 md5sum = subprocess.Popen(md5cmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.read().split()[0] 01590 self.printFlush("The md5 checksum of the tarball: %s" %(md5sum)) 01591 AbsTarFileMD5 = AbsTarFile + ".md5" 01592 md5filecmd = "echo %s > %s" % (md5sum, AbsTarFileMD5) 01593 self.printFlush(subprocess.Popen(md5filecmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read()) 01594 01595 #Archive it on CASTOR 01596 #Before archiving check if it already exist if it does print a message, but do not overwrite, so do not delete it from local dir: 01597 fullcastorpathfile=os.path.join(castordir,TarFile) 01598 fullcastorpathmd5=os.path.join(castordir,TarFile + ".md5") 01599 01600 checkcastor="nsls %s" % fullcastorpathfile 01601 #Obsolete os.popen-> subprocess.Popen 01602 #checkcastorout=os.popen3(checkcastor)[1].read() 01603 checkcastorout=subprocess.Popen(checkcastor,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stdout.read() 01604 if checkcastorout.rstrip()==fullcastorpathfile: 01605 castorcmdstderr="File %s is already on CASTOR! Will NOT OVERWRITE!!!"%fullcastorpathfile 01606 else: 01607 castorcmd="rfcp %s %s" % (AbsTarFile,fullcastorpathfile) 01608 castormd5cmd="rfcp %s %s" % (AbsTarFileMD5,fullcastorpathmd5) 01609 castorlogcmd="rfcp %s %s" % (AbsTarFileLOG,fullcastorpathlog) 01610 self.printFlush(castorcmd) 01611 self.printFlush(castormd5cmd) 01612 self.printFlush(castorlogcmd) 01613 #Obsolete os.popen-> subprocess.Popen 01614 #castorcmdstderr=os.popen3(castorcmd)[2].read() 01615 castorcmdstderr=subprocess.Popen(castorcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read() 01616 subprocess.Popen(castormd5cmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read() 01617 subprocess.Popen(castorlogcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).stderr.read() 01618 #Checking the stderr of the rfcp command to copy the tarball (.tgz) on CASTOR: 01619 if castorcmdstderr: 01620 #If it failed print the stderr message to the log and tell the user the tarball (.tgz) is kept in the working directory 01621 self.printFlush(castorcmdstderr) 01622 self.printFlush("Since the CASTOR archiving for the tarball failed the file %s is kept in directory %s"%(TarFile, perfsuitedir)) 01623 else: 01624 #If it was successful then remove the tarball from the working directory: 01625 self.printFlush("Successfully archived the tarball %s in CASTOR!"%(TarFile)) 01626 self.printFlush("The tarball can be found: %s"%(fullcastorpathfile)) 01627 self.printFlush("The logfile can be found: %s"%(fullcastorpathlog)) 01628 self.printFlush("Deleting the local copy of the tarballs") 01629 rmtarballcmd="rm -Rf %s"%(AbsTarFile) 01630 rmtarballmd5cmd="rm -Rf %s"%(AbsTarFileMD5) 01631 rmtarballlogcmd="rm -Rf %s"%(AbsTarFileLOG) 01632 self.printFlush(rmtarballcmd) 01633 self.printFlush(rmtarballmd5cmd) 01634 self.printFlush(rmtarballlogcmd) 01635 #Obsolete os.popen-> subprocess.Popen 01636 #self.printFlush(os.popen4(rmtarballcmd)[1].read()) 01637 self.printFlush(subprocess.Popen(rmtarballcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout.read() ) 01638 self.printFlush(subprocess.Popen(rmtarballmd5cmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout.read() ) 01639 self.printFlush(subprocess.Popen(rmtarballlogcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout.read() ) 01640 tarballTime.set_end(datetime.datetime.now()) 01641 else: 01642 self.printFlush("Performance Suite directory will not be archived in a tarball since --no_tarball option was chosen") 01643 01644 #End of script actions! 01645 01646 #Print a time stamp at the end: 01647 date=time.ctime(time.time()) 01648 self.logh.write("Performance Suite finished running at %s on %s in directory %s\n" % (date,self.host,path)) 01649 if self.ERRORS == 0: 01650 self.logh.write("There were no errors detected in any of the log files!\n") 01651 else: 01652 self.logh.write("ERROR: There were %s errors detected in the log files, please revise!\n" % self.ERRORS) 01653 #print "No exit code test" 01654 #sys.exit(1) 01655 except exceptions.Exception, detail: 01656 self.logh.write(str(detail) + "\n") 01657 self.logh.flush() 01658 if not self.logh.isatty(): 01659 self.logh.close() 01660 raise 01661 #Add the possibility to send as an email the execution logfile to the user and whoever else interested: 01662 if MailLogRecipients != "": #Basically leave the option to turn it off too.. --mail "" 01663 self.printFlush("Sending email notification for this execution of the performance suite with command:") 01664 sendLogByMailcmd='cat cmsPerfSuite.log |mail -s "Performance Suite finished running on %s" '%self.host + MailLogRecipients 01665 self.printFlush(sendLogByMailcmd) 01666 #Obsolete os.popen-> subprocess.Popen 01667 #self.printFlush(os.popen4(sendLogByMailcmd)[1].read()) 01668 self.printFlush(subprocess.Popen(sendLogByMailcmd,shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout.read() ) 01669 else: 01670 self.printFlush('No email notification will be sent for this execution of the performance suite since option --mail "" was used') 01671 01672 TotalTime.set_end(datetime.datetime.now()) 01673 self.printFlush("Total Running Time\t%s hrs (%s mins)"%(TotalTime.get_duration()['hours'],TotalTime.get_duration()['minutes'])) 01674 01675 #Dump of the TimerInfo information 01676 #First dump it as a pickleq file... 01677 #Well in order to do so without the complication of serializing a custom class instance need to make the dictionary fully string-made: 01678 TimerInfoStr={} 01679 PerfSuiteTimerInfo=open("PerfSuiteTimerInfo.pkl","wb") 01680 #pickle.dump(TimerInfo,PerfSuiteTimerInfo) 01681 #PerfSuiteTimerInfo.close() 01682 #For now print it at the bottom of the log: 01683 self.logh.write("Test type\tActual Test\tDuration\tStart Time\tEnd Time\n") 01684 for key in TimerInfo.keys(): 01685 #self.printFlush(key) 01686 TimerInfoStr.update({key:{}}) 01687 for test in TimerInfo[key].keys(): 01688 TimerInfoStr[key].update({test:[str(TimerInfo[key][test].get_duration()['hours'])+" hrs ("+str(TimerInfo[key][test].get_duration()['minutes'])+" mins)",TimerInfo[key][test].get_start(),TimerInfo[key][test].get_end()]}) 01689 self.logh.write(key+"\t"+test+"\t") 01690 self.logh.write("%s hrs (%s mins)\t"%(TimerInfo[key][test].get_duration()['hours'],TimerInfo[key][test].get_duration()['minutes'])) 01691 self.logh.write("%s\t"%TimerInfo[key][test].get_start()) 01692 self.logh.write("%s\n"%TimerInfo[key][test].get_end()) 01693 pickle.dump(TimerInfoStr,PerfSuiteTimerInfo) 01694 PerfSuiteTimerInfo.close() 01695 01696 self.logh.write("Final Performance Suite exit code was %s"%FinalExitCode) 01697 self.logh.flush() sys.exit(FinalExitCode)
| def cmsPerfSuite::PerfSuite::simpleGenReport | ( | self, | |
| cpus, | |||
perfdir = os.getcwd(), |
|||
NumEvents = 1, |
|||
candles = ['MinBias'], |
|||
cmsdriverOptions = '', |
|||
stepOptions = '', |
|||
Name = '', |
|||
profilers = '', |
|||
bypasshlt = '', |
|||
userInputRootFiles = '' |
|||
| ) |
Prepares the profiling directory and runs all the selected profiles (if this is not a unit test)
Making parameters named to facilitate the handling of arguments (especially with the threading use case)
Definition at line 779 of file cmsPerfSuite.py.
00780 : 00781 callgrind = Name == "Callgrind" 00782 memcheck = Name == "Memcheck" 00783 00784 profCodes = {"TimeSize" : "0123", 00785 "IgProf" : "4567", 00786 "IgProf_Perf":"47", #Added the Analyse to IgProf_Perf #FIXME: At the moment Analyse is always run whether 7 is selected or not! Issue to solve in cmsRelvalreportInput.py... but not really important (it's always been there, not impacting our use-cases). 00787 "IgProf_Mem":"567", 00788 "Callgrind": "8", 00789 "Memcheck" : "9", 00790 None : "-1"} 00791 00792 profiles = profCodes[Name] 00793 if not profilers == "": 00794 profiles = profilers 00795 00796 RelvalreportExitCode=0 00797 00798 for cpu in cpus: 00799 pfdir = perfdir 00800 if len(cpus) > 1: 00801 pfdir = os.path.join(perfdir,"cpu_%s" % cpu) 00802 for candle in candles: 00803 #Create the directory for cmsRelvalreport.py running (e.g. MinBias_TimeSize, etc) 00804 #Catch the case of PILE UP: 00805 if "--pileup" in cmsdriverOptions: 00806 candlename=candle+"_PU" 00807 else: 00808 candlename=candle 00809 adir=self.mkCandleDir(pfdir,candlename,Name) 00810 if self._unittest: 00811 # Run cmsDriver.py 00812 if userInputRootFiles: 00813 self.logh.write(userInputRootFiles) 00814 userInputFile=userInputRootFiles[0] 00815 else: 00816 userInputFile="" 00817 self.logh.flush() 00818 self.runCmsInput(cpu,adir,NumEvents,candle,cmsdriverOptions,stepOptions,profiles,bypasshlt,userInputFile) 00819 self.testCmsDriver(cpu,adir,candle) 00820 else: 00821 if userInputRootFiles: 00822 self.logh.write("Variable userInputRootFiles is %s\n"%userInputRootFiles) 00823 #Need to use regexp, cannot rely on the order... since for different tests there are different candles... 00824 #userInputFile=userInputRootFiles[candles.index(candle)] 00825 #FIXME: 00826 #Note the issue that the input files HAVE to have in their name the candle as is used in cmsPerfSuite.py command line! 00827 #This is currently caught by a printout in the log: should be either taken care of with some exception to throw? 00828 #Will put this in the documentation 00829 userInputFile="" 00830 candleregexp=re.compile(candle) 00831 for file in userInputRootFiles: 00832 if candleregexp.search(file): 00833 userInputFile=file 00834 self.logh.write("For these %s %s tests will use user input file %s\n"%(candlename,Name,userInputFile)) 00835 if userInputFile == "": 00836 self.logh.write("***WARNING: For these %s %s tests could not find a matching input file in %s: will try to do without it!!!!!\n"%(candlename,Name,userInputRootFiles)) 00837 self.logh.flush() 00838 else: 00839 userInputFile="" 00840 DummyTestName=candlename+"_"+stepOptions.split("=")[1] 00841 DummyTimer=PerfSuiteTimer(start=datetime.datetime.now()) #Start the timer (DummyTimer is just a reference, but we will use the dictionary to access this later... 00842 TimerInfo[Name].update({DummyTestName:DummyTimer}) #Add the TimeSize timer to the dictionary 00843 #The following command will create the appropriate SimulationCandlesX.txt file in the relevant directory, ready to run cmsRelvalreport.py on it. 00844 self.runCmsInput(cpu,adir,NumEvents,candle,cmsdriverOptions,stepOptions,profiles,bypasshlt,userInputFile) 00845 #Here where the no_exec option kicks in (do everything but do not launch cmsRelvalreport.py, it also prevents cmsScimark spawning...): 00846 if self._noexec: 00847 self.logh.write("Running in debugging mode, without executing cmsRelvalreport.py\n") 00848 self.logh.flush() 00849 pass 00850 else: 00851 #The following command will launch cmsRelvalreport.py on the SimulationCandlesX.txt input file created above. 00852 ExitCode=self.runCmsReport(cpu,adir,candle) 00853 self.logh.write("Individual cmsRelvalreport.py ExitCode %s\n"%ExitCode) 00854 RelvalreportExitCode=RelvalreportExitCode+ExitCode 00855 self.logh.write("Summed cmsRelvalreport.py ExitCode %s\n"%RelvalreportExitCode) 00856 self.logh.flush() 00857 DummyTimer.set_end(datetime.datetime.now()) 00858 00859 #for proflog in proflogs: 00860 #With the change from 2>1&|tee to >& to preserve exit codes, we need now to check all logs... 00861 #less nice... we might want to do this externally so that in post-processing its a re-usable tool 00862 globpath = os.path.join(adir,"*.log") #"%s.log"%candle) 00863 self.logh.write("Looking for logs that match %s\n" % globpath) 00864 logs = glob.glob(globpath) 00865 for log in logs: 00866 self.logh.write("Found log %s\n" % log) 00867 self.displayErrors(log) 00868 self.printFlush("Returned cumulative RelvalreportExitCode is %s"%RelvalreportExitCode) 00869 return RelvalreportExitCode
| def cmsPerfSuite::PerfSuite::testCmsDriver | ( | self, | |
| cpu, | |||
| dir, | |||
| cmsver, | |||
| candle | |||
| ) |
Test cmsDriver.py (parses the simcandles file, removing duplicate lines, and runs the cmsDriver part)
Definition at line 717 of file cmsPerfSuite.py.
00718 : 00719 cmsdrvreg = re.compile("^cmsDriver.py") 00720 cmd = self.Commands[cpu][0] 00721 noExit = True 00722 stepreg = re.compile("--step=([^ ]*)") 00723 previousCmdOnline = "" 00724 for line in open(os.path.join(dir,"SimulationCandles_%s.txt" % (cmsver))): 00725 if (not line.lstrip().startswith("#")) and not (line.isspace() or len(line) == 0): 00726 cmdonline = line.split("@@@",1)[0] 00727 if cmsdrvreg.search(cmdonline) and not previousCmdOnline == cmdonline: 00728 stepbeingrun = "Unknown" 00729 matches = stepreg.search(cmdonline) 00730 if not matches == None: 00731 stepbeingrun = matches.groups()[0] 00732 if "PILEUP" in cmdonline: 00733 stepbeingrun += "_PILEUP" 00734 self.logh.write(cmdonline + "\n") 00735 cmds = ("cd %s" % (dir), 00736 "%s >& ../cmsdriver_unit_test_%s_%s.log" % (cmdonline,candle,stepbeingrun)) 00737 if self._dryrun: 00738 self.logh.write(cmds + "\n") 00739 else: 00740 out = self.runCmdSet(cmds) 00741 if not out == None: 00742 sig = out >> 16 # Get the top 16 bits 00743 xstatus = out & 0xffff # Mask out all bits except the first 16 00744 self.logh.write("FATAL ERROR: CMS Driver returned a non-zero exit status (which is %s) when running %s for candle %s. Signal interrupt was %s\n" % (xstatus,stepbeingrun,candle,sig)) 00745 sys.exit() 00746 previousCmdOnline = cmdonline
| def cmsPerfSuite::PerfSuite::valFilterReport | ( | self, | |
| dir | |||
| ) |
Filter lines in the valgrind report that match GEN,SIM.
Definition at line 635 of file cmsPerfSuite.py.
00636 : 00637 #cmds = ("cd %s" % dir, 00638 # "grep -v \"step=GEN,SIM\" SimulationCandles_%s.txt > tmp" % (self.cmssw_version), 00639 # "mv tmp SimulationCandles_%s.txt" % (self.cmssw_version)) 00640 #FIXME: 00641 #Quick and dirty hack to have valgrind MemCheck run on 5 events on both GEN,SIM and DIGI in QCD_80_120, while removing the line for GEN,SIM for Callgrind 00642 InputFileName=os.path.join(dir,"SimulationCandles_%s.txt"%(self.cmssw_version)) 00643 InputFile=open(InputFileName,"r") 00644 InputLines=InputFile.readlines() 00645 InputFile.close() 00646 Outputfile=open(InputFileName,"w") 00647 simRegxp=re.compile("step=GEN,SIM") 00648 digiRegxp=re.compile("step=DIGI") 00649 CallgrindRegxp=re.compile("ValgrindFCE") 00650 MemcheckRegxp=re.compile("Memcheck") 00651 NumEvtRegxp=re.compile("-n 1")#FIXME Either use the ValgrindEventNumber or do a more general match! 00652 for line in InputLines: 00653 if simRegxp.search(line) and CallgrindRegxp.search(line): 00654 continue 00655 elif simRegxp.search(line) and MemcheckRegxp.search(line): 00656 #Modify 00657 if NumEvtRegxp.search(line): 00658 line=NumEvtRegxp.sub(r"-n 5",line) 00659 else: 00660 self.logh.write("The number of Memcheck event was not changed since the original number of Callgrind event was not 1!\n") 00661 Outputfile.write(line) 00662 elif digiRegxp.search(line) and MemcheckRegxp.search(line): 00663 #Modify 00664 if NumEvtRegxp.search(line): 00665 line=NumEvtRegxp.sub(r"-n 5",line) 00666 else: 00667 self.logh.write("The number of Memcheck event was not changed since the original number of Callgrind event was not 1!\n") 00668 Outputfile.write(line) 00669 else: 00670 Outputfile.write(line) 00671 self.logh.flush() 00672 Outputfile.close() 00673 00674 #self.runCmdSet(cmds)
cmsPerfSuite::PerfSuite::_CASTOR_DIR [private] |
Definition at line 61 of file cmsPerfSuite.py.
cmsPerfSuite::PerfSuite::_debug [private] |
Developer options.
Definition at line 61 of file cmsPerfSuite.py.
cmsPerfSuite::PerfSuite::_dryrun [private] |
Definition at line 61 of file cmsPerfSuite.py.
cmsPerfSuite::PerfSuite::_noexec [private] |
Definition at line 61 of file cmsPerfSuite.py.
cmsPerfSuite::PerfSuite::_unittest [private] |
Definition at line 61 of file cmsPerfSuite.py.
cmsPerfSuite::PerfSuite::_verbose [private] |
Check logfile option.
Check step Options Check profile option Check output directory option Check cpu option Check previous release directory Setup quicktest option Setup unit test option
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 877 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 877 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
FIXME: We may want to introduce a switch here or agree on a different default (currently 10 cmsScimark and 10 cmsScimarkLarge)
PERF## Prepare the simpleGenReport arguments for this test: MEM## Prepare the simpleGenReport arguments for this test: PERF## Prepare the simpleGenReport arguments for this test: MEM## Prepare the simpleGenReport arguments for this test:
Definition at line 887 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
Definition at line 61 of file cmsPerfSuite.py.
1.7.3