parserPerfsuiteMetadata.parserPerfsuiteMetadata Class Reference

Public Member Functions
def	__init__
def	doQuery
def	findLineAfter
def	findLineBefore
def	firstTimeStampAfter
def	firstTimeStampBefore
def	get_tarball_fromlog
def	getIgSummary
def	getMachineInfo
def	getSummaryInfo
def	handleParsingError
def	parseAll
def	parseAllOtherTests
def	parseGeneralInfo
def	parseTheCompletion
def	parseTimeSize
def	readCmsScimark
def	readCmsScimarkTest
def	readInput
def	validateSteps

Static Public Member Functions
def	findFirstIndex_ofStartsWith
def	isTimeStamp

Public Attributes
	lines_general
	lines_other
	lines_timesize
	missing_fields
	reCmsScimarkTest

Private Member Functions
def	_applyParsingRules

Private Attributes
	_DEBUG
	_MAX_STEPS
	_otherStart
	_path
	_timeSizeEnd
	_timeSizeStart

Static Private Attributes
string	_LINE_SEPARATOR = "|"

Detailed Description

    The whole parsing works as following. We split the file into 3 parts (we keep 3 variables of line lists:self.lines_general, self.lines_timesize, self.lines_other ):

        * General info
    As most of the info are simple one line strings, we define some regular expressions defining and matching each of those lines. The regular expressions are associated with data which we can get from them. e.g. ^Suite started at (.+) on (.+) by user (.+)$ would match only the line defining the time suite started and on which machine. It's associated with tuple of field names for general info which will be filled in. in this way we get info = {'start_time': start-taken-from-regexp, 'host': host, 'user': user}. This is done by calling simple function _applyParsingRules which checks each lines with each if one passes another, if it does fills in the result dictionary with the result.
    Additionaly we get the cpu and memmory info from /proc/cpuinfo /proc/meminfo

        * TimeSize test
    We use the same technique a little bit also. But at first we divide the timesize lines by job (individual run of cmssw - per candle, and pileup/not). Then for each of the jobs we apply our parsing rules, also we find the starting and ending times (i.e. We know that start timestamp is somethere after certain line containing "Written out cmsRelvalreport.py input file at:")

        * All other tests
    We find the stating that the test is being launched (containing the test name, core and num events). Above we have the thread number, and below the starting time.
    The ending time can be ONLY connected with the starting time by the Thread-ID. The problem is that the file names different the same test instance like <Launching "PILE UP Memcheck"> and <"Memcheck" stopped>.

Definition at line 8 of file parserPerfsuiteMetadata.py.

Constructor & Destructor Documentation

def parserPerfsuiteMetadata.parserPerfsuiteMetadata.__init__	(		self,
			path
	)

Definition at line 28 of file parserPerfsuiteMetadata.py.

    def __init__(self, path):
        
        self._MAX_STEPS  = 5 # MAXIMUM NUMBER OF STEPS PER RUN (taskset relvalreport.py...)
        self._DEBUG = False


        self._path = path
        
        """ some initialisation to speedup the other functions """
        #for cmsscimark
        self.reCmsScimarkTest = re.compile(r"""^Composite Score:(\s*)([^\s]+)$""")

        #TimeSize
        """ the separator for beginning of timeSize / end of general statistics """
        self._timeSizeStart = re.compile(r"""^Launching the TimeSize tests \(TimingReport, TimeReport, SimpleMemoryCheck, EdmSize\) with (\d+) events each$""")
        """ (the first timestamp is the start of TimeSize) """


        """ the separator for end of timeSize / beginning of IgProf_Perf, IgProf_Mem,  Memcheck, Callgrind tests """
        self._timeSizeEnd = re.compile(r"""^Stopping all cmsScimark jobs now$""")

        #Other tests:
        self._otherStart = re.compile(r"^Preparing")

        """ 
        ----- READ THE DATA -----
        """
        lines = self.readInput(path)
        """ split the whole file  into parts """
        #Let's not assume there are ALWAYS TimeSize tests in the runs of the Performance Suite!:
        #Check first:  
        #FIXME: Vidmantas did not think to this case... will need to implement protectionb against it for all the IB tests...
        #To do as soon as possible...
        #Maybe revisit the strategy if it can be done quickly.
        timesize_end= [lines.index(line)  for line in lines if self._timeSizeEnd.match(line)]
        if timesize_end:
            timesize_end_index = timesize_end[0]
        else:
            timesize_end_index=0
        timesize_start=[lines.index(line) for line in lines if self._timeSizeStart.match(line)]
        general_stop=[lines.index(line) for line in lines if self._otherStart.match(line)]
        if timesize_start:
            timesize_start_index = timesize_start[0]
            general_stop_index=timesize_start_index
        elif general_stop:
            timesize_start_index=0
            general_stop_index=general_stop[0]
        else:
            timesize_start_index=0
            general_stop_index=-1

        """ we split the structure:
            * general
            * timesize
            * all others [igprof etc]
        """
    
        """ we get the indexes of spliting """
        #Not OK to use timsize_start_index for the general lines... want to be general, also to cases of no TimeSize tests...
        #self.lines_general = lines[:timesize_start_index]
        self.lines_general = lines[:general_stop_index]
        self.lines_timesize = lines[timesize_start_index:timesize_end_index+1]
        self.lines_other = lines[timesize_end_index:]       
    
        """ a list of missing fields """
        self.missing_fields = []