|
|
|
Static Public Member Functions | |
| def | findFirstIndex_ofStartsWith |
| def | isTimeStamp |
Public Attributes | |
| lines_general | |
| lines_other | |
| lines_timesize | |
| missing_fields | |
| reCmsScimarkTest | |
Private Member Functions | |
| def | _applyParsingRules |
Private Attributes | |
| _DEBUG | |
| _MAX_STEPS | |
| _otherStart | |
| _path | |
| _timeSizeEnd | |
| _timeSizeStart | |
Static Private Attributes | |
| string | _LINE_SEPARATOR = "|" |
The whole parsing works as following. We split the file into 3 parts (we keep 3 variables of line lists:self.lines_general, self.lines_timesize, self.lines_other ):
* General info
As most of the info are simple one line strings, we define some regular expressions defining and matching each of those lines. The regular expressions are associated with data which we can get from them. e.g. ^Suite started at (.+) on (.+) by user (.+)$ would match only the line defining the time suite started and on which machine. It's associated with tuple of field names for general info which will be filled in. in this way we get info = {'start_time': start-taken-from-regexp, 'host': host, 'user': user}. This is done by calling simple function _applyParsingRules which checks each lines with each if one passes another, if it does fills in the result dictionary with the result.
Additionaly we get the cpu and memmory info from /proc/cpuinfo /proc/meminfo
* TimeSize test
We use the same technique a little bit also. But at first we divide the timesize lines by job (individual run of cmssw - per candle, and pileup/not). Then for each of the jobs we apply our parsing rules, also we find the starting and ending times (i.e. We know that start timestamp is somethere after certain line containing "Written out cmsRelvalreport.py input file at:")
* All other tests
We find the stating that the test is being launched (containing the test name, core and num events). Above we have the thread number, and below the starting time.
The ending time can be ONLY connected with the starting time by the Thread-ID. The problem is that the file names different the same test instance like <Launching "PILE UP Memcheck"> and <"Memcheck" stopped>.
Definition at line 8 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.__init__ | ( | self, | |
| path | |||
| ) |
Definition at line 28 of file parserPerfsuiteMetadata.py.
|
private |
Applies the (provided) regular expression rules (=rule[1] for rule in parsing_rules)
to each line and if it matches the line,
puts the mached information to the dictionary as the specified keys (=rule[0]) which is later returned
Rule[3] contains whether the field is required to be found. If so and it isn't found the exception would be raised.
rules = [
( (field_name_1_to_match, field_name_2), regular expression, /optionaly: is the field required? if so "req"/ )
]we call a shared parsing helper
Definition at line 235 of file parserPerfsuiteMetadata.py.
|
static |
Definition at line 113 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.findLineAfter | ( | self, | |
| line_index, | |||
| lines, | |||
| test_condition, | |||
return_index = False |
|||
| ) |
finds a line satisfying the `test_condition` comming after the `line_index`
Definition at line 129 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.findLineBefore | ( | self, | |
| line_index, | |||
| lines, | |||
| test_condition | |||
| ) |
finds a line satisfying the `test_condition` comming before the `line_index`
Definition at line 118 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.firstTimeStampAfter | ( | self, | |
| line_index, | |||
| lines | |||
| ) |
returns the first timestamp AFTER the line with given index
Definition at line 145 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.firstTimeStampBefore | ( | self, | |
| line_index, | |||
| lines | |||
| ) |
returns the first timestamp BEFORE the line with given index
Definition at line 140 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.get_tarball_fromlog | ( | self | ) |
Return the tarball castor location by parsing the cmsPerfSuite.log file
Definition at line 707 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.getMachineInfo | ( | self | ) |
Returns the cpu and memory info
cpu info
we assume that: * num_cores = max(core id+1) [it's counted from 0] * 'model name' is processor type [we will return only the first one - we assume others to be same!!?? * cpu MHz - is the speed of CPU
for
model name : Intel(R) Core(TM)2 Duo CPU L9400 @ 1.86GHz
cpu MHz : 800.000
cache size : 6144 KB
Definition at line 175 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.handleParsingError | ( | self, | |
| message | |||
| ) |
Definition at line 150 of file parserPerfsuiteMetadata.py.
|
static |
Returns whether the string is a timestamp (if not returns None)
>>> parserPerfsuiteMetadata.isTimeStamp("Fri Aug 14 01:16:03 2009")
True
>>> parserPerfsuiteMetadata.isTimeStamp("Fri Augx 14 01:16:03 2009")
Definition at line 96 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.parseAll | ( | self | ) |
Definition at line 722 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.parseAllOtherTests | ( | self | ) |
Definition at line 360 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.parseGeneralInfo | ( | self | ) |
Definition at line 255 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.parseTheCompletion | ( | self | ) |
checks if the suite has successfully finished
and if the tarball was successfully archived and uploaded to the castor
Definition at line 654 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.parseTimeSize | ( | self | ) |
parses the timeSize
Definition at line 493 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.readCmsScimark | ( | self, | |
main_cores = [1] |
|||
| ) |
Definition at line 629 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.readCmsScimarkTest | ( | self, | |
| testName, | |||
| testType, | |||
| core | |||
| ) |
Definition at line 617 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.readInput | ( | self, | |
| path, | |||
fileName = "cmsPerfSuite.log" |
|||
| ) |
Definition at line 161 of file parserPerfsuiteMetadata.py.
| def parserPerfsuiteMetadata.parserPerfsuiteMetadata.validateSteps | ( | self, | |
| steps | |||
| ) |
Simple function for error detection. TODO: we could use a list of possible steps also
Definition at line 24 of file parserPerfsuiteMetadata.py.
|
private |
Definition at line 31 of file parserPerfsuiteMetadata.py.
|
staticprivate |
Definition at line 23 of file parserPerfsuiteMetadata.py.
|
private |
Definition at line 30 of file parserPerfsuiteMetadata.py.
|
private |
Definition at line 50 of file parserPerfsuiteMetadata.py.
|
private |
Definition at line 34 of file parserPerfsuiteMetadata.py.
|
private |
Definition at line 47 of file parserPerfsuiteMetadata.py.
|
private |
Definition at line 42 of file parserPerfsuiteMetadata.py.
| parserPerfsuiteMetadata.parserPerfsuiteMetadata.lines_general |
Definition at line 88 of file parserPerfsuiteMetadata.py.
| parserPerfsuiteMetadata.parserPerfsuiteMetadata.lines_other |
Definition at line 90 of file parserPerfsuiteMetadata.py.
| parserPerfsuiteMetadata.parserPerfsuiteMetadata.lines_timesize |
Definition at line 89 of file parserPerfsuiteMetadata.py.
| parserPerfsuiteMetadata.parserPerfsuiteMetadata.missing_fields |
Definition at line 93 of file parserPerfsuiteMetadata.py.
| parserPerfsuiteMetadata.parserPerfsuiteMetadata.reCmsScimarkTest |
Definition at line 38 of file parserPerfsuiteMetadata.py.
1.8.5