mergeLHE.DefaultLHEMerger Class Reference

Inheritance diagram for mergeLHE.DefaultLHEMerger:

Public Member Functions
def	__init__ (self, input_files, output_file, kwargs)
def	check_header_compatibility (self)
def	file_iterator (self, path)
def	merge (self)
def	merge_headers (self)
def	merge_init_blocks (self)
Public Member Functions inherited from mergeLHE.BaseLHEMerger
def	__init__ (self, input_files, output_file)
def	merge (self)

Public Attributes
	bypass_check
Public Attributes inherited from mergeLHE.BaseLHEMerger
	input_files
	output_file

Private Attributes
	_f
	_header_lines
	_header_str
	_init_str
	_is_mglo
	_merged_init_str
	_nevent
	_uwgt
	_xsec_combined

Detailed Description

Default LHE merge scheme that copies the header of the first LHE file,
merges and outputs the init block, then concatenates all event blocks.

Definition at line 26 of file mergeLHE.py.

Constructor & Destructor Documentation

__init__()

def mergeLHE.DefaultLHEMerger.__init__	(		self,
			input_files,
			output_file,
			kwargs
	)

Definition at line 30 of file mergeLHE.py.

    def __init__(self, input_files, output_file, **kwargs):
        super(DefaultLHEMerger, self).__init__(input_files, output_file)

        self.bypass_check = kwargs.get('bypass_check', False)
        # line-by-line iterator for each input file
        self._f = [self.file_iterator(name) for name in self.input_files]
        self._header_str = []
        self._is_mglo = False
        self._xsec_combined = 0.
        self._uwgt = 0.
        self._init_str = [] # initiated blocks for each input file
        self._nevent = [] # number of events for each input file

Member Function Documentation

check_header_compatibility()

def mergeLHE.DefaultLHEMerger.check_header_compatibility

(

self

)

Check if all headers for input files are consistent.

Definition at line 49 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger.bypass_check.

Referenced by mergeLHE.DefaultLHEMerger.merge().

    def check_header_compatibility(self):
        """Check if all headers for input files are consistent."""

        if self.bypass_check:
            return

        inconsistent_error_info = ("Incompatibility found in LHE headers: %s. "
                                  "Use -b/--bypass-check to bypass the check.")
        allow_diff_keys = [
            'nevent', 'numevts', 'iseed', 'Seed', 'Random', '.log', '.dat', '.lhe',
            'Number of Events', 'Integrated weight'
            ]
        self._header_lines = [header.split('\n') for header in self._header_str]

        # Iterate over header lines for all input files and check consistency
        logging.debug('header line number: %s' \
            % ', '.join([str(len(lines)) for lines in self._header_lines]))
        assert all([
            len(self._header_lines[0]) == len(lines) for lines in self._header_lines]
            ), inconsistent_error_info % "line number does not match"
        inconsistent_lines_set = [set() for _ in self._header_lines]
        for line_zip in zip(*self._header_lines):
            if any([k in line_zip[0] for k in allow_diff_keys]):
                logging.debug('Captured \'%s\', we allow difference in this line' % line_zip[0])
                continue
            if not all([line_zip[0] == line for line in line_zip]):
                # Ok so meet inconsistency in some lines, then temporarily store them
                for i, line in enumerate(line_zip):
                    inconsistent_lines_set[i].add(line)
        # Those inconsistent lines still match, meaning that it is only a change of order
        assert all([inconsistent_lines_set[0] == lset for lset in inconsistent_lines_set]), \
            inconsistent_error_info % ('{' + ', '.join(inconsistent_lines_set[0]) + '}')

file_iterator()

def mergeLHE.DefaultLHEMerger.file_iterator	(		self,
			path
	)

Line-by-line iterator of a txt file

Definition at line 43 of file mergeLHE.py.

    def file_iterator(self, path):
        """Line-by-line iterator of a txt file"""
        with open(path, 'r') as f:            for line in f:
                yield line

merge()

def mergeLHE.DefaultLHEMerger.merge

(

self

)

Definition at line 196 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger._f, edmStreamStallGrapher.StallMonitorParser._f, edmTracerCompactLogViewer.TracerCompactFileParser._f, mergeLHE.DefaultLHEMerger._header_str, mergeLHE.DefaultLHEMerger._init_str, mergeLHE.DefaultLHEMerger._is_mglo, mergeLHE.DefaultLHEMerger._nevent, mergeLHE.DefaultLHEMerger._uwgt, mergeLHE.DefaultLHEMerger._xsec_combined, mps_setup.append, mergeLHE.DefaultLHEMerger.bypass_check, mergeLHE.DefaultLHEMerger.check_header_compatibility(), nano_mu_digi_cff.float, watchdog.group, join(), mergeLHE.DefaultLHEMerger.merge_headers(), mergeLHE.DefaultLHEMerger.merge_init_blocks(), GetRecoTauVFromDQM_MC_cff.next, DTT0WireWorkflow.DTT0WireWorkflow.output_file, mergeLHE.BaseLHEMerger.output_file, DTVdriftWorkflow.DTvdriftWorkflow.output_file, DTTtrigWorkflow.DTttrigWorkflow.output_file, FastTimerService_cff.range, and Validation_hcalonly_cfi.sign.

    def merge(self):
        with open(self.output_file, 'w') as fw:
            # Read the header for the all input files
            for i in range(len(self._f)):
                header = []
                line = next(self._f[i])
                while not re.search('\s*<init(>|\s)', line):
                    header.append(line)
                    line = next(self._f[i])
                # 'header' includes all contents before reaches <init>
                self._header_str.append(''.join(header))
            self.check_header_compatibility()

            # Read <init> blocks for all input_files
            for i in range(len(self._f)):
                init = []
                line = next(self._f[i])
                while not re.search('\s*</init>', line):
                    init.append(line)
                    line = next(self._f[i])
                # 'init_str' includes all contents inside <init>...</init>
                self._init_str.append(''.join(init))

            # Iterate over all events file-by-file and write events temporarily
            # to .tmp.lhe
            with open('.tmp.lhe', 'w') as _fwtmp:
                for i in range(len(self._f)):
                    nevent = 0
                    while True:
                        line = next(self._f[i])
                        if re.search('\s*</event>', line):
                            nevent += 1
                        if re.search('\s*</LesHouchesEvents>', line):
                            break
                        _fwtmp.write(line)
                    self._nevent.append(nevent)
                    self._f[i].close()

            # Merge the header and init blocks and write to the output
            fw.write(self.merge_headers())
            fw.write('<init>\n' + self.merge_init_blocks() + '</init>\n')

            # Write event blocks in .tmp.lhe back to the output
            # If is MG5 LO LHE, will recalculate the weights based on combined xsec
            # and nevent read from <MGGenerationInfo>, and the 'event_norm' mode
            if self._is_mglo and not self.bypass_check:
                event_norm = re.search(
                    r'\s(\w+)\s*=\s*event_norm\s',
                    self._header_str[0]).group(1)
                if event_norm == 'sum':
                    self._uwgt = self._xsec_combined / sum(self._nevent)
                elif event_norm == 'average':
                    self._uwgt = self._xsec_combined
                logging.info(("MG5 LO LHE with event_norm = %s detected. Will "
                             "recalculate weights in each event block.\n"
                             "Unit weight: %+.7E") % (event_norm, self._uwgt))

                # Modify event wgt when transfering .tmp.lhe to the output file
                event_line = -999
                with open('.tmp.lhe', 'r') as ftmp:
                    sign = lambda x: -1 if x < 0 else 1
                    for line in ftmp:
                        event_line += 1
                        if re.search('\s*<event.*>', line) and not re.search('\s*<event_num.*>', line):
                            event_line = 0
                        if event_line == 1:
                            # modify the XWGTUP appeared in the first line of the
                            # <event> block
                            orig_wgt = float(line.split()[2])
                            fw.write(re.sub(r'(^\s*\S+\s+\S+\s+)\S+(.+)', r'\g<1>%+.7E\g<2>' \
                                % (sign(orig_wgt) * self._uwgt), line))
                        elif re.search('\s*<wgt.*>.*</wgt>', line):
                            addi_wgt_str = re.search(r'<wgt.*>\s*(\S+)\s*<\/wgt>', line).group(1)
                            fw.write(line.replace(
                                addi_wgt_str, '%+.7E' % (float(addi_wgt_str) / orig_wgt * self._uwgt)))
                        else:
                            fw.write(line)
            else:
                # Simply transfer all lines
                with open('.tmp.lhe', 'r') as ftmp:
                    for line in ftmp:
                        fw.write(line)
            fw.write('</LesHouchesEvents>\n')
            os.remove('.tmp.lhe')

merge_headers()

def mergeLHE.DefaultLHEMerger.merge_headers

(

self

)

Merge the headers of input LHEs. Need special handle for the MG5 LO case.

Definition at line 82 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger._header_str, mergeLHE.DefaultLHEMerger._is_mglo, mergeLHE.DefaultLHEMerger._nevent, mergeLHE.DefaultLHEMerger._xsec_combined, python.cmstools.all(), mergeLHE.DefaultLHEMerger.bypass_check, nano_mu_digi_cff.float, watchdog.group, python.rootplot.root2matplotlib.replace(), and reco.zip().

Referenced by mergeLHE.DefaultLHEMerger.merge().

    def merge_headers(self):
        """Merge the headers of input LHEs. Need special handle for the MG5 LO case."""

        self._is_mglo = all(['MGGenerationInfo' in header for header in self._header_str])
        if self._is_mglo and not self.bypass_check:
            # Special handling of MadGraph5 LO LHEs
            match_geninfo = [
                re.search(
                    (r"<MGGenerationInfo>\s+#\s*Number of Events\s*\:\s*(\S+)\s+"
                    r"#\s*Integrated weight \(pb\)\s*\:\s*(\S+)\s+<\/MGGenerationInfo>"),
                    header
                    ) for header in self._header_str
                ]
            self._xsec_combined = sum(
                [float(info.group(2)) * nevt for info, nevt in zip(match_geninfo, self._nevent)]
                ) / sum(self._nevent)
            geninfo_combined = ("<MGGenerationInfo>\n"
                                "#  Number of Events        : %d\n"
                                "#  Integrated weight (pb)  : %.10f\n</MGGenerationInfo>") \
                                    % (sum(self._nevent), self._xsec_combined)
            logging.info('Detected: MG5 LO LHEs. Input <MGGenerationInfo>:\n\tnevt\txsec')
            for info, nevt in zip(match_geninfo, self._nevent):
                logging.info('\t%d\t%.10f' % (nevt, float(info.group(2))))
            logging.info('Combined <MGGenerationInfo>:\n\t%d\t%.10f' \
                % (sum(self._nevent), self._xsec_combined))

            header_combined = self._header_str[0].replace(match_geninfo[0].group(), geninfo_combined)
            return header_combined

        else:
            # No need to merge the headers
            return self._header_str[0]

merge_init_blocks()

def mergeLHE.DefaultLHEMerger.merge_init_blocks

(

self

)

If all <init> blocks are identical, return the same <init> block
(in the case of Powheg LHEs); otherwise, calculate the output <init>
blocks by merging the input blocks info using formula (same with the
MG5LOLHEMerger scheme):
    XSECUP = sum(xsecup * no.events) / tot.events
    XERRUP = sqrt( sum(sigma^2 * no.events^2) ) / tot.events
    XMAXUP = max(xmaxup)

Definition at line 115 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger._f, edmStreamStallGrapher.StallMonitorParser._f, edmTracerCompactLogViewer.TracerCompactFileParser._f, mergeLHE.DefaultLHEMerger._init_str, mergeLHE.DefaultLHEMerger._nevent, python.cmstools.all(), mergeLHE.DefaultLHEMerger.bypass_check, nano_mu_digi_cff.float, mergeLHE.BaseLHEMerger.input_files, DTWorkflow.DTWorkflow.input_files, createfilelist.int, relativeConstraints.keys, SiStripPI.max, FastTimerService_cff.range, submitPVValidationJobs.split(), nano_mu_digi_cff.strip, and mkLumiAveragedPlots.tuple.

Referenced by mergeLHE.DefaultLHEMerger.merge().

    def merge_init_blocks(self):
        """If all <init> blocks are identical, return the same <init> block
        (in the case of Powheg LHEs); otherwise, calculate the output <init>
        blocks by merging the input blocks info using formula (same with the
        MG5LOLHEMerger scheme):
            XSECUP = sum(xsecup * no.events) / tot.events
            XERRUP = sqrt( sum(sigma^2 * no.events^2) ) / tot.events
            XMAXUP = max(xmaxup)
        """

        if self.bypass_check:
            # If bypass the consistency check, simply use the first LHE <init>
            # block as the output
            return self._init_str[0]

        # Initiate collected init block info. Will be in format of
        # {iprocess: [xsecup, xerrup, xmaxup]}
        new_init_block = {}
        old_init_block = [{} for _ in self._init_str]

        # Read the xsecup, xerrup, and xmaxup info from the <init> block for
        # all input LHEs
        for i, bl in enumerate(self._init_str): # loop over files
            nline = int(bl.split('\n')[0].strip().split()[-1])

            # loop over lines in <init> block
            for bl_line in bl.split('\n')[1:nline + 1]:
                bl_line_sp = bl_line.split()
                old_init_block[i][int(bl_line_sp[3])] = [
                    float(bl_line_sp[0]), float(bl_line_sp[1]), float(bl_line_sp[2])]

            # After reading all subprocesses info, store the rest content in
            # <init> block for the first file
            if i == 0:
                info_after_subprocess = bl.strip().split('\n')[nline + 1:]

            logging.info('Input file: %s' % self.input_files[i])
            for ipr in sorted(list(old_init_block[i].keys()), reverse=True):
                # reverse order: follow the MG5 custom
                logging.info('  xsecup, xerrup, xmaxup, lprup: %.6E, %.6E, %.6E, %d' \
                    % tuple(old_init_block[i][ipr] + [ipr]))

        # Adopt smarter <init> block merging method
        # If all <init> blocks from input files are identical, return the same block;
        # otherwise combine them based on MG5LOLHEMerger scheme
        if all([old_init_block[i] == old_init_block[0] for i in range(len(self._f))]):
            # All <init> blocks are identical
            logging.info(
                'All input <init> blocks are identical. Output the same "<init> block.')
            return self._init_str[0]

        # Otherwise, calculate merged init block
        for i in range(len(self._f)):
            for ipr in old_init_block[i]:
                # Initiate the subprocess for the new block if it is found for the
                # first time in one input file
                if ipr not in new_init_block:
                    new_init_block[ipr] = [0., 0., 0.]
                new_init_block[ipr][0] += old_init_block[i][ipr][0] * self._nevent[i] # xsecup
                new_init_block[ipr][1] += old_init_block[i][ipr][1]**2 * self._nevent[i]**2 # xerrup
                new_init_block[ipr][2] = max(new_init_block[ipr][2], old_init_block[i][ipr][2]) # xmaxup
        tot_nevent = sum([self._nevent[i] for i in range(len(self._f))])

        # Write first line of the <init> block (modify the nprocess at the last)
        self._merged_init_str = self._init_str[0].split('\n')[0].strip().rsplit(' ', 1)[0] \
            + ' ' + str(len(new_init_block)) + '\n'
        # Form the merged init block
        logging.info('Output file: %s' % self.output_file)
        for ipr in sorted(list(new_init_block.keys()), reverse=True):
            # reverse order: follow the MG5 custom
            new_init_block[ipr][0] /= tot_nevent
            new_init_block[ipr][1] = math.sqrt(new_init_block[ipr][1]) / tot_nevent
            logging.info('  xsecup, xerrup, xmaxup, lprup: %.6E, %.6E, %.6E, %d' \
                % tuple(new_init_block[ipr] + [ipr]))
            self._merged_init_str += '%.6E %.6E %.6E %d\n' % tuple(new_init_block[ipr] + [ipr])
        self._merged_init_str += '\n'.join(info_after_subprocess)
        if len(info_after_subprocess):
            self._merged_init_str += '\n'

        return self._merged_init_str

Member Data Documentation

_f

mergeLHE.DefaultLHEMerger._f

private

Definition at line 35 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

_header_lines

mergeLHE.DefaultLHEMerger._header_lines

private

Definition at line 61 of file mergeLHE.py.

_header_str

mergeLHE.DefaultLHEMerger._header_str

private

Definition at line 36 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_headers().

_init_str

mergeLHE.DefaultLHEMerger._init_str

private

Definition at line 40 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

_is_mglo

mergeLHE.DefaultLHEMerger._is_mglo

private

Definition at line 37 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_headers().

_merged_init_str

mergeLHE.DefaultLHEMerger._merged_init_str

private

Definition at line 179 of file mergeLHE.py.

_nevent

mergeLHE.DefaultLHEMerger._nevent

private

Definition at line 41 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), mergeLHE.DefaultLHEMerger.merge_headers(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

_uwgt

mergeLHE.DefaultLHEMerger._uwgt

private

Definition at line 39 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge().

_xsec_combined

mergeLHE.DefaultLHEMerger._xsec_combined

private

Definition at line 38 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_headers().

bypass_check

mergeLHE.DefaultLHEMerger.bypass_check

Definition at line 33 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.check_header_compatibility(), mergeLHE.DefaultLHEMerger.merge(), mergeLHE.DefaultLHEMerger.merge_headers(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

---

Generated for CMSSW Reference Manual by  1.8.14