Inheritance diagram for mergeLHE.DefaultLHEMerger:

Public Member Functions
def	__init__ (self, input_files, output_file, kwargs)

def	check_header_compatibility (self)

def	file_iterator (self, path)

def	merge (self)

def	merge_headers (self)

def	merge_init_blocks (self)

Public Member Functions inherited from mergeLHE.BaseLHEMerger
def	__init__ (self, input_files, output_file)

def	merge (self)

Public Attributes
	bypass_check

Public Attributes inherited from mergeLHE.BaseLHEMerger
	input_files

	output_file

Private Attributes
	_f

	_header_lines

	_header_str

	_init_str

	_is_mglo

	_merged_init_str

	_nevent

	_uwgt

	_xsec_combined

Detailed Description

Default LHE merge scheme that copies the header of the first LHE file,
merges and outputs the init block, then concatenates all event blocks.

Definition at line 26 of file mergeLHE.py.

Constructor & Destructor Documentation

◆ init()

def mergeLHE.DefaultLHEMerger.__init__	(	self,
		input_files,
		output_file,
		kwargs
	)

Definition at line 30 of file mergeLHE.py.

     def __init__(self, input_files, output_file, **kwargs):
         super(DefaultLHEMerger, self).__init__(input_files, output_file)
 
         self.bypass_check = kwargs.get('bypass_check', False)
         # line-by-line iterator for each input file
         self._f = [self.file_iterator(name) for name in self.input_files]
         self._header_str = []
         self._is_mglo = False
         self._xsec_combined = 0.
         self._uwgt = 0.
         self._init_str = [] # initiated blocks for each input file
         self._nevent = [] # number of events for each input file
 

Member Function Documentation

◆ check_header_compatibility()

def mergeLHE.DefaultLHEMerger.check_header_compatibility ( self )

Check if all headers for input files are consistent.

Definition at line 49 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger.bypass_check.

Referenced by mergeLHE.DefaultLHEMerger.merge().

     def check_header_compatibility(self):
         """Check if all headers for input files are consistent."""
 
         if self.bypass_check:
             return
 
         inconsistent_error_info = ("Incompatibility found in LHE headers: %s. "
                                   "Use -b/--bypass-check to bypass the check.")
         allow_diff_keys = [
             'nevent', 'numevts', 'iseed', 'Seed', 'Random', '.log', '.dat', '.lhe',
             'Number of Events', 'Integrated weight'
             ]
         self._header_lines = [header.split('\n') for header in self._header_str]
 
         # Iterate over header lines for all input files and check consistency
         logging.debug('header line number: %s' \
             % ', '.join([str(len(lines)) for lines in self._header_lines]))
         assert all([
             len(self._header_lines[0]) == len(lines) for lines in self._header_lines]
             ), inconsistent_error_info % "line number does not match"
         inconsistent_lines_set = [set() for _ in self._header_lines]
         for line_zip in zip(*self._header_lines):
             if any([k in line_zip[0] for k in allow_diff_keys]):
                 logging.debug('Captured \'%s\', we allow difference in this line' % line_zip[0])
                 continue
             if not all([line_zip[0] == line for line in line_zip]):
                 # Ok so meet inconsistency in some lines, then temporarily store them
                 for i, line in enumerate(line_zip):
                     inconsistent_lines_set[i].add(line)
         # Those inconsistent lines still match, meaning that it is only a change of order
         assert all([inconsistent_lines_set[0] == lset for lset in inconsistent_lines_set]), \
             inconsistent_error_info % ('{' + ', '.join(inconsistent_lines_set[0]) + '}')
 

◆ file_iterator()

def mergeLHE.DefaultLHEMerger.file_iterator	(	self,
		path
	)

Line-by-line iterator of a txt file

Definition at line 43 of file mergeLHE.py.

     def file_iterator(self, path):
         """Line-by-line iterator of a txt file"""
         with open(path, 'r') as f:            for line in f:
                 yield line
 
 

◆ merge()

def mergeLHE.DefaultLHEMerger.merge ( self )

Definition at line 196 of file mergeLHE.py.

     def merge(self):
         with open(self.output_file, 'w') as fw:
             # Read the header for the all input files
             for i in range(len(self._f)):
                 header = []
                 line = next(self._f[i])
                 while not re.search('\s*<init(>|\s)', line):
                     header.append(line)
                     line = next(self._f[i])
                 # 'header' includes all contents before reaches <init>
                 self._header_str.append(''.join(header))
             self.check_header_compatibility()
 
             # Read <init> blocks for all input_files
             for i in range(len(self._f)):
                 init = []
                 line = next(self._f[i])
                 while not re.search('\s*</init>', line):
                     init.append(line)
                     line = next(self._f[i])
                 # 'init_str' includes all contents inside <init>...</init>
                 self._init_str.append(''.join(init))
 
             # Iterate over all events file-by-file and write events temporarily
             # to .tmp.lhe
             with open('.tmp.lhe', 'w') as _fwtmp:
                 for i in range(len(self._f)):
                     nevent = 0
                     while True:
                         line = next(self._f[i])
                         if re.search('\s*</event>', line):
                             nevent += 1
                         if re.search('\s*</LesHouchesEvents>', line):
                             break
                         _fwtmp.write(line)
                     self._nevent.append(nevent)
                     self._f[i].close()
 
             # Merge the header and init blocks and write to the output
             fw.write(self.merge_headers())
             fw.write('<init>\n' + self.merge_init_blocks() + '</init>\n')
 
             # Write event blocks in .tmp.lhe back to the output
             # If is MG5 LO LHE, will recalculate the weights based on combined xsec
             # and nevent read from <MGGenerationInfo>, and the 'event_norm' mode
             if self._is_mglo and not self.bypass_check:
                 event_norm = re.search(
                     r'\s(\w+)\s*=\s*event_norm\s',
                     self._header_str[0]).group(1)
                 if event_norm == 'sum':
                     self._uwgt = self._xsec_combined / sum(self._nevent)
                 elif event_norm == 'average':
                     self._uwgt = self._xsec_combined
                 logging.info(("MG5 LO LHE with event_norm = %s detected. Will "
                              "recalculate weights in each event block.\n"
                              "Unit weight: %+.7E") % (event_norm, self._uwgt))
 
                 # Modify event wgt when transfering .tmp.lhe to the output file
                 event_line = -999
                 with open('.tmp.lhe', 'r') as ftmp:
                     sign = lambda x: -1 if x < 0 else 1
                     for line in ftmp:
                         event_line += 1
                         if re.search('\s*<event.*>', line) and not re.search('\s*<event_num.*>', line):
                             event_line = 0
                         if event_line == 1:
                             # modify the XWGTUP appeared in the first line of the
                             # <event> block
                             orig_wgt = float(line.split()[2])
                             fw.write(re.sub(r'(^\s*\S+\s+\S+\s+)\S+(.+)', r'\g<1>%+.7E\g<2>' \
                                 % (sign(orig_wgt) * self._uwgt), line))
                         elif re.search('\s*<wgt.*>.*</wgt>', line):
                             addi_wgt_str = re.search(r'<wgt.*>\s*(\S+)\s*<\/wgt>', line).group(1)
                             fw.write(line.replace(
                                 addi_wgt_str, '%+.7E' % (float(addi_wgt_str) / orig_wgt * self._uwgt)))
                         else:
                             fw.write(line)
             else:
                 # Simply transfer all lines
                 with open('.tmp.lhe', 'r') as ftmp:
                     for line in ftmp:
                         fw.write(line)
             fw.write('</LesHouchesEvents>\n')
             os.remove('.tmp.lhe')
 
 

◆ merge_headers()

def mergeLHE.DefaultLHEMerger.merge_headers ( self )

Merge the headers of input LHEs. Need special handle for the MG5 LO case.

Definition at line 82 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger._header_str, mergeLHE.DefaultLHEMerger._is_mglo, mergeLHE.DefaultLHEMerger._nevent, mergeLHE.DefaultLHEMerger._xsec_combined, python.cmstools.all(), mergeLHE.DefaultLHEMerger.bypass_check, ALCARECOEcalPhiSym_cff.float, watchdog.group, python.rootplot.root2matplotlib.replace(), and reco.zip().

Referenced by mergeLHE.DefaultLHEMerger.merge().

     def merge_headers(self):
         """Merge the headers of input LHEs. Need special handle for the MG5 LO case."""
 
         self._is_mglo = all(['MGGenerationInfo' in header for header in self._header_str])
         if self._is_mglo and not self.bypass_check:
             # Special handling of MadGraph5 LO LHEs
             match_geninfo = [
                 re.search(
                     (r"<MGGenerationInfo>\s+#\s*Number of Events\s*\:\s*(\S+)\s+"
                     r"#\s*Integrated weight \(pb\)\s*\:\s*(\S+)\s+<\/MGGenerationInfo>"),
                     header
                     ) for header in self._header_str
                 ]
             self._xsec_combined = sum(
                 [float(info.group(2)) * nevt for info, nevt in zip(match_geninfo, self._nevent)]
                 ) / sum(self._nevent)
             geninfo_combined = ("<MGGenerationInfo>\n"
                                 "#  Number of Events        : %d\n"
                                 "#  Integrated weight (pb)  : %.10f\n</MGGenerationInfo>") \
                                     % (sum(self._nevent), self._xsec_combined)
             logging.info('Detected: MG5 LO LHEs. Input <MGGenerationInfo>:\n\tnevt\txsec')
             for info, nevt in zip(match_geninfo, self._nevent):
                 logging.info('\t%d\t%.10f' % (nevt, float(info.group(2))))
             logging.info('Combined <MGGenerationInfo>:\n\t%d\t%.10f' \
                 % (sum(self._nevent), self._xsec_combined))
 
             header_combined = self._header_str[0].replace(match_geninfo[0].group(), geninfo_combined)
             return header_combined
 
         else:
             # No need to merge the headers
             return self._header_str[0]
 

◆ merge_init_blocks()

def mergeLHE.DefaultLHEMerger.merge_init_blocks ( self )

If all <init> blocks are identical, return the same <init> block
(in the case of Powheg LHEs); otherwise, calculate the output <init>
blocks by merging the input blocks info using formula (same with the
MG5LOLHEMerger scheme):
    XSECUP = sum(xsecup * no.events) / tot.events
    XERRUP = sqrt( sum(sigma^2 * no.events^2) ) / tot.events
    XMAXUP = max(xmaxup)

Definition at line 115 of file mergeLHE.py.

References mergeLHE.DefaultLHEMerger._f, edmStreamStallGrapher.StallMonitorParser._f, edmTracerCompactLogViewer.TracerCompactFileParser._f, mergeLHE.DefaultLHEMerger._init_str, mergeLHE.DefaultLHEMerger._nevent, python.cmstools.all(), mergeLHE.DefaultLHEMerger.bypass_check, ALCARECOEcalPhiSym_cff.float, mergeLHE.BaseLHEMerger.input_files, DTWorkflow.DTWorkflow.input_files, createfilelist.int, relativeConstraints.keys, WZElectronSkims53X_cff.max, isotrackApplyRegressor.range, submitPVValidationJobs.split(), nano_mu_digi_cff.strip, and mkLumiAveragedPlots.tuple.

Referenced by mergeLHE.DefaultLHEMerger.merge().

     def merge_init_blocks(self):
         """If all <init> blocks are identical, return the same <init> block
         (in the case of Powheg LHEs); otherwise, calculate the output <init>
         blocks by merging the input blocks info using formula (same with the
         MG5LOLHEMerger scheme):
             XSECUP = sum(xsecup * no.events) / tot.events
             XERRUP = sqrt( sum(sigma^2 * no.events^2) ) / tot.events
             XMAXUP = max(xmaxup)
         """
 
         if self.bypass_check:
             # If bypass the consistency check, simply use the first LHE <init>
             # block as the output
             return self._init_str[0]
 
         # Initiate collected init block info. Will be in format of
         # {iprocess: [xsecup, xerrup, xmaxup]}
         new_init_block = {}
         old_init_block = [{} for _ in self._init_str]
 
         # Read the xsecup, xerrup, and xmaxup info from the <init> block for
         # all input LHEs
         for i, bl in enumerate(self._init_str): # loop over files
             nline = int(bl.split('\n')[0].strip().split()[-1])
 
             # loop over lines in <init> block
             for bl_line in bl.split('\n')[1:nline + 1]:
                 bl_line_sp = bl_line.split()
                 old_init_block[i][int(bl_line_sp[3])] = [
                     float(bl_line_sp[0]), float(bl_line_sp[1]), float(bl_line_sp[2])]
 
             # After reading all subprocesses info, store the rest content in
             # <init> block for the first file
             if i == 0:
                 info_after_subprocess = bl.strip().split('\n')[nline + 1:]
 
             logging.info('Input file: %s' % self.input_files[i])
             for ipr in sorted(list(old_init_block[i].keys()), reverse=True):
                 # reverse order: follow the MG5 custom
                 logging.info('  xsecup, xerrup, xmaxup, lprup: %.6E, %.6E, %.6E, %d' \
                     % tuple(old_init_block[i][ipr] + [ipr]))
 
         # Adopt smarter <init> block merging method
         # If all <init> blocks from input files are identical, return the same block;
         # otherwise combine them based on MG5LOLHEMerger scheme
         if all([old_init_block[i] == old_init_block[0] for i in range(len(self._f))]):
             # All <init> blocks are identical
             logging.info(
                 'All input <init> blocks are identical. Output the same "<init> block.')
             return self._init_str[0]
 
         # Otherwise, calculate merged init block
         for i in range(len(self._f)):
             for ipr in old_init_block[i]:
                 # Initiate the subprocess for the new block if it is found for the
                 # first time in one input file
                 if ipr not in new_init_block:
                     new_init_block[ipr] = [0., 0., 0.]
                 new_init_block[ipr][0] += old_init_block[i][ipr][0] * self._nevent[i] # xsecup
                 new_init_block[ipr][1] += old_init_block[i][ipr][1]**2 * self._nevent[i]**2 # xerrup
                 new_init_block[ipr][2] = max(new_init_block[ipr][2], old_init_block[i][ipr][2]) # xmaxup
         tot_nevent = sum([self._nevent[i] for i in range(len(self._f))])
 
         # Write first line of the <init> block (modify the nprocess at the last)
         self._merged_init_str = self._init_str[0].split('\n')[0].strip().rsplit(' ', 1)[0] \
             + ' ' + str(len(new_init_block)) + '\n'
         # Form the merged init block
         logging.info('Output file: %s' % self.output_file)
         for ipr in sorted(list(new_init_block.keys()), reverse=True):
             # reverse order: follow the MG5 custom
             new_init_block[ipr][0] /= tot_nevent
             new_init_block[ipr][1] = math.sqrt(new_init_block[ipr][1]) / tot_nevent
             logging.info('  xsecup, xerrup, xmaxup, lprup: %.6E, %.6E, %.6E, %d' \
                 % tuple(new_init_block[ipr] + [ipr]))
             self._merged_init_str += '%.6E %.6E %.6E %d\n' % tuple(new_init_block[ipr] + [ipr])
         self._merged_init_str += '\n'.join(info_after_subprocess)
         if len(info_after_subprocess):
             self._merged_init_str += '\n'
 
         return self._merged_init_str
 

Member Data Documentation

◆ _f

mergeLHE.DefaultLHEMerger._f

private

Definition at line 35 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), mergeLHE.DefaultLHEMerger.merge_init_blocks(), and edmModuleAllocMonitorAnalyze.ModuleAllocCompactFileParser.processingSteps().

◆ _header_lines

mergeLHE.DefaultLHEMerger._header_lines

private

Definition at line 61 of file mergeLHE.py.

◆ _header_str

mergeLHE.DefaultLHEMerger._header_str

private

Definition at line 36 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_headers().

◆ _init_str

mergeLHE.DefaultLHEMerger._init_str

private

Definition at line 40 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

◆ _is_mglo

mergeLHE.DefaultLHEMerger._is_mglo

private

Definition at line 37 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_headers().

◆ _merged_init_str

mergeLHE.DefaultLHEMerger._merged_init_str

private

Definition at line 179 of file mergeLHE.py.

◆ _nevent

mergeLHE.DefaultLHEMerger._nevent

private

Definition at line 41 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), mergeLHE.DefaultLHEMerger.merge_headers(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

◆ _uwgt

mergeLHE.DefaultLHEMerger._uwgt

private

Definition at line 39 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge().

◆ _xsec_combined

mergeLHE.DefaultLHEMerger._xsec_combined

private

Definition at line 38 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.merge(), and mergeLHE.DefaultLHEMerger.merge_headers().

◆ bypass_check

mergeLHE.DefaultLHEMerger.bypass_check

Definition at line 33 of file mergeLHE.py.

Referenced by mergeLHE.DefaultLHEMerger.check_header_compatibility(), mergeLHE.DefaultLHEMerger.merge(), mergeLHE.DefaultLHEMerger.merge_headers(), and mergeLHE.DefaultLHEMerger.merge_init_blocks().

Public Member Functions

Public Attributes

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ check_header_compatibility()

◆ file_iterator()

◆ merge()

◆ merge_headers()

◆ merge_init_blocks()

Member Data Documentation

◆ _f

◆ _header_lines

◆ _header_str

◆ _init_str

◆ _is_mglo

◆ _merged_init_str

◆ _nevent

◆ _uwgt

◆ _xsec_combined

◆ bypass_check

◆ init()