CMS 3D CMS Logo

List of all members | Public Member Functions | Private Attributes
TreeCrawler.mymf Class Reference
Inheritance diagram for TreeCrawler.mymf:

Public Member Functions

def __init__ (self, args, kwargs)
 
def import_hook (self, name, caller=None, fromlist=None, level=-1)
 
def import_module (self, partnam, fqname, parent)
 
def load_module (self, fqname, fp, pathname, aux_info)
 
def scan_opcodes_25 (self, co, unpack=struct.unpack)
 

Private Attributes

 _depgraph
 
 _globalarea
 
 _last_caller
 
 _localarea
 
 _types
 

Detailed Description

Definition at line 93 of file TreeCrawler.py.

Constructor & Destructor Documentation

◆ __init__()

def TreeCrawler.mymf.__init__ (   self,
  args,
  kwargs 
)

Definition at line 94 of file TreeCrawler.py.

94  def __init__(self,*args,**kwargs):
95  self._depgraph = {}
96  self._types = {}
97  self._last_caller = None
98  #TODO - replace by environment variables CMSSW_BASE and CMSSW_RELEASE_BASE (*and* do it only if the global one is not empty like for IB areas)
99  self._localarea = os.path.expandvars('$CMSSW_BASE')
100  self._globalarea = os.path.expandvars('$CMSSW_RELEASE_BASE')
101  modulefinder.ModuleFinder.__init__(self,*args,**kwargs)
def __init__(self, dataset, job_number, job_id, job_name, isDA, isMC, applyBOWS, applyEXTRACOND, extraconditions, runboundary, lumilist, intlumi, maxevents, gt, allFromGT, alignmentDB, alignmentTAG, apeDB, apeTAG, bowDB, bowTAG, vertextype, tracktype, refittertype, ttrhtype, applyruncontrol, ptcut, CMSSW_dir, the_dir)

Member Function Documentation

◆ import_hook()

def TreeCrawler.mymf.import_hook (   self,
  name,
  caller = None,
  fromlist = None,
  level = -1 
)

Definition at line 102 of file TreeCrawler.py.

References TreeCrawler.mymf._last_caller.

102  def import_hook(self, name, caller=None, fromlist=None, level=-1):
103  old_last_caller = self._last_caller
104  try:
105  self._last_caller = caller
106  return modulefinder.ModuleFinder.import_hook(self,name,caller,fromlist, level=level)
107  finally:
108  self._last_caller = old_last_caller
109 

◆ import_module()

def TreeCrawler.mymf.import_module (   self,
  partnam,
  fqname,
  parent 
)

Definition at line 110 of file TreeCrawler.py.

References TreeCrawler.mymf._depgraph, TreeCrawler.mymf._globalarea, TreeCrawler.mymf._last_caller, TreeCrawler.mymf._localarea, and python.rootplot.root2matplotlib.replace().

110  def import_module(self,partnam,fqname,parent):
111 
112  if partnam in ("os","unittest"):
113  r = None
114  else:
115  r = modulefinder.ModuleFinder.import_module(self,partnam,fqname,parent)
116  # since the modulefinder is not able to look into the global area when coming from the local area, we force a second try
117  if parent and not r and self._localarea != '' and self._globalarea != '':
118  parent.__file__ = parent.__file__.replace(self._localarea,self._globalarea)
119  parent.__path__[0] = parent.__path__[0].replace(self._localarea,self._globalarea)
120  r = modulefinder.ModuleFinder.import_module(self,partnam,fqname,parent)
121 
122  if r is not None:
123  self._depgraph.setdefault(self._last_caller.__name__,{})[r.__name__] = 1
124  return r
def replace(string, replacements)

◆ load_module()

def TreeCrawler.mymf.load_module (   self,
  fqname,
  fp,
  pathname,
  aux_info 
)

Definition at line 125 of file TreeCrawler.py.

References TreeCrawler.mymf._types, and hcaldqm::quantity::EventType._types.

125  def load_module(self, fqname, fp, pathname, aux_info):
126  (suffix, mode, type) = aux_info
127  r = modulefinder.ModuleFinder.load_module(self, fqname, fp, pathname, (suffix, mode, type))
128  if r is not None:
129  self._types[r.__name__] = type
130  return r
131 

◆ scan_opcodes_25()

def TreeCrawler.mymf.scan_opcodes_25 (   self,
  co,
  unpack = struct.unpack 
)
This is basically just the default opcode scanner from ModuleFinder, but extended to also
look for "process.load(<module>)' commands. Since the Process object might not necassarily
be called "process", it scans for a call to a "load" method with a single parameter on
*any* object. If one is found it checks if the parameter is a string that refers to a valid
python module in the local or global area. If it does, the scanner assumes this was a call
to a Process object and yields the module name.
It's not possible to scan first for Process object declarations to get the name of the
objects since often (e.g. for customisation functions) the object is passed to a function
in a different file.

The ModuleFinder.scan_opcodes_25 implementation this is based was taken from
https://hg.python.org/cpython/file/2.7/Lib/modulefinder.py#l364

Definition at line 132 of file TreeCrawler.py.

References TreeCrawler.mymf._globalarea, TreeCrawler.mymf._localarea, print(), and BeamSpotPI.unpack().

132  def scan_opcodes_25(self, co, unpack = struct.unpack):
133  """
134  This is basically just the default opcode scanner from ModuleFinder, but extended to also
135  look for "process.load(<module>)' commands. Since the Process object might not necassarily
136  be called "process", it scans for a call to a "load" method with a single parameter on
137  *any* object. If one is found it checks if the parameter is a string that refers to a valid
138  python module in the local or global area. If it does, the scanner assumes this was a call
139  to a Process object and yields the module name.
140  It's not possible to scan first for Process object declarations to get the name of the
141  objects since often (e.g. for customisation functions) the object is passed to a function
142  in a different file.
143 
144  The ModuleFinder.scan_opcodes_25 implementation this is based was taken from
145  https://hg.python.org/cpython/file/2.7/Lib/modulefinder.py#l364
146  """
147  # Scan the code, and yield 'interesting' opcode combinations
148  # Python 2.5 version (has absolute and relative imports)
149  code = co.co_code
150  names = co.co_names
151  consts = co.co_consts
152  LOAD_CONST = modulefinder.LOAD_CONST
153  IMPORT_NAME = modulefinder.IMPORT_NAME
154  STORE_OPS = modulefinder.STORE_OPS
155  HAVE_ARGUMENT = modulefinder.HAVE_ARGUMENT
156  LOAD_ATTR = chr(dis.opname.index('LOAD_ATTR'))
157  LOAD_NAME = chr(dis.opname.index('LOAD_NAME'))
158  CALL_FUNCTION = chr(dis.opname.index('CALL_FUNCTION'))
159  LOAD_LOAD_AND_IMPORT = LOAD_CONST + LOAD_CONST + IMPORT_NAME
160 
161  try :
162  indexOfLoadConst = names.index("load") # This might throw a ValueError
163  # These are the opcodes required to access the "load" attribute. This might
164  # not even be a function, but I check for that later.
165  loadMethodOpcodes = LOAD_ATTR+struct.pack('<H',indexOfLoadConst)
166  except ValueError :
167  # doesn't look like "load" is used anywhere in this file
168  loadMethodOpcodes=None
169 
170  while code:
171  c = code[0]
172 
173  # Check to see if this is a call to a "load" method
174  if loadMethodOpcodes!=None and len(code)>=9 : # Need at least 9 codes for the full call
175  if code[:3]==loadMethodOpcodes :
176  # The attribute "load" is being accessed, need to make sure this is a function call.
177  # I'll look ahead and see if the CALL_FUNCTION code is used - this could be in a different
178  # place depending on the number of arguments, but I'm only interested in methods with a
179  # single argument so I know exactly where CALL_FUNCTION should be.
180  if code[6]==CALL_FUNCTION :
181  # I know this is calling a method called "load" with one argument. I need
182  # to find out what the argument is. Note that I still don't know if this is
183  # on a cms.Process object.
184  indexInTable=unpack('<H',code[4:6])[0]
185  if code[3]==LOAD_CONST :
186  # The argument is a constant, so retrieve that from the table
187  loadMethodArgument=consts[indexInTable]
188  # I know a load method with one argument has been called on *something*, but I don't
189  # know if it was a cms.Process object. All I can do is check to see if the argument is
190  # a string, and if so if it refers to a python file in the user or global areas.
191  try :
192  loadMethodArgument = loadMethodArgument.replace("/",".")
193  # I can only use imp.find_module on submodules (i.e. each bit between a "."), so try
194  # that on each submodule in turn using the previously found filename. Note that I have
195  # to try this twice, because if the first pass traverses into a package in the local
196  # area but the subpackage has not been checked out it will report that the subpackage
197  # doesn't exist, even though it is available in the global area.
198  try :
199  parentFilename=[self._localarea+"/python"]
200  for subModule in loadMethodArgument.split(".") :
201  moduleInfo=imp.find_module( subModule, parentFilename )
202  parentFilename=[moduleInfo[1]]
203  # If control got this far without raising an exception, then it must be a valid python module
204  yield "import", (None, loadMethodArgument)
205  except ImportError :
206  # Didn't work in the local area, try in the global area.
207  parentFilename=[self._globalarea+"/python"]
208  for subModule in loadMethodArgument.split(".") :
209  moduleInfo=imp.find_module( subModule, parentFilename )
210  parentFilename=[moduleInfo[1]]
211  # If control got this far without raising an exception, then it must be a valid python module
212  yield "import", (None, loadMethodArgument)
213  except Exception as error:
214  # Either there was an import error (not a python module) or there was a string
215  # manipulaton error (argument not a string). Assume this wasn't a call on a
216  # cms.Process object and move on silently.
217  pass
218 
219  elif code[3]==LOAD_NAME :
220  # The argument is a variable. I can get the name of the variable quite easily but
221  # not the value, unless I execute all of the opcodes. Not sure what to do here,
222  # guess I'll just print a warning so that the user knows?
223  print("Unable to determine the value of variable '"+names[indexInTable]+"' to see if it is a proces.load(...) statement in file "+co.co_filename)
224 
225  code=code[9:]
226  continue
227 
228  if c in STORE_OPS:
229  oparg, = unpack('<H', code[1:3])
230  yield "store", (names[oparg],)
231  code = code[3:]
232  continue
233  if code[:9:3] == LOAD_LOAD_AND_IMPORT:
234  oparg_1, oparg_2, oparg_3 = unpack('<xHxHxH', code[:9])
235  level = consts[oparg_1]
236  if level == -1: # normal import
237  yield "import", (consts[oparg_2], names[oparg_3])
238  elif level == 0: # absolute import
239  yield "absolute_import", (consts[oparg_2], names[oparg_3])
240  else: # relative import
241  yield "relative_import", (level, consts[oparg_2], names[oparg_3])
242  code = code[9:]
243  continue
244  if c >= HAVE_ARGUMENT:
245  code = code[3:]
246  else:
247  code = code[1:]
248 
std::pair< unsigned int, unsigned int > unpack(cond::Time_t since)
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47

Member Data Documentation

◆ _depgraph

TreeCrawler.mymf._depgraph
private

Definition at line 95 of file TreeCrawler.py.

Referenced by TreeCrawler.mymf.import_module().

◆ _globalarea

TreeCrawler.mymf._globalarea
private

◆ _last_caller

TreeCrawler.mymf._last_caller
private

Definition at line 97 of file TreeCrawler.py.

Referenced by TreeCrawler.mymf.import_hook(), and TreeCrawler.mymf.import_module().

◆ _localarea

TreeCrawler.mymf._localarea
private

◆ _types

TreeCrawler.mymf._types
private