CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
Functions | Variables
dqmiodatasetharvest Namespace Reference

Functions

def check_interesting
 
def dasquery
 
def harvestfile
 
def tosqlite
 

Variables

tuple args = parser.parse_args()
 
int ctr = 0
 
tuple db = sqlite3.connect(args.output)
 
string dumpmes
 
tuple files = dasquery(args.dataset)
 
tuple inf = re.compile("([- \[])inf([,}\]])")
 
string insertinto
 
list interesting_mes
 
dictionary interesting_types
 
string makeindex
 
string maketable
 
tuple nan = re.compile("([- \[])nan([,}\]])")
 
tuple parser = argparse.ArgumentParser(description="Collect a MEs from DQMIO data, with maximum possible granularity")
 
tuple pool = multiprocessing.Pool(processes=args.njobs)
 
string ROOTPREFIX = "root://cms-xrd-global.cern.ch/"
 
string sqlite2tree
 
dictionary treenames
 

Function Documentation

def dqmiodatasetharvest.check_interesting (   mename)

Definition at line 41 of file dqmiodatasetharvest.py.

Referenced by harvestfile().

41 
42 def check_interesting(mename):
43  for pattern in interesting_mes:
44  if fnmatch.fnmatch(mename,pattern):
45  return True
46  return False
def dqmiodatasetharvest.dasquery (   dataset)

Definition at line 70 of file dqmiodatasetharvest.py.

References print().

70 
71 def dasquery(dataset):
72  if not dataset.endswith("DQMIO"):
73  raise Exception("This tool probably cannot read the dataset you specified. The name should end with DQMIO.")
74  dasquery = ["dasgoclient", "-query=file dataset=%s" % dataset]
75  print("Querying das ... %s" % dasquery)
76  files = subprocess.check_output(dasquery)
77  files = files.splitlines()
78  print("Got %d files." % len(files))
79  return files
80 
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
def dqmiodatasetharvest.harvestfile (   fname)

Definition at line 123 of file dqmiodatasetharvest.py.

References check_interesting(), SiStripPI.max, SiStripPI.min, sistrip::SpyUtilities.range(), str, and tosqlite().

123 
124 def harvestfile(fname):
125  f = ROOT.TFile.Open(ROOTPREFIX + fname)
126  idxtree = getattr(f, "Indices")
127  #idxtree.GetEntry._threaded = True # now the blocking call should release the GIL...
128 
129  # we have no good way to find out which lumis where processed in a job.
130  # so we watch the per-lumi indices and assume that all mentioned lumis
131  # are covered in the end-of-job MEs. This might fail if there are no
132  # per-lumi MEs.
133  knownlumis = set()
134  mes_to_store = []
135 
136  for i in range(idxtree.GetEntries()):
137  idxtree.GetEntry(i)
138  run, lumi, metype = idxtree.Run, idxtree.Lumi, idxtree.Type
139  if lumi != 0:
140  knownlumis.add(lumi)
141 
142  if not treenames[metype] in interesting_types:
143  continue
144 
145  endrun = run # assume no multi-run files for now
146  if lumi == 0: # per-job ME
147  endlumi = max(knownlumis)
148  lumi = min(knownlumis)
149  else:
150  endlumi = lumi
151 
152  # inclusive range -- for 0 entries, row is left out
153  firstidx, lastidx = idxtree.FirstIndex, idxtree.LastIndex
154  metree = getattr(f, treenames[metype])
155  metree.GetEntry(0)
156  metree.SetBranchStatus("*",0)
157  metree.SetBranchStatus("FullName",1)
158 
159  for x in range(firstidx, lastidx+1):
160  metree.GetEntry(x)
161  mename = str(metree.FullName)
162 
163  if mename.find("AlCaReco") != -1:
164  continue
165 
166  if mename.find("Isolated") != -1:
167  continue
168 
169  if mename.find("HLT") != -1:
170  continue
171 
172  if not ((mename.find("SiStrip") >= 0) or (mename.find("OfflinePV") >= 0) or (mename.find("PixelPhase1") >= 0) or (mename.find("Tracking") >= 0 )):
173  continue
174 
175  if check_interesting(mename):
176  metree.GetEntry(x, 1)
177  value = metree.Value
178 
179  mes_to_store.append((
180  mename,
181  run, lumi, endrun, endlumi,
182  metype,
183  tosqlite(value),
184  ))
185 
186  return mes_to_store
sistrip::SpyUtilities::range
const uint16_t range(const Frame &aFrame)
Definition: SiStripSpyUtilities.cc:76
str
#define str(s)
Definition: TestProcessor.cc:54
def dqmiodatasetharvest.tosqlite (   x)

Definition at line 47 of file dqmiodatasetharvest.py.

References edmScanValgrind.buffer, and betterConfigParser.unicode.

Referenced by harvestfile().

47 
48 def tosqlite(x):
49  if isinstance(x, ROOT.string):
50  try:
51  return unicode(x.data())
52  except:
53  return buffer(x.data())
54  if isinstance(x, int):
55  return x
56  if isinstance(x, float):
57  return x
58  if isinstance(x, int):
59  return x
60  else:
61  try:
62  rootobj = unicode(ROOT.TBufferJSON.ConvertToJSON(x))
63  # turns out ROOT does not generate valid JSON for NaN/inf
64  clean = nan.sub('\\g<1>0\\g<2>', inf.sub('\\g<1>1e38\\g<2>', rootobj))
65  obj = json.loads(clean)
66  jsonobj = json.dumps(obj, allow_nan=False)
67  return jsonobj
68  except Exception as e:
69  return json.dumps({"root2sqlite_error": e.__repr__(), "root2sqlite_object": x.__repr__()})

Variable Documentation

tuple dqmiodatasetharvest.args = parser.parse_args()

Definition at line 21 of file dqmiodatasetharvest.py.

int dqmiodatasetharvest.ctr = 0

Definition at line 191 of file dqmiodatasetharvest.py.

Referenced by CaloCellCrossing.CaloCellCrossing(), Tracker_OldtoNewConverter.createMap(), MultiVertexFitter.fit(), MuonGeometryNumbering.geoHistoryToBaseNumber(), cms::MuonNumbering.geoHistoryToBaseNumber(), PreshowerStrip.initCorners(), EcalTBHodoscopeGeometryLoaderFromDDD.makeGeometry(), HLTPerformanceInfo.moduleIndexInPath(), MuonGeometryArrange.MuonGeometryArrange(), MultiClusterizer1D< T >.operator()(), HLTPerformanceInfo.setStatusOfModulesFromPath(), TrackerGeometryCompare.TrackerGeometryCompare(), and AdaptiveVertexReconstructor.vertices().

tuple dqmiodatasetharvest.db = sqlite3.connect(args.output)

Definition at line 119 of file dqmiodatasetharvest.py.

string dqmiodatasetharvest.dumpmes
Initial value:
1 = """
2  SELECT fromlumi, tolumi, fromrun, name, value FROM monitorelements ORDER BY fromrun, fromlumi ASC;
3 """

Definition at line 115 of file dqmiodatasetharvest.py.

tuple dqmiodatasetharvest.files = dasquery(args.dataset)

Definition at line 187 of file dqmiodatasetharvest.py.

tuple dqmiodatasetharvest.inf = re.compile("([- \[])inf([,}\]])")

Definition at line 38 of file dqmiodatasetharvest.py.

string dqmiodatasetharvest.insertinto
Initial value:
1 = """
2  INSERT INTO monitorelements (
3  name,
4  fromrun, fromlumi, torun, tolumi,
5  metype,
6  value
7  ) VALUES (
8  ?, ?, ?, ?, ?, ?, ?
9  ); """

Definition at line 106 of file dqmiodatasetharvest.py.

list dqmiodatasetharvest.interesting_mes
Initial value:
1 = [
2 
3 "PixelPhase1/Phase1_MechanicalView/PXBarrel/adc_PXLayer*",
4 
5 ]

Definition at line 32 of file dqmiodatasetharvest.py.

dictionary dqmiodatasetharvest.interesting_types
Initial value:
1 = {
2  "TH1Fs",
3  "TH1Ds",
4  "TH2Fs"
5 }

Definition at line 25 of file dqmiodatasetharvest.py.

string dqmiodatasetharvest.makeindex
Initial value:
1 = """
2  CREATE INDEX runorder ON monitorelements(fromrun, fromlumi);
3 """

Definition at line 103 of file dqmiodatasetharvest.py.

string dqmiodatasetharvest.maketable
Initial value:
1 = """
2  CREATE TABLE IF NOT EXISTS monitorelements (
3  name,
4  fromrun, fromlumi, torun, tolumi,
5  metype,
6  value
7  ); """

Definition at line 96 of file dqmiodatasetharvest.py.

tuple dqmiodatasetharvest.nan = re.compile("([- \[])nan([,}\]])")

Definition at line 39 of file dqmiodatasetharvest.py.

Referenced by L1TMuon::GeometryTranslator.calculateBendAngle(), L1TMuon::GeometryTranslator.calculateGlobalEta(), L1TMuon::GeometryTranslator.calculateGlobalPhi(), L1TMuon::GeometryTranslator.getGlobalPoint(), and RPCFw.UTtoT().

tuple dqmiodatasetharvest.parser = argparse.ArgumentParser(description="Collect a MEs from DQMIO data, with maximum possible granularity")

Definition at line 15 of file dqmiodatasetharvest.py.

tuple dqmiodatasetharvest.pool = multiprocessing.Pool(processes=args.njobs)

Definition at line 190 of file dqmiodatasetharvest.py.

string dqmiodatasetharvest.ROOTPREFIX = "root://cms-xrd-global.cern.ch/"

Definition at line 12 of file dqmiodatasetharvest.py.

string dqmiodatasetharvest.sqlite2tree

Definition at line 200 of file dqmiodatasetharvest.py.

dictionary dqmiodatasetharvest.treenames
Initial value:
1 = {
2  0: "Ints",
3  1: "Floats",
4  2: "Strings",
5  3: "TH1Fs",
6  4: "TH1Ss",
7  5: "TH1Ds",
8  6: "TH2Fs",
9  7: "TH2Ss",
10  8: "TH2Ds",
11  9: "TH3Fs",
12  10: "TProfiles",
13  11: "TProfile2Ds",
14 }

Definition at line 81 of file dqmiodatasetharvest.py.

Generated for CMSSW Reference Manual by doxygen 1.8.5