test
CMS 3D CMS Logo

 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Public Attributes | Static Public Attributes
personalPlayback.Playback Class Reference
Inheritance diagram for personalPlayback.Playback:
personalPlayback.Applet

Public Member Functions

def discover_files
 
def do_create_lumi
 
def do_exec
 
def do_init
 
- Public Member Functions inherited from personalPlayback.Applet
def __init__
 
def do_exec
 
def do_init
 
def log
 
def write
 

Public Attributes

 global_file
 
 input
 
 lumi_backlog
 
 lumi_backlog_size
 
 lumi_found
 
 lumi_order
 
 next_lumi_index
 
 output
 
 ramdisk
 
 run
 
- Public Attributes inherited from personalPlayback.Applet
 control_fp
 
 kwargs
 
 name
 
 opts
 

Static Public Attributes

tuple re_pattern = re.compile(r'run([0-9]+)_ls([0-9]+)_stream([A-Za-z0-9]+)_([A-Za-z0-9_-]+)\.jsn')
 

Additional Inherited Members

- Static Public Member Functions inherited from personalPlayback.Applet
def read
 

Detailed Description

Definition at line 63 of file personalPlayback.py.

Member Function Documentation

def personalPlayback.Playback.discover_files (   self)

Definition at line 66 of file personalPlayback.py.

66 
67  def discover_files(self):
68  self.lumi_found = {}
69 
70  files_found = set()
71  streams_found = set()
72  run_found = None
73  for f in os.listdir(self.input):
74  r = self.re_pattern.match(f)
75  if r:
76  run, lumi, stream, stream_source = r.groups()
77  run, lumi = int(run), int(lumi)
78 
79  if run_found is None:
80  run_found = run
81  elif run_found != run:
82  raise Exception("Files from multiple runs are not (yet) supported for as playback input.")
83 
84  lumi_dct = self.lumi_found.setdefault(lumi, { 'streams': {} })
85  lumi_dct["streams"][stream] = (f, stream_source)
86  files_found.add(f)
87  streams_found.add(stream)
88 
89  if run_found is None:
90  raise Exception("Playback files not found.")
91 
92  if self.run < 0:
93  self.run = run_found
94 
95  self.log.info("Found run %s, will map output to run %s", run_found, self.run)
96  self.log.info("Found %d lumisections with %d files", len(self.lumi_found), len(files_found))
97  self.log.info("Found %d streams: %s", len(streams_found), list(streams_found))
98 
99  self.lumi_order = list(self.lumi_found.keys())
100  self.lumi_order.sort()
101  self.log.info("Lumi order: %s", str(self.lumi_order))
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*","!HLTx*"if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL.It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of"!*"before the partial wildcard feature was incorporated).Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run
def personalPlayback.Playback.do_create_lumi (   self)

Definition at line 125 of file personalPlayback.py.

References WorkFlow.WorkFlow.input, personalPlayback.Playback.input, personalPlayback.Playback.lumi_backlog, personalPlayback.Playback.lumi_backlog_size, personalPlayback.Playback.lumi_found, personalPlayback.Playback.lumi_order, personalPlayback.Playback.next_lumi_index, GeometryComparisonPlotter.output, CastorLedAnalysis.output, HcalLedAnalysis.output, personalPlayback.Playback.output, PhysicsTools::Calibration::MVAComputer.output, ConfigBuilder.ConfigBuilder.output, RPCRunIOV::RunIOV_Item.run, TB06Reco.run, TB06RecoH2.run, RPCDQMObject.run, EfficiencyPlotter.run, WZInterestingEventSelector::event.run, MuonRecoTest.run, DTResolutionAnalysisTest.run, DTBlockedROChannelsTest.run, DTDataIntegrityTest.run, DTChamberEfficiencyTest.run, DTResolutionTest.run, ME::Header.run, DTEfficiencyTest.run, HitEff.run, personalPlayback.Playback.run, DQMNet::CoreObject.run, MatrixUtil.InputInfo.run, DTLocalTriggerBaseTest.run, conddblib.TimeType.run, and SiPixelErrorEstimation.run.

Referenced by personalPlayback.Playback.do_exec().

126  def do_create_lumi(self):
127  orig_lumi = self.lumi_order[(self.next_lumi_index - 1) % len(self.lumi_order)]
128  play_lumi = self.next_lumi_index;
129  self.next_lumi_index += 1
130 
131  self.log.info("Start copying lumi (original) %06d -> %06d (playback)", orig_lumi, play_lumi)
132 
133  lumi_dct = self.lumi_found[orig_lumi]
134  streams = lumi_dct["streams"]
135 
136  def ijoin(f):
137  return os.path.join(self.input, f)
138 
139  def ojoin(f):
140  return os.path.join(self.output, f)
141 
142  written_files = set()
143  for stream, v in streams.items():
144  jsn_orig_fn, stream_source = v
145  jsn_play_fn = "run%06d_ls%04d_stream%s_%s.jsn" % (self.run, play_lumi, stream, stream_source)
146 
147  # define dat filename
148  ext = "dat"
149  if stream.startswith("streamDQMHistograms"):
150  ext = "pb"
151  dat_play_fn = "run%06d_ls%04d_stream%s_%s.%s" % (self.run, play_lumi, stream, stream_source, ext)
152 
153  # read the original file name, for copying
154  with open(ijoin(jsn_orig_fn), 'r') as f:
155  jsn_data = json.load(f)
156  dat_orig_fn = jsn_data["data"][3]
157 
158  # copy the data file
159  if os.path.exists(ijoin(dat_orig_fn)):
160  self.log.info("C: %s -> %s", dat_orig_fn, dat_play_fn)
161  shutil.copyfile(ijoin(dat_orig_fn), ojoin(dat_play_fn))
162 
163  written_files.add(dat_play_fn)
164  else:
165  log.warning("Dat file is missing: %s", dat_orig_fn)
166 
167  # write a new json file point to a different data file
168  # this has to be atomic!
169  jsn_data["data"][3] = dat_play_fn
170 
171  f = tempfile.NamedTemporaryFile(prefix=jsn_play_fn+ ".", suffix=".tmp", dir = self.output, delete=False)
172  tmp_fp = f.name
173  json.dump(jsn_data, f)
174  f.close()
175 
176  os.rename(tmp_fp, ojoin(jsn_play_fn))
177  written_files.add(jsn_play_fn)
178 
179  self.log.info("Copied %d files for lumi %06d", len(written_files), play_lumi)
180 
181  self.lumi_backlog.append((play_lumi, written_files))
182  while len(self.lumi_backlog) > self.lumi_backlog_size:
183  old_lumi, files_to_delete = self.lumi_backlog.popleft()
184 
185  self.log.info("Deleting %d files for old lumi %06d", len(files_to_delete), old_lumi)
186  for f in files_to_delete:
187  os.unlink(ojoin(f))
def personalPlayback.Playback.do_exec (   self)

Definition at line 188 of file personalPlayback.py.

References personalPlayback.Playback.do_create_lumi(), GeometryComparisonPlotter.output, CastorLedAnalysis.output, HcalLedAnalysis.output, personalPlayback.Playback.output, PhysicsTools::Calibration::MVAComputer.output, ConfigBuilder.ConfigBuilder.output, RPCRunIOV::RunIOV_Item.run, TB06RecoH2.run, TB06Reco.run, RPCDQMObject.run, EfficiencyPlotter.run, WZInterestingEventSelector::event.run, MuonRecoTest.run, DTResolutionAnalysisTest.run, DTBlockedROChannelsTest.run, DTDataIntegrityTest.run, DTChamberEfficiencyTest.run, DTResolutionTest.run, ME::Header.run, DTEfficiencyTest.run, HitEff.run, personalPlayback.Playback.run, DQMNet::CoreObject.run, MatrixUtil.InputInfo.run, DTLocalTriggerBaseTest.run, conddblib.TimeType.run, and SiPixelErrorEstimation.run.

189  def do_exec(self):
190  last_write = 0
191  lumi_produced = 0
192 
193  while True:
194  time.sleep(1)
195 
196  now = time.time()
197  if (now - last_write) > self.opts.playback_time_lumi:
198  last_write = now
199 
200  if self.opts.playback_nlumi > -1 and lumi_produced >= self.opts.playback_nlumi:
201  break
202 
203  self.do_create_lumi()
204  lumi_produced += 1
205 
206  # write eor
207  eor_fn = "run%06d_ls0000_EoR.jsn" % (self.run, )
208  eor_fp = os.path.join(self.output, eor_fn)
209  with open(eor_fp, "w"):
210  pass
211 
212  self.log.info("Wrote EoR: %s", eor_fp)
def personalPlayback.Playback.do_init (   self)

Definition at line 102 of file personalPlayback.py.

103  def do_init(self):
104  # check if our input directory is okay
105  self.input = self.opts.playback
106  self.ramdisk = self.opts.work_ramdisk
107  self.run = self.opts.run
108  self.log.info("Using input directory: %s", self.input)
109 
110  self.discover_files()
112  self.output = os.path.join(self.ramdisk, "run%06d" % self.run)
113  if not os.path.isdir(self.output):
114  os.makedirs(self.output)
115  self.log.info("Using output directory: %s", self.output)
117  self.global_file = os.path.join(self.ramdisk, ".run%06d.global" % self.run)
118  self.log.info("Writing: %s", self.global_file)
119  with open(self.global_file, "w") as f:
120  f.write("run_key = pp_run")
122  self.lumi_backlog = collections.deque()
124  self.next_lumi_index = 1

Member Data Documentation

personalPlayback.Playback.global_file

Definition at line 116 of file personalPlayback.py.

personalPlayback.Playback.input

Definition at line 104 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

personalPlayback.Playback.lumi_backlog

Definition at line 121 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

personalPlayback.Playback.lumi_backlog_size

Definition at line 122 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

personalPlayback.Playback.lumi_found

Definition at line 67 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

personalPlayback.Playback.lumi_order

Definition at line 98 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

personalPlayback.Playback.next_lumi_index

Definition at line 123 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

personalPlayback.Playback.output

Definition at line 111 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi(), and personalPlayback.Playback.do_exec().

personalPlayback.Playback.ramdisk

Definition at line 105 of file personalPlayback.py.

Referenced by personalPlayback.FrameworkJob.discover_latest().

tuple personalPlayback.Playback.re_pattern = re.compile(r'run([0-9]+)_ls([0-9]+)_stream([A-Za-z0-9]+)_([A-Za-z0-9_-]+)\.jsn')
static

Definition at line 64 of file personalPlayback.py.

personalPlayback.Playback.run

Definition at line 92 of file personalPlayback.py.

Referenced by generateEDF.LumiInfo.__str__(), Types.EventID.cppID(), Types.LuminosityBlockID.cppID(), personalPlayback.Playback.do_create_lumi(), personalPlayback.Playback.do_exec(), generateEDF.LumiInfo.fixXingInfo(), and dqm_interfaces.DirWalkerFile.walk().