CMS 3D CMS Logo

List of all members | Public Member Functions | Public Attributes | Static Public Attributes
personalPlayback.Playback Class Reference
Inheritance diagram for personalPlayback.Playback:
personalPlayback.Applet

Public Member Functions

def discover_files (self)
 
def do_create_lumi (self)
 
def do_exec (self)
 
def do_init (self)
 
- Public Member Functions inherited from personalPlayback.Applet
def __init__ (self, name, opts, **kwargs)
 
def log (self)
 
def write (self, fp)
 

Public Attributes

 global_file
 
 input
 
 lumi_backlog
 
 lumi_backlog_size
 
 lumi_found
 
 lumi_order
 
 next_lumi_index
 
 output
 
 ramdisk
 
 run
 
- Public Attributes inherited from personalPlayback.Applet
 control_fp
 
 kwargs
 
 name
 
 opts
 

Static Public Attributes

 re_pattern
 

Additional Inherited Members

- Static Public Member Functions inherited from personalPlayback.Applet
def read (fp)
 

Detailed Description

Definition at line 64 of file personalPlayback.py.

Member Function Documentation

◆ discover_files()

def personalPlayback.Playback.discover_files (   self)

Definition at line 67 of file personalPlayback.py.

67  def discover_files(self):
68  self.lumi_found = {}
69 
70  files_found = set()
71  streams_found = set()
72  run_found = None
73  for f in os.listdir(self.input):
74  r = self.re_pattern.match(f)
75  if r:
76  run, lumi, stream, stream_source = r.groups()
77  run, lumi = int(run), int(lumi)
78 
79  if run_found is None:
80  run_found = run
81  elif run_found != run:
82  raise Exception("Files from multiple runs are not (yet) supported for as playback input.")
83 
84  lumi_dct = self.lumi_found.setdefault(lumi, { 'streams': {} })
85  lumi_dct["streams"][stream] = (f, stream_source)
86  files_found.add(f)
87  streams_found.add(stream)
88 
89  if run_found is None:
90  raise Exception("Playback files not found.")
91 
92  if self.run < 0:
93  self.run = run_found
94 
95  self.log.info("Found run %s, will map output to run %s", run_found, self.run)
96  self.log.info("Found %d lumisections with %d files", len(self.lumi_found), len(files_found))
97  self.log.info("Found %d streams: %s", len(streams_found), list(streams_found))
98 
99  self.lumi_order = list(self.lumi_found.keys())
100  self.lumi_order.sort()
101  self.log.info("Lumi order: %s", str(self.lumi_order))
102 

◆ do_create_lumi()

def personalPlayback.Playback.do_create_lumi (   self)

Definition at line 126 of file personalPlayback.py.

126  def do_create_lumi(self):
127  orig_lumi = self.lumi_order[(self.next_lumi_index - 1) % len(self.lumi_order)]
128  play_lumi = self.next_lumi_index;
129  self.next_lumi_index += 1
130 
131  self.log.info("Start copying lumi (original) %06d -> %06d (playback)", orig_lumi, play_lumi)
132 
133  lumi_dct = self.lumi_found[orig_lumi]
134  streams = lumi_dct["streams"]
135 
136  def ijoin(f):
137  return os.path.join(self.input, f)
138 
139  def ojoin(f):
140  return os.path.join(self.output, f)
141 
142  written_files = set()
143  for stream, v in streams.items():
144  jsn_orig_fn, stream_source = v
145  jsn_play_fn = "run%06d_ls%04d_stream%s_%s.jsn" % (self.run, play_lumi, stream, stream_source)
146 
147  # define dat filename
148  ext = "dat"
149  if stream.startswith("streamDQMHistograms"):
150  ext = "pb"
151  dat_play_fn = "run%06d_ls%04d_stream%s_%s.%s" % (self.run, play_lumi, stream, stream_source, ext)
152 
153  # read the original file name, for copying
154  with open(ijoin(jsn_orig_fn), 'r') as f:
155  jsn_data = json.load(f)
156  dat_orig_fn = jsn_data["data"][3]
157 
158  # copy the data file
159  if os.path.exists(ijoin(dat_orig_fn)):
160  self.log.info("C: %s -> %s", dat_orig_fn, dat_play_fn)
161  shutil.copyfile(ijoin(dat_orig_fn), ojoin(dat_play_fn))
162 
163  written_files.add(dat_play_fn)
164  else:
165  log.warning("Dat file is missing: %s", dat_orig_fn)
166 
167  # write a new json file point to a different data file
168  # this has to be atomic!
169  jsn_data["data"][3] = dat_play_fn
170 
171  f = tempfile.NamedTemporaryFile(prefix=jsn_play_fn+ ".", suffix=".tmp", dir = self.output, delete=False)
172  tmp_fp = f.name
173  json.dump(jsn_data, f)
174  f.close()
175 
176  os.rename(tmp_fp, ojoin(jsn_play_fn))
177  written_files.add(jsn_play_fn)
178 
179  self.log.info("Copied %d files for lumi %06d", len(written_files), play_lumi)
180 
181  self.lumi_backlog.append((play_lumi, written_files))
182  while len(self.lumi_backlog) > self.lumi_backlog_size:
183  old_lumi, files_to_delete = self.lumi_backlog.popleft()
184 
185  self.log.info("Deleting %d files for old lumi %06d", len(files_to_delete), old_lumi)
186  for f in files_to_delete:
187  os.unlink(ojoin(f))
188 

References mps_setup.append, info(), WorkFlow.WorkFlow.input, personalPlayback.Playback.input, personalPlayback.Applet.log(), conddbCopyTest.CopyTest.log(), conditionUploadTest.UploadTest.log(), cond::persistency::Logger.log(), crabFunctions.CrabTask.log, personalPlayback.Playback.lumi_backlog, personalPlayback.Playback.lumi_backlog_size, personalPlayback.Playback.lumi_found, personalPlayback.Playback.lumi_order, personalPlayback.Playback.next_lumi_index, GeometryComparisonPlotter.output, personalPlayback.Playback.output, PhysicsTools::Calibration::MVAComputer.output, RPCRunIOV::RunIOV_Item.run, TB06Reco.run, TB06RecoH2.run, RPCDQMObject.run, EfficiencyPlotter.run, WZInterestingEventSelector::event.run, MuonRecoTest.run, DTResolutionAnalysisTest.run, Point.run, BeamSpotRcdReader::theBSfromDB.run, BeamSpotOnlineLegacyRcdReader::theBSOfromDB.run, DTDataIntegrityTest.run, BeamSpotOnlineHLTRcdReader::theBSOfromDB.run, DTBlockedROChannelsTest.run, DTChamberEfficiencyTest.run, DTResolutionTest.run, ME::Header.run, DTEfficiencyTest.run, DQMNet::CoreObject.run, personalPlayback.Playback.run, DTLocalTriggerBaseTest.run, MatrixUtil.InputInfo.run, HitEff.run, cond::RunInfo_t.run, and SiPixelErrorEstimation.run.

Referenced by personalPlayback.Playback.do_exec().

◆ do_exec()

def personalPlayback.Playback.do_exec (   self)

Reimplemented from personalPlayback.Applet.

Definition at line 189 of file personalPlayback.py.

189  def do_exec(self):
190  last_write = 0
191  lumi_produced = 0
192 
193  while True:
194  time.sleep(1)
195 
196  now = time.time()
197  if (now - last_write) > self.opts.playback_time_lumi:
198  last_write = now
199 
200  if self.opts.playback_nlumi > -1 and lumi_produced >= self.opts.playback_nlumi:
201  break
202 
203  self.do_create_lumi()
204  lumi_produced += 1
205 
206  # write eor
207  eor_fn = "run%06d_ls0000_EoR.jsn" % (self.run, )
208  eor_fp = os.path.join(self.output, eor_fn)
209  with open(eor_fp, "w"):
210  pass
211 
212  self.log.info("Wrote EoR: %s", eor_fp)
213 

References personalPlayback.Playback.do_create_lumi(), info(), personalPlayback.Applet.log(), conddbCopyTest.CopyTest.log(), conditionUploadTest.UploadTest.log(), cond::persistency::Logger.log(), crabFunctions.CrabTask.log, personalPlayback.Applet.opts, GeometryComparisonPlotter.output, personalPlayback.Playback.output, PhysicsTools::Calibration::MVAComputer.output, RPCRunIOV::RunIOV_Item.run, RPCDQMObject.run, TB06Reco.run, TB06RecoH2.run, EfficiencyPlotter.run, WZInterestingEventSelector::event.run, MuonRecoTest.run, DTResolutionAnalysisTest.run, Point.run, BeamSpotRcdReader::theBSfromDB.run, BeamSpotOnlineHLTRcdReader::theBSOfromDB.run, DTDataIntegrityTest.run, BeamSpotOnlineLegacyRcdReader::theBSOfromDB.run, DTBlockedROChannelsTest.run, DTChamberEfficiencyTest.run, DTResolutionTest.run, DTEfficiencyTest.run, ME::Header.run, DQMNet::CoreObject.run, personalPlayback.Playback.run, DTLocalTriggerBaseTest.run, MatrixUtil.InputInfo.run, HitEff.run, cond::RunInfo_t.run, and SiPixelErrorEstimation.run.

◆ do_init()

def personalPlayback.Playback.do_init (   self)

Reimplemented from personalPlayback.Applet.

Definition at line 103 of file personalPlayback.py.

103  def do_init(self):
104  # check if our input directory is okay
105  self.input = self.opts.playback
106  self.ramdisk = self.opts.work_ramdisk
107  self.run = self.opts.run
108  self.log.info("Using input directory: %s", self.input)
109 
110  self.discover_files()
111 
112  self.output = os.path.join(self.ramdisk, "run%06d" % self.run)
113  if not os.path.isdir(self.output):
114  os.makedirs(self.output)
115  self.log.info("Using output directory: %s", self.output)
116 
117  self.global_file = os.path.join(self.ramdisk, ".run%06d.global" % self.run)
118  self.log.info("Writing: %s", self.global_file)
119  with open(self.global_file, "w") as f:
120  f.write("run_key = pp_run")
121 
122  self.lumi_backlog = collections.deque()
123  self.lumi_backlog_size = 10
124  self.next_lumi_index = 1
125 

Member Data Documentation

◆ global_file

personalPlayback.Playback.global_file

Definition at line 117 of file personalPlayback.py.

◆ input

personalPlayback.Playback.input

Definition at line 105 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

◆ lumi_backlog

personalPlayback.Playback.lumi_backlog

Definition at line 122 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

◆ lumi_backlog_size

personalPlayback.Playback.lumi_backlog_size

Definition at line 123 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

◆ lumi_found

personalPlayback.Playback.lumi_found

Definition at line 68 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

◆ lumi_order

personalPlayback.Playback.lumi_order

Definition at line 99 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

◆ next_lumi_index

personalPlayback.Playback.next_lumi_index

Definition at line 124 of file personalPlayback.py.

Referenced by personalPlayback.Playback.do_create_lumi().

◆ output

personalPlayback.Playback.output

◆ ramdisk

personalPlayback.Playback.ramdisk

Definition at line 106 of file personalPlayback.py.

Referenced by personalPlayback.FrameworkJob.discover_latest().

◆ re_pattern

personalPlayback.Playback.re_pattern
static

Definition at line 65 of file personalPlayback.py.

◆ run

personalPlayback.Playback.run
info
static const TGPicture * info(bool iBackgroundIsBlack)
Definition: FWCollectionSummaryWidget.cc:152
relativeConstraints.keys
keys
Definition: relativeConstraints.py:89
str
#define str(s)
Definition: TestProcessor.cc:48
Exception
mps_setup.append
append
Definition: mps_setup.py:85
match
std::pair< typename Association::data_type::first_type, double > match(Reference key, Association association, bool bestMatchByMaxValue)
Generic matching function.
Definition: Utils.h:10
personalPlayback.int
int
Definition: personalPlayback.py:465
list
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger list("!*", "!HLTx*" if it matches 2 triggers or more) will accept the event if all the matching triggers are FAIL. It will reject the event if any of the triggers are PASS or EXCEPTION(this matches the behavior of "!*" before the partial wildcard feature was incorporated). Triggers which are in the READY state are completely ignored.(READY should never be returned since the trigger paths have been run