CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Static Public Member Functions | Public Attributes
dataset.PrivateDataset Class Reference
Inheritance diagram for dataset.PrivateDataset:
dataset.BaseDataset

Public Member Functions

def __init__
 
def buildListOfFiles
 
def buildListOfFilesDBS
 
def getPrimaryDatasetEntries
 
- Public Member Functions inherited from dataset.BaseDataset
def __init__
 def init(self, name, user, pattern='. More...
 
def buildListOfBadFiles
 
def buildListOfFiles
 
def extractFileSizes
 
def getPrimaryDatasetEntries
 
def listOfFiles
 
def listOfGoodFiles
 
def listOfGoodFilesWithPrescale
 
def printFiles
 
def printInfo
 

Static Public Member Functions

def findPrimaryDatasetEntries
 
def findPrimaryDatasetNumFiles
 

Public Attributes

 files
 
- Public Attributes inherited from dataset.BaseDataset
 bad_files
 
 dbsInstance
 MM. More...
 
 files
 
 filesAndSizes
 
 good_files
 
 name
 
 pattern
 
 primaryDatasetEntries
 MM. More...
 
 report
 
 run_range
 
 user
 

Detailed Description

Definition at line 333 of file dataset.py.

Constructor & Destructor Documentation

def dataset.PrivateDataset.__init__ (   self,
  name,
  dbsInstance = None 
)

Definition at line 335 of file dataset.py.

335 
336  def __init__(self, name, dbsInstance=None):
337  super(PrivateDataset, self).__init__(name, 'PRIVATE', dbsInstance=dbsInstance)

Member Function Documentation

def dataset.PrivateDataset.buildListOfFiles (   self,
  pattern = '.*root' 
)

Definition at line 352 of file dataset.py.

352 
353  def buildListOfFiles(self, pattern='.*root'):
354  self.files = self.buildListOfFilesDBS(self.name, self.dbsInstance)
355 
def dataset.PrivateDataset.buildListOfFilesDBS (   self,
  name,
  dbsInstance 
)

Definition at line 338 of file dataset.py.

References dataset.CMSDataset.findPrimaryDatasetNumFiles(), and dataset.PrivateDataset.findPrimaryDatasetNumFiles().

338 
339  def buildListOfFilesDBS(self, name, dbsInstance):
340  entries = self.findPrimaryDatasetNumFiles(name, dbsInstance, -1, -1)
341  files = []
342  dbs = 'das_client.py --query="file dataset=%s instance=prod/%s" --limit=%s' % (name, dbsInstance, entries)
343  dbsOut = os.popen(dbs)
344  for line in dbsOut:
345  if line.find('/store')==-1:
346  continue
347  line = line.rstrip()
348  # print 'line',line
349  files.append(line)
350  #return ['root://eoscms//eos/cms%s' % f for f in files]
351  return files
dataset.PrivateDataset.findPrimaryDatasetNumFiles
def findPrimaryDatasetNumFiles
Definition: dataset.py:381
def dataset.PrivateDataset.findPrimaryDatasetEntries (   dataset,
  dbsInstance,
  runmin,
  runmax 
)
static

Definition at line 357 of file dataset.py.

Referenced by dataset.PrivateDataset.getPrimaryDatasetEntries().

357 
358  def findPrimaryDatasetEntries(dataset, dbsInstance, runmin, runmax):
359 
360  query, qwhat = dataset, "dataset"
361  if "#" in dataset: qwhat = "block"
362  if runmin >0 or runmax > 0:
363  if runmin == runmax:
364  query = "%s run=%d" % (query,runmin)
365  else:
366  print "WARNING: queries with run ranges are slow in DAS"
367  query = "%s run between [%d, %d]" % (query,runmin if runmin > 0 else 1, runmax if runmax > 0 else 999999)
368  dbs='das_client.py --query="summary %s=%s instance=prod/%s"'%(qwhat, query, dbsInstance)
369  dbsOut = os.popen(dbs).readlines()
370 
371  entries = []
372  for line in dbsOut:
373  line = line.replace('\n','')
374  if "nevents" in line:
375  entries.append(int(line.split(":")[1]))
376  if entries:
377  return sum(entries)
378  return -1
379 
def dataset.PrivateDataset.findPrimaryDatasetNumFiles (   dataset,
  dbsInstance,
  runmin,
  runmax 
)
static

Definition at line 381 of file dataset.py.

Referenced by dataset.PrivateDataset.buildListOfFilesDBS().

381 
382  def findPrimaryDatasetNumFiles(dataset, dbsInstance, runmin, runmax):
383 
384  query, qwhat = dataset, "dataset"
385  if "#" in dataset: qwhat = "block"
386  if runmin >0 or runmax > 0:
387  if runmin == runmax:
388  query = "%s run=%d" % (query,runmin)
389  else:
390  print "WARNING: queries with run ranges are slow in DAS"
391  query = "%s run between [%d, %d]" % (query,runmin if runmin > 0 else 1, runmax if runmax > 0 else 999999)
392  dbs='das_client.py --query="summary %s=%s instance=prod/%s"'%(qwhat, query, dbsInstance)
393  dbsOut = os.popen(dbs).readlines()
394 
395  entries = []
396  for line in dbsOut:
397  line = line.replace('\n','')
398  if "nfiles" in line:
399  entries.append(int(line.split(":")[1]))
400  if entries:
401  return sum(entries)
402  return -1
dataset.PrivateDataset.findPrimaryDatasetNumFiles
def findPrimaryDatasetNumFiles
Definition: dataset.py:381
def dataset.PrivateDataset.getPrimaryDatasetEntries (   self)

Definition at line 403 of file dataset.py.

References dataset.BaseDataset.dbsInstance, dataset.CMSDataset.findPrimaryDatasetEntries(), dataset.PrivateDataset.findPrimaryDatasetEntries(), ElectronMVAID.ElectronMVAID.name, counter.Counter.name, entry.name, average.Average.name, histograms.Histograms.name, TmModule.name, cond::persistency::TAG::NAME.name, cond::persistency::GLOBAL_TAG::NAME.name, core.autovars.NTupleVariable.name, cond::persistency::TAG::TIME_TYPE.name, genericValidation.GenericValidation.name, cond::persistency::GLOBAL_TAG::VALIDITY.name, cond::persistency::TAG::OBJECT_TYPE.name, cond::persistency::GLOBAL_TAG::DESCRIPTION.name, cond::persistency::TAG::SYNCHRONIZATION.name, preexistingValidation.PreexistingValidation.name, cond::persistency::GLOBAL_TAG::RELEASE.name, ora::RecordSpecImpl::Item.name, cond::persistency::TAG::END_OF_VALIDITY.name, cond::persistency::GLOBAL_TAG::SNAPSHOT_TIME.name, cond::persistency::TAG::DESCRIPTION.name, cond::persistency::GTEditorData.name, cond::persistency::GLOBAL_TAG::INSERTION_TIME.name, cond::persistency::TAG::LAST_VALIDATED_TIME.name, FWTGeoRecoGeometry::Info.name, Types._Untracked.name, cond::persistency::TAG::INSERTION_TIME.name, cond::persistency::TAG::MODIFICATION_TIME.name, dataset.BaseDataset.name, personalPlayback.Applet.name, ParameterSet.name, PixelDCSObject< class >::Item.name, analyzer.Analyzer.name, DQMRivetClient::LumiOption.name, MagCylinder.name, alignment.Alignment.name, ParSet.name, o2o.O2OJob.name, DQMRivetClient::ScaleFactorOption.name, SingleObjectCondition.name, EgHLTOfflineSummaryClient::SumHistBinData.name, XMLHTRZeroSuppressionLoader::_loaderBaseConfig.name, XMLRBXPedestalsLoader::_loaderBaseConfig.name, DQMGenericClient::EfficOption.name, cond::persistency::GTProxyData.name, core.autovars.NTupleObjectType.name, MyWatcher.name, edm::PathTimingSummary.name, lumi::TriggerInfo.name, cond::TimeTypeSpecs.name, edm::PathSummary.name, PixelEndcapLinkMaker::Item.name, perftools::EdmEventSize::BranchRecord.name, cond::persistency::GLOBAL_TAG_MAP::GLOBAL_TAG_NAME.name, FWTableViewManager::TableEntry.name, PixelBarrelLinkMaker::Item.name, cond::persistency::GLOBAL_TAG_MAP::RECORD.name, Mapper::definition< ScannerT >.name, EcalLogicID.name, cond::persistency::GLOBAL_TAG_MAP::LABEL.name, cond::persistency::GLOBAL_TAG_MAP::TAG_NAME.name, ExpressionHisto< T >.name, XMLProcessor::_loaderBaseConfig.name, DQMGenericClient::ProfileOption.name, cond::persistency::PAYLOAD::HASH.name, TreeCrawler.Package.name, cond::persistency::PAYLOAD::OBJECT_TYPE.name, cond::persistency::PAYLOAD::DATA.name, cond::persistency::PAYLOAD::STREAMER_INFO.name, cond::persistency::PAYLOAD::VERSION.name, options.ConnectionHLTMenu.name, MagGeoBuilderFromDDD::volumeHandle.name, cond::persistency::PAYLOAD::INSERTION_TIME.name, DQMGenericClient::NormOption.name, Node.name, DQMGenericClient::CDOption.name, FastHFShowerLibrary.name, h4DSegm.name, PhysicsTools::Calibration::Variable.name, cond::TagInfo_t.name, CounterChecker.name, EDMtoMEConverter.name, looper.Looper.name, MEtoEDM< T >::MEtoEDMObject.name, cond::persistency::IOV::TAG_NAME.name, TrackerSectorStruct.name, cond::persistency::IOV::SINCE.name, cond::persistency::IOV::PAYLOAD_HASH.name, cond::persistency::IOV::INSERTION_TIME.name, MuonGeometrySanityCheckPoint.name, config.Analyzer.name, options.HLTProcessOptions.name, config.Service.name, h2DSegm.name, core.autovars.NTupleSubObject.name, DQMNet::WaitObject.name, AlpgenParameterName.name, SiStripMonitorDigi.name, core.autovars.NTupleObject.name, cond::persistency::TAG_LOG::TAG_NAME.name, cond::persistency::TAG_LOG::EVENT_TIME.name, cond::persistency::TAG_LOG::USER_NAME.name, cond::persistency::TAG_LOG::HOST_NAME.name, cond::persistency::TAG_LOG::COMMAND.name, cond::persistency::TAG_LOG::ACTION.name, cond::persistency::TAG_LOG::USER_TEXT.name, core.autovars.NTupleCollection.name, FastTimerService::LuminosityDescription.name, conddblib.Tag.name, conddblib.GlobalTag.name, personalPlayback.FrameworkJob.name, plotscripts.SawTeethFunction.name, FastTimerService::ProcessDescription.name, hTMaxCell.name, cscdqm::ParHistoDef.name, BeautifulSoup.Tag.name, TiXmlAttribute.name, BeautifulSoup.SoupStrainer.name, and dataset.BaseDataset.run_range.

403 
404  def getPrimaryDatasetEntries(self):
405  runmin = -1
406  runmax = -1
407  if self.run_range is not None:
408  runmin = self.run_range[0]
409  runmax = self.run_range[1]
return self.findPrimaryDatasetEntries(self.name, self.dbsInstance, runmin, runmax)

Member Data Documentation

dataset.PrivateDataset.files

Definition at line 353 of file dataset.py.

Generated for CMSSW Reference Manual by doxygen 1.8.5