CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Private Member Functions | Private Attributes
validation.Validation Class Reference
Inheritance diagram for validation.Validation:
vertexPlots.VertexValidation

Public Member Functions

def __init__
 
def createHtmlReport
 
def doPlots
 
def download
 

Private Member Functions

def _doPlots
 
def _doPlotsFastFull
 
def _doPlotsPileup
 
def _getDirectoryName
 
def _getSelectionName
 

Private Attributes

 _fastsimSamples
 
 _fullsimSamples
 
 _newBaseDir
 
 _newFileModifier
 
 _newRelease
 
 _plotter
 
 _plotterDrawArgs
 
 _refRelease
 
 _refRepository
 
 _selectionName
 

Detailed Description

Base class for Tracking/Vertex validation.

Definition at line 345 of file validation.py.

Constructor & Destructor Documentation

def validation.Validation.__init__ (   self,
  fullsimSamples,
  fastsimSamples,
  refRelease,
  refRepository,
  newRelease,
  newRepository,
  newFileModifier = None,
  selectionName = "" 
) 
Constructor.

Arguments:
fullsimSamples -- List of Sample objects for FullSim samples (may be empty)
fastsimSamples -- List of Sample objects for FastSim samples (may be empty)
refRelease    -- String for reference CMSSW release
newRepository -- String for directory whete to put new files
newRelease     -- CMSSW release to be validated
refRepository  -- String for directory where reference root files are
newFileModifier -- If given, a function to modify the names of the new files (function takes a string and returns a string)
selectionName  -- If given, use this string as the selection name (appended to GlobalTag for directory names)

Definition at line 347 of file validation.py.

Referenced by vertexPlots.VertexValidation._init__().

347 
348  def __init__(self, fullsimSamples, fastsimSamples, refRelease, refRepository, newRelease, newRepository, newFileModifier=None, selectionName=""):
349  """Constructor.
350 
351  Arguments:
352  fullsimSamples -- List of Sample objects for FullSim samples (may be empty)
353  fastsimSamples -- List of Sample objects for FastSim samples (may be empty)
354  refRelease -- String for reference CMSSW release
355  newRepository -- String for directory whete to put new files
356  newRelease -- CMSSW release to be validated
357  refRepository -- String for directory where reference root files are
358  newFileModifier -- If given, a function to modify the names of the new files (function takes a string and returns a string)
359  selectionName -- If given, use this string as the selection name (appended to GlobalTag for directory names)
360  """
361  try:
362  self._newRelease = os.environ["CMSSW_VERSION"]
363  except KeyError:
364  print >>sys.stderr, 'Error: CMSSW environment variables are not available.'
365  print >>sys.stderr, ' Please run cmsenv'
366  sys.exit()
367 
368  self._fullsimSamples = fullsimSamples
369  self._fastsimSamples = fastsimSamples
370  self._refRelease = refRelease
371  self._refRepository = refRepository
372  self._newRelease = newRelease
373  self._newBaseDir = os.path.join(newRepository, self._newRelease)
374  self._newFileModifier = newFileModifier
375  self._selectionName = selectionName

Member Function Documentation

def validation.Validation._doPlots (   self,
  sample,
  harvestedFile,
  plotterFolder,
  dqmSubFolder 
)
private 
Do the real plotting work for a given sample and DQM subfolder

Definition at line 502 of file validation.py.

References validation._copySubDir(), validation._getGlobalTag(), validation.Validation._newBaseDir, html.HtmlReport._newBaseDir, validation.Validation._newRelease, validation.Validation._plotterDrawArgs, validation.Validation._refRelease, validation.Validation._refRepository, validation._stripRelease(), join(), and python.multivaluedict.map().

502 
503  def _doPlots(self, sample, harvestedFile, plotterFolder, dqmSubFolder):
504  """Do the real plotting work for a given sample and DQM subfolder"""
505  # Get GlobalTags
506  refGlobalTag = _getGlobalTag(sample, self._refRelease)
507  newGlobalTag = _getGlobalTag(sample, self._newRelease)
508 
509  # Construct selection string
510  selectionNameBase = ""
511  if sample.hasScenario():
512  selectionNameBase += "_"+sample.scenario()
513  selectionNameBase += "_"+sample.pileup()
514  newSelection = newGlobalTag+selectionNameBase+plotterFolder.getSelectionName(dqmSubFolder)
515  if sample.hasPileup():
516  newPu = sample.pileupType(self._newRelease)
517  if newPu != "":
518  newSelection += "_"+newPu
519  def _createRefSelection(selectionName):
520  sel = refGlobalTag+selectionNameBase+selectionName
521  if sample.hasPileup():
522  refPu = sample.pileupType(self._refRelease)
523  if refPu != "":
524  sel += "_"+refPu
525  return sel
526  refSelection = _createRefSelection(plotterFolder.getSelectionName(dqmSubFolder))
527 
528  valname = "val.{sample}.root".format(sample=sample.name())
529 
530  # Construct reference directory name, and open reference file it it exists
531  refValFile = None
532  triedRefValFiles = []
533  tmp = [self._refRepository, self._refRelease]
534  if sample.fastsim():
535  tmp.extend(["fastsim", self._refRelease])
536  for selName in plotterFolder.getSelectionNameIterator(dqmSubFolder):
537  refSel = _createRefSelection(selName)
538  refdir = os.path.join(*(tmp+[refSel, sample.name()]))
539 
540  # Open reference file if it exists
541  refValFilePath = os.path.join(refdir, valname)
542  if os.path.exists(refValFilePath):
543  refSelection = refSel
544  refValFile = ROOT.TFile.Open(refValFilePath)
545  break
546  else:
547  triedRefValFiles.append(refValFilePath)
548  if refValFile is None:
549  if len(triedRefValFiles) == 1:
550  print "Reference file %s not found" % triedRefValFiles[0]
551  else:
552  print "None of the possible reference files %s not found" % ",".join(triedRefValFiles)
553 
554  # Construct new directory name
555  tmp = []
556  if sample.fastsim():
557  tmp.extend(["fastsim", self._newRelease])
558  tmp.extend([newSelection, sample.name()])
559  newsubdir = os.path.join(*tmp)
560  newdir = os.path.join(self._newBaseDir, newsubdir)
561 
562  # Copy the relevant histograms to a new validation root file
563  # TODO: treat the case where dqmSubFolder is empty
564  newValFile = _copySubDir(harvestedFile, valname, plotterFolder.getPossibleDQMFolders(), dqmSubFolder.subfolder if dqmSubFolder is not None else None)
565  fileList = []
566 
567  # Do the plots
568  print "Comparing ref and new {sim} {sample} {translatedFolder}".format(
569  sim="FullSim" if not sample.fastsim() else "FastSim",
570  sample=sample.name(), translatedFolder=str(dqmSubFolder.translated) if dqmSubFolder is not None else "")
571  plotterFolder.create([refValFile, newValFile], [
572  "%s, %s %s" % (sample.name(), _stripRelease(self._refRelease), refSelection),
573  "%s, %s %s" % (sample.name(), _stripRelease(self._newRelease), newSelection)
574  ],
575  dqmSubFolder,
576  isPileupSample=sample.hasPileup()
577  )
578  fileList.extend(plotterFolder.draw(**self._plotterDrawArgs))
579  fileList.append(valname)
580 
581  newValFile.Close()
582  if refValFile is not None:
583  refValFile.Close()
584 
585  # Move plots to new directory
586  print "Moving plots and %s to %s" % (valname, newdir)
587  if not os.path.exists(newdir):
588  os.makedirs(newdir)
589  for f in fileList:
590  shutil.move(f, os.path.join(newdir, f))
591  return map(lambda n: os.path.join(newsubdir, n), fileList)
validation._copySubDir
def _copySubDir
Definition: validation.py:699
validation._getGlobalTag
def _getGlobalTag
Definition: validation.py:115
validation._stripRelease
def _stripRelease
Definition: validation.py:108
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
def validation.Validation._doPlotsFastFull (   self,
  fastSample,
  fullSample,
  plotterFolder,
  dqmSubFolder 
)
private 
Do the real plotting work for FastSim vs. FullSim for a given algorithm, quality flag, and sample.

Definition at line 592 of file validation.py.

References validation._getGlobalTag(), validation.Validation._newBaseDir, html.HtmlReport._newBaseDir, validation.Validation._newRelease, validation.Validation._plotterDrawArgs, validation._stripRelease(), and python.multivaluedict.map().

592 
593  def _doPlotsFastFull(self, fastSample, fullSample, plotterFolder, dqmSubFolder):
594  """Do the real plotting work for FastSim vs. FullSim for a given algorithm, quality flag, and sample."""
595  # Get GlobalTags
596  fastGlobalTag = _getGlobalTag(fastSample, self._newRelease)
597  fullGlobalTag = _getGlobalTag(fullSample, self._newRelease)
598 
599  # Construct selection string
600  tmp = plotterFolder.getSelectionName(dqmSubFolder)
601  fastSelection = fastGlobalTag+"_"+fastSample.pileup()+tmp
602  fullSelection = fullGlobalTag+"_"+fullSample.pileup()+tmp
603  if fullSample.hasPileup():
604  fullSelection += "_"+fullSample.pileupType(self._newRelease)
605  fastSelection += "_"+fastSample.pileupType(self._newRelease)
606 
607  # Construct directories for FastSim, FullSim, and for the results
608  fastdir = os.path.join(self._newBaseDir, "fastsim", self._newRelease, fastSelection, fastSample.name())
609  fulldir = os.path.join(self._newBaseDir, fullSelection, fullSample.name())
610  newsubdir = os.path.join("fastfull", self._newRelease, fastSelection, fastSample.name())
611  newdir = os.path.join(self._newBaseDir, newsubdir)
612 
613  # Open input root files
614  valname = "val.{sample}.root".format(sample=fastSample.name())
615  fastValFilePath = os.path.join(fastdir, valname)
616  if not os.path.exists(fastValFilePath):
617  print "FastSim file %s not found" % fastValFilePath
618  fullValFilePath = os.path.join(fulldir, valname)
619  if not os.path.exists(fullValFilePath):
620  print "FullSim file %s not found" % fullValFilePath
621 
622  fastValFile = ROOT.TFile.Open(fastValFilePath)
623  fullValFile = ROOT.TFile.Open(fullValFilePath)
624 
625  # Do plots
626  print "Comparing FullSim and FastSim {sample} {translatedFolder}".format(
627  sample=fastSample.name(), translatedFolder=str(dqmSubFolder.translated) if dqmSubFolder is not None else "")
628  plotterFolder.create([fullValFile, fastValFile], [
629  "FullSim %s, %s %s" % (fullSample.name(), _stripRelease(self._newRelease), fullSelection),
630  "FastSim %s, %s %s" % (fastSample.name(), _stripRelease(self._newRelease), fastSelection),
631  ],
632  dqmSubFolder,
633  isPileupSample=fastSample.hasPileup(),
634  requireAllHistograms=True
635  )
636  fileList = plotterFolder.draw(**self._plotterDrawArgs)
637 
638  fullValFile.Close()
639  fastValFile.Close()
640 
641  # Move plots to new directory
642  print "Moving plots to %s" % (newdir)
643  if not os.path.exists(newdir):
644  os.makedirs(newdir)
645  for f in fileList:
646  shutil.move(f, os.path.join(newdir, f))
647  return map(lambda n: os.path.join(newsubdir, n), fileList)
validation._getGlobalTag
def _getGlobalTag
Definition: validation.py:115
validation._stripRelease
def _stripRelease
Definition: validation.py:108
def validation.Validation._doPlotsPileup (   self,
  algo,
  quality,
  sample 
)
private 
Do the real plotting work for Old vs. New pileup scenarios for a given algorithm, quality flag, and sample.

Definition at line 649 of file validation.py.

References validation.Validation._getDirectoryName(), validation._getGlobalTag(), validation.Validation._getSelectionName(), validation.Validation._newBaseDir, html.HtmlReport._newBaseDir, validation.Validation._newRelease, validation.Validation._plotterDrawArgs, validation._stripRelease(), and python.multivaluedict.map().

649 
650  def _doPlotsPileup(self, algo, quality, sample):
651  """Do the real plotting work for Old vs. New pileup scenarios for a given algorithm, quality flag, and sample."""
652  # Get GlobalTags
653  newGlobalTag = _getGlobalTag(sample, self._newRelease)
654  refGlobalTag = newGlobalTag + "_OldPU"
655 
656  # Construct selection string
657  tmp = self._getSelectionName(quality, algo)
658  refSelection = refGlobalTag+"_"+sample.pileup()+tmp+"_"+sample.pileupType(self._newRelease)
659  newSelection = newGlobalTag+"_"+sample.pileup()+tmp+"_"+sample.pileupType(self._newRelease)
660 
661  # Construct directories for FastSim, FullSim, and for the results
662  refdir = os.path.join(self._newBaseDir, refSelection, sample.name())
663  newdir = os.path.join(self._newBaseDir, newSelection, sample.name())
664  resdir = os.path.join(self._newBaseDir, "pileup", self._newRelease, newSelection, sample.name())
665 
666  # Open input root files
667  valname = "val.{sample}.root".format(sample=sample.name())
668  refValFilePath = os.path.join(refdir, valname)
669  if not os.path.exists(refValFilePath):
670  print "Ref pileup file %s not found" % refValFilePath
671  newValFilePath = os.path.join(newdir, valname)
672  if not os.path.exists(newValFilePath):
673  print "New pileup file %s not found" % newValFilePath
674 
675  refValFile = ROOT.TFile.Open(refValFilePath)
676  newValFile = ROOT.TFile.Open(newValFilePath)
677 
678  # Do plots
679  print "Comparing Old and New pileup {sample} {algo} {quality}".format(
680  sample=sample.name(), algo=algo, quality=quality)
681  self._plotter.create([refValFile, newValFile], [
682  "%d BX %s, %s %s" % ({"25ns": 10, "50ns": 20}[sample.pileupType(self._newRelease)], sample.name(), _stripRelease(self._newRelease), refSelection),
683  "35 BX %s, %s %s" % (sample.name(), _stripRelease(self._newRelease), newSelection),
684  ],
685  subdir = self._getDirectoryName(quality, algo))
686  fileList = self._plotter.draw(**self._plotterDrawArgs)
687 
688  newValFile.Close()
689  refValFile.Close()
690 
691  # Move plots to new directory
692  print "Moving plots to %s" % (resdir)
693  if not os.path.exists(resdir):
694  os.makedirs(resdir)
695  for f in fileList:
696  shutil.move(f, os.path.join(resdir, f))
697  subdir = newdir.replace(self._newBaseDir+"/", "")
698  return map(lambda n: os.path.join(subdir, n), fileList)
validation._getGlobalTag
def _getGlobalTag
Definition: validation.py:115
validation._stripRelease
def _stripRelease
Definition: validation.py:108
def validation.Validation._getDirectoryName (   self,
  args,
  kwargs 
)
private

Definition at line 376 of file validation.py.

Referenced by validation.Validation._doPlotsPileup().

376 
377  def _getDirectoryName(self, *args, **kwargs):
378  return None
def validation.Validation._getSelectionName (   self,
  args,
  kwargs 
)
private

Definition at line 379 of file validation.py.

References validation.Validation._selectionName.

Referenced by validation.Validation._doPlotsPileup().

379 
380  def _getSelectionName(self, *args, **kwargs):
381  return self._selectionName
def validation.Validation.createHtmlReport (   self)

Definition at line 430 of file validation.py.

References validation.Validation._newBaseDir, html.HtmlReport._newBaseDir, and validation.Validation._newRelease.

430 
431  def createHtmlReport(self):
432  baseUrl = "http://cmsdoc.cern.ch/cms/Physics/tracking/validation/MC/%s/" % self._newRelease
433  return html.HtmlReport(self._newRelease, self._newBaseDir, baseUrl)
def validation.Validation.doPlots (   self,
  plotter,
  plotterDrawArgs = {},
  limitSubFoldersOnlyTo = None,
  htmlReport = html.HtmlReportDummy(),
  doFastVsFull = True 
) 
Create validation plots.

Arguments:
plotter       -- plotting.Plotter object that does the plotting

Keyword arguments:
plotterDrawArgs -- Dictionary for additional arguments to Plotter.draw() (default: {})
limitSubFoldersOnlyTo   -- If not None, should be a dictionary from string to an object. The string is the name of a PlotFolder, and the object is PlotFolder-type specific to limit the subfolders to be processed. In general case the object is a list of strings, but e.g. for track iteration plots it is a function taking the algo and quality as parameters.
htmlReport      -- Object returned by createHtmlReport(), in case HTML report generation is desired
doFastVsFull    -- Do FastSim vs. FullSim comparison? (default: True)

Definition at line 434 of file validation.py.

434 
435  def doPlots(self, plotter, plotterDrawArgs={}, limitSubFoldersOnlyTo=None, htmlReport=html.HtmlReportDummy(), doFastVsFull=True):
436  """Create validation plots.
437 
438  Arguments:
439  plotter -- plotting.Plotter object that does the plotting
440 
441  Keyword arguments:
442  plotterDrawArgs -- Dictionary for additional arguments to Plotter.draw() (default: {})
443  limitSubFoldersOnlyTo -- If not None, should be a dictionary from string to an object. The string is the name of a PlotFolder, and the object is PlotFolder-type specific to limit the subfolders to be processed. In general case the object is a list of strings, but e.g. for track iteration plots it is a function taking the algo and quality as parameters.
444  htmlReport -- Object returned by createHtmlReport(), in case HTML report generation is desired
445  doFastVsFull -- Do FastSim vs. FullSim comparison? (default: True)
446  """
447  self._plotter = plotter
448  self._plotterDrawArgs = plotterDrawArgs
449 
450  # New vs. Ref
451  for sample in self._fullsimSamples+self._fastsimSamples:
452  # Check that the new DQM file exists
453  harvestedFile = sample.filename(self._newRelease)
454  if not os.path.exists(harvestedFile):
455  print "Harvested file %s does not exist!" % harvestedFile
456  sys.exit(1)
457 
458  plotterInstance = plotter.readDirs(harvestedFile)
459  htmlReport.beginSample(sample)
460  for plotterFolder, dqmSubFolder in plotterInstance.iterFolders(limitSubFoldersOnlyTo=limitSubFoldersOnlyTo):
461  if plotterFolder.onlyForPileup() and not sample.hasPileup():
462  continue
463  plotFiles = self._doPlots(sample, harvestedFile, plotterFolder, dqmSubFolder)
464  htmlReport.addPlots(plotterFolder, dqmSubFolder, plotFiles)
465  # TODO: the pileup case is still to be migrated
466 # if s.fullsim() and s.hasPileup():
467 # self._doPlotsPileup(a, q, s)
468 
469  # Fast vs. Full
470  if not doFastVsFull:
471  return
472  for fast in self._fastsimSamples:
473  correspondingFull = None
474  for full in self._fullsimSamples:
475  if fast.name() != full.name():
476  continue
477  if fast.hasPileup():
478  if not full.hasPileup():
479  continue
480  if fast.fastsimCorrespondingFullsimPileup() != full.pileupType():
481  continue
482  else:
483  if full.hasPileup():
484  continue
485 
486  if correspondingFull is None:
487  correspondingFull = full
488  else:
489  raise Exception("Got multiple compatible FullSim samples for FastSim sample %s %s" % (fast.name(), fast.pileup()))
490  if correspondingFull is None:
491  raise Exception("Did not find compatible FullSim sample for FastSim sample %s %s" % (fast.name(), fast.pileup()))
492 
493  # If we reach here, the harvestedFile must exist
494  harvestedFile = fast.filename(self._newRelease)
495  plotterInstance = plotter.readDirs(harvestedFile)
496  htmlReport.beginSample(fast, fastVsFull=True)
497  for plotterFolder, dqmSubFolder in plotterInstance.iterFolders(limitSubFoldersOnlyTo=limitSubFoldersOnlyTo):
498  if plotterFolder.onlyForPileup() and not fast.hasPileup():
499  continue
500  plotFiles = self._doPlotsFastFull(fast, correspondingFull, plotterFolder, dqmSubFolder)
501  htmlReport.addPlots(plotterFolder, dqmSubFolder, plotFiles)
def validation.Validation.download (   self) 
Download DQM files. Requires grid certificate and asks your password for it.

Definition at line 382 of file validation.py.

References validation.Validation._fastsimSamples, validation.Validation._fullsimSamples, validation._getRelValUrl(), validation.Validation._newFileModifier, validation.Validation._newRelease, alcazmumu_cfi.filter, join(), and python.multivaluedict.map().

382 
383  def download(self):
384  """Download DQM files. Requires grid certificate and asks your password for it."""
385  filenames = [s.filename(self._newRelease) for s in self._fullsimSamples+self._fastsimSamples]
386  if self._newFileModifier is not None:
387  filenames = map(self._newFileModifier, filenames)
388  filenames = filter(lambda f: not os.path.exists(f), filenames)
389  if len(filenames) == 0:
390  print "All files already downloaded"
391  return
392 
393  relvalUrl = _getRelValUrl(self._newRelease)
394  urls = [relvalUrl+f for f in filenames]
395  certfile = os.path.join(os.environ["HOME"], ".globus", "usercert.pem")
396  if not os.path.exists(certfile):
397  print "Certificate file {certfile} does not exist, unable to download RelVal files from {url}".format(certfile=certfile, url=relvalUrl)
398  sys.exit(1)
399  keyfile = os.path.join(os.environ["HOME"], ".globus", "userkey.pem")
400  if not os.path.exists(certfile):
401  print "Private key file {keyfile} does not exist, unable to download RelVal files from {url}".format(keyfile=keyfile, url=relvalUrl)
402  sys.exit(1)
403 
404  cmd = ["curl", "--cert-type", "PEM", "--cert", certfile, "--key", keyfile, "-k"]
405  for u in urls:
406  cmd.extend(["-O", u])
407  print "Downloading %d files from RelVal URL %s:" % (len(filenames), relvalUrl)
408  print " "+"\n ".join(filenames)
409  print "Please provide your private key pass phrase when curl asks it"
410  ret = subprocess.call(cmd)
411  if ret != 0:
412  print "Downloading failed with exit code %d" % ret
413  sys.exit(1)
414 
415  # verify
416  allFine = True
417  for f in filenames:
418  p = subprocess.Popen(["file", f], stdout=subprocess.PIPE)
419  stdout = p.communicate()[0]
420  if p.returncode != 0:
421  print "file command failed with exit code %d" % p.returncode
422  sys.exit(1)
423  if not "ROOT" in stdout:
424  print "File {f} is not ROOT, please check the correct version, GobalTag etc. from {url}".format(f=f, url=relvalUrl)
425  allFine = False
426  if os.path.exists(f):
427  os.remove(f)
428  if not allFine:
429  sys.exit(1)
alcazmumu_cfi.filter
tuple filter
Definition: alcazmumu_cfi.py:34
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:18
validation._getRelValUrl
def _getRelValUrl
Definition: validation.py:159

Member Data Documentation

validation.Validation._fastsimSamples
private

Definition at line 368 of file validation.py.

Referenced by validation.Validation.download().

validation.Validation._fullsimSamples
private

Definition at line 367 of file validation.py.

Referenced by validation.Validation.download().

validation.Validation._newBaseDir
private

Definition at line 372 of file validation.py.

Referenced by validation.Validation._doPlots(), validation.Validation._doPlotsFastFull(), validation.Validation._doPlotsPileup(), and validation.Validation.createHtmlReport().

validation.Validation._newFileModifier
private

Definition at line 373 of file validation.py.

Referenced by validation.Validation.download().

validation.Validation._newRelease
private

Definition at line 361 of file validation.py.

Referenced by validation.Validation._doPlots(), validation.Validation._doPlotsFastFull(), validation.Validation._doPlotsPileup(), validation.Validation.createHtmlReport(), and validation.Validation.download().

validation.Validation._plotter
private

Definition at line 446 of file validation.py.

validation.Validation._plotterDrawArgs
private

Definition at line 447 of file validation.py.

Referenced by validation.Validation._doPlots(), validation.SimpleValidation._doPlots(), validation.Validation._doPlotsFastFull(), and validation.Validation._doPlotsPileup().

validation.Validation._refRelease
private

Definition at line 369 of file validation.py.

Referenced by validation.Validation._doPlots().

validation.Validation._refRepository
private

Definition at line 370 of file validation.py.

Referenced by validation.Validation._doPlots().

validation.Validation._selectionName
private

Definition at line 374 of file validation.py.

Referenced by validation.Validation._getSelectionName().

Generated for CMSSW Reference Manual by doxygen 1.8.5