CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
List of all members | Public Member Functions | Static Public Member Functions | Private Types | Private Attributes
edm::DuplicateChecker Class Reference

#include <DuplicateChecker.h>

Public Member Functions

bool checkDisabled () const
 
bool checkingAllFiles () const
 
void disable ()
 
 DuplicateChecker (ParameterSet const &pset)
 
void inputFileClosed ()
 
void inputFileOpened (bool realData, IndexIntoFile const &indexIntoFile, std::vector< std::shared_ptr< IndexIntoFile > > const &indexesIntoFiles, std::vector< std::shared_ptr< IndexIntoFile > >::size_type currentIndexIntoFile)
 
bool isDuplicateAndCheckActive (int index, RunNumber_t run, LuminosityBlockNumber_t lumi, EventNumber_t event, std::string const &fileName)
 
bool noDuplicatesInFile () const
 

Static Public Member Functions

static void fillDescription (ParameterSetDescription &desc)
 

Private Types

enum  DataType { isRealData, isSimulation, unknown }
 
enum  DuplicateCheckMode { noDuplicateCheck, checkEachFile, checkEachRealDataFile, checkAllFilesOpened }
 

Private Attributes

DataType dataType_
 
bool disabled_
 
DuplicateCheckMode duplicateCheckMode_
 
bool itIsKnownTheFileHasNoDuplicates_
 
std::set
< IndexIntoFile::IndexRunLumiEventKey > 		relevantPreviousEvents_
 

Detailed Description

Definition at line 32 of file DuplicateChecker.h.

Member Enumeration Documentation

enum edm::DuplicateChecker::DataType
private
Enumerator
isRealData 
isSimulation 
unknown 

Definition at line 73 of file DuplicateChecker.h.

73 { isRealData, isSimulation, unknown };
enum edm::DuplicateChecker::DuplicateCheckMode
private
Enumerator
noDuplicateCheck 
checkEachFile 
checkEachRealDataFile 
checkAllFilesOpened 

Definition at line 69 of file DuplicateChecker.h.

69 { noDuplicateCheck, checkEachFile, checkEachRealDataFile, checkAllFilesOpened };

Constructor & Destructor Documentation

edm::DuplicateChecker::DuplicateChecker ( ParameterSet const &  pset)

Definition at line 13 of file DuplicateChecker.cc.

References checkAllFilesOpened, checkEachFile, checkEachRealDataFile, EcnaPython_AdcPeg12_S1_10_R170298_1_0_150_Dee0::duplicateCheckMode, duplicateCheckMode_, Exception, edm::ParameterSet::getUntrackedParameter(), noDuplicateCheck, and AlCaHLTBitMon_QueryRunRegistry::string.

13  :
14  dataType_(unknown),
15  itIsKnownTheFileHasNoDuplicates_(false),
16  disabled_(false)
17  {
18  // The default value provided as the second argument to the getUntrackedParameter function call
19  // is not used when the ParameterSet has been validated and the parameters are not optional
20  // in the description. This is currently true when PoolSource is the primary input source.
21  // The modules that use PoolSource as a SecSource have not defined their fillDescriptions function
22  // yet, so the ParameterSet does not get validated yet. As soon as all the modules with a SecSource
23  // have defined descriptions, the defaults in the getUntrackedParameterSet function calls can
24  // and should be deleted from the code.
25  std::string duplicateCheckMode =
26  pset.getUntrackedParameter<std::string>("duplicateCheckMode", std::string("checkAllFilesOpened"));
27 
28  if (duplicateCheckMode == std::string("noDuplicateCheck")) duplicateCheckMode_ = noDuplicateCheck;
29  else if (duplicateCheckMode == std::string("checkEachFile")) duplicateCheckMode_ = checkEachFile;
30  else if (duplicateCheckMode == std::string("checkEachRealDataFile")) duplicateCheckMode_ = checkEachRealDataFile;
31  else if (duplicateCheckMode == std::string("checkAllFilesOpened")) duplicateCheckMode_ = checkAllFilesOpened;
32  else {
33  throw cms::Exception("Configuration")
34  << "Illegal configuration parameter value passed to PoolSource for\n"
35  << "the \"duplicateCheckMode\" parameter, legal values are:\n"
36  << "\"noDuplicateCheck\", \"checkEachFile\", \"checkEachRealDataFile\", \"checkAllFilesOpened\"\n";
37  }
38  }
AlCaHLTBitMon_QueryRunRegistry.string
string string
Definition: AlCaHLTBitMon_QueryRunRegistry.py:255
EcnaPython_AdcPeg12_S1_10_R170298_1_0_150_Dee0.duplicateCheckMode
tuple duplicateCheckMode
Definition: EcnaPython_AdcPeg12_S1_10_R170298_1_0_150_Dee0.py:23
edm::DuplicateChecker::duplicateCheckMode_
DuplicateCheckMode duplicateCheckMode_
Definition: DuplicateChecker.h:71

Member Function Documentation

bool edm::DuplicateChecker::checkDisabled ( ) const
inline

Definition at line 49 of file DuplicateChecker.h.

References checkEachRealDataFile, dataType_, disabled_, duplicateCheckMode_, isSimulation, and noDuplicateCheck.

Referenced by inputFileOpened(), and isDuplicateAndCheckActive().

49  {
50  return duplicateCheckMode_ == noDuplicateCheck ||
51  (duplicateCheckMode_ == checkEachRealDataFile && dataType_ == isSimulation) ||
52  disabled_;
53  }
edm::DuplicateChecker::duplicateCheckMode_
DuplicateCheckMode duplicateCheckMode_
Definition: DuplicateChecker.h:71
bool edm::DuplicateChecker::checkingAllFiles ( ) const
inline

Definition at line 63 of file DuplicateChecker.h.

References checkAllFilesOpened, and duplicateCheckMode_.

63 {return checkAllFilesOpened == duplicateCheckMode_;}
edm::DuplicateChecker::duplicateCheckMode_
DuplicateCheckMode duplicateCheckMode_
Definition: DuplicateChecker.h:71
void edm::DuplicateChecker::disable ( )

Definition at line 40 of file DuplicateChecker.cc.

References dataType_, disabled_, itIsKnownTheFileHasNoDuplicates_, relevantPreviousEvents_, and unknown.

40  {
41  disabled_ = true;
42  dataType_ = unknown;
43  relevantPreviousEvents_.clear();
44  itIsKnownTheFileHasNoDuplicates_ = false;
45  }
edm::DuplicateChecker::relevantPreviousEvents_
std::set< IndexIntoFile::IndexRunLumiEventKey > relevantPreviousEvents_
Definition: DuplicateChecker.h:82
void edm::DuplicateChecker::fillDescription ( ParameterSetDescription &  desc)
static

Definition at line 119 of file DuplicateChecker.cc.

References edm::ParameterSetDescription::addUntracked(), and AlCaHLTBitMon_QueryRunRegistry::string.

Referenced by edm::RootPrimaryFileSequence::fillDescription().

119  {
120  std::string defaultString("checkAllFilesOpened");
121  desc.addUntracked<std::string>("duplicateCheckMode", defaultString)->setComment(
122  "'checkAllFilesOpened': check across all input files\n"
123  "'checkEachFile': check each input file independently\n"
124  "'checkEachRealDataFile': check each real data input file independently\n"
125  "'noDuplicateCheck': no duplicate checking\n"
126  );
127  }
AlCaHLTBitMon_QueryRunRegistry.string
string string
Definition: AlCaHLTBitMon_QueryRunRegistry.py:255
void edm::DuplicateChecker::inputFileClosed ( )

Definition at line 77 of file DuplicateChecker.cc.

References dataType_, itIsKnownTheFileHasNoDuplicates_, relevantPreviousEvents_, and unknown.

78  {
79  dataType_ = unknown;
80  relevantPreviousEvents_.clear();
81  itIsKnownTheFileHasNoDuplicates_ = false;
82  }
edm::DuplicateChecker::relevantPreviousEvents_
std::set< IndexIntoFile::IndexRunLumiEventKey > relevantPreviousEvents_
Definition: DuplicateChecker.h:82
void edm::DuplicateChecker::inputFileOpened ( bool  realData,
IndexIntoFile const &  indexIntoFile,
std::vector< std::shared_ptr< IndexIntoFile > > const &  indexesIntoFiles,
std::vector< std::shared_ptr< IndexIntoFile > >::size_type  currentIndexIntoFile 
)

Definition at line 47 of file DuplicateChecker.cc.

References checkAllFilesOpened, checkDisabled(), edm::IndexIntoFile::containsDuplicateEvents(), dataType_, duplicateCheckMode_, i, isRealData, isSimulation, itIsKnownTheFileHasNoDuplicates_, relevantPreviousEvents_, and edm::IndexIntoFile::set_intersection().

51  {
52 
53  dataType_ = realData ? isRealData : isSimulation;
54  if (checkDisabled()) return;
55 
56  relevantPreviousEvents_.clear();
57  itIsKnownTheFileHasNoDuplicates_ = false;
58 
59  if (duplicateCheckMode_ == checkAllFilesOpened) {
60 
61  // Compares the current IndexIntoFile to all the previous ones and saves any duplicates.
62  // One unintended thing, it also saves the duplicate runs and lumis.
63  for(std::vector<std::shared_ptr<IndexIntoFile> >::size_type i = 0; i < currentIndexIntoFile; ++i) {
64  if (indexesIntoFiles[i].get() != 0) {
65 
66  indexIntoFile.set_intersection(*indexesIntoFiles[i], relevantPreviousEvents_);
67  }
68  }
69  }
70  if (relevantPreviousEvents_.empty()) {
71  if(!indexIntoFile.containsDuplicateEvents()) {
72  itIsKnownTheFileHasNoDuplicates_ = true;
73  }
74  }
75  }
i
int i
Definition: DBlmapReader.cc:9
edm::DuplicateChecker::checkDisabled
bool checkDisabled() const
Definition: DuplicateChecker.h:49
trigger::size_type
uint16_t size_type
Definition: TriggerTypeDefs.h:19
edm::DuplicateChecker::relevantPreviousEvents_
std::set< IndexIntoFile::IndexRunLumiEventKey > relevantPreviousEvents_
Definition: DuplicateChecker.h:82
edm::DuplicateChecker::duplicateCheckMode_
DuplicateCheckMode duplicateCheckMode_
Definition: DuplicateChecker.h:71
bool edm::DuplicateChecker::isDuplicateAndCheckActive ( int  index,
RunNumber_t  run,
LuminosityBlockNumber_t  lumi,
EventNumber_t  event,
std::string const &  fileName 
)

Definition at line 84 of file DuplicateChecker.cc.

References checkAllFilesOpened, checkDisabled(), cond::persistency::duplicate(), duplicateCheckMode_, itIsKnownTheFileHasNoDuplicates_, and relevantPreviousEvents_.

88  {
89  if (itIsKnownTheFileHasNoDuplicates_) return false;
90  if (checkDisabled()) return false;
91 
92  IndexIntoFile::IndexRunLumiEventKey newEvent(index, run, lumi, event);
93  bool duplicate = !relevantPreviousEvents_.insert(newEvent).second;
94 
95  if (duplicate) {
96  if (duplicateCheckMode_ == checkAllFilesOpened) {
97  LogWarning("DuplicateEvent")
98  << "Duplicate Events found in entire set of input files.\n"
99  << "Both events were from run " << run
100  << " and luminosity block " << lumi
101  << " with event number " << event << ".\n"
102  << "The duplicate was from file " << fileName << ".\n"
103  << "The duplicate will be skipped.\n";
104  }
105  else {
106  LogWarning("DuplicateEvent")
107  << "Duplicate Events found in file " << fileName << ".\n"
108  << "Both events were from run " << run
109  << " and luminosity block " << lumi
110  << " with event number " << event << ".\n"
111  << "The duplicate will be skipped.\n";
112  }
113  return true;
114  }
115  return false;
116  }
edm::DuplicateChecker::checkDisabled
bool checkDisabled() const
Definition: DuplicateChecker.h:49
fjr2json.lumi
tuple lumi
Definition: fjr2json.py:35
cond::persistency::duplicate
static const boost::regex duplicate("duplicateIOV[[:print:]]+?[S|s]ince[=| ]([[:alnum:]]+?);.*")
MillePedeFileConverter_cfg.fileName
tuple fileName
Definition: MillePedeFileConverter_cfg.py:32
event
How EventSelector::AcceptEvent() decides whether to accept an event for output otherwise it is excluding the probing of A single or multiple positive and the trigger will pass if any such matching triggers are PASS or EXCEPTION[A criterion thatmatches no triggers at all is detected and causes a throw.] A single negative with an expectation of appropriate bit checking in the decision and the trigger will pass if any such matching triggers are FAIL or EXCEPTION A wildcarded negative criterion that matches more than one trigger in the trigger but the state exists so we define the behavior If all triggers are the negative crieriion will lead to accepting the event(this again matches the behavior of"!*"before the partial wildcard feature was incorporated).The per-event"cost"of each negative criterion with multiple relevant triggers is about the same as!*was in the past
edm::DuplicateChecker::relevantPreviousEvents_
std::set< IndexIntoFile::IndexRunLumiEventKey > relevantPreviousEvents_
Definition: DuplicateChecker.h:82
edm::DuplicateChecker::duplicateCheckMode_
DuplicateCheckMode duplicateCheckMode_
Definition: DuplicateChecker.h:71
bool edm::DuplicateChecker::noDuplicatesInFile ( ) const
inline

Definition at line 47 of file DuplicateChecker.h.

References itIsKnownTheFileHasNoDuplicates_.

47 { return itIsKnownTheFileHasNoDuplicates_; }

Member Data Documentation

DataType edm::DuplicateChecker::dataType_
private

Definition at line 75 of file DuplicateChecker.h.

Referenced by checkDisabled(), disable(), inputFileClosed(), and inputFileOpened().

bool edm::DuplicateChecker::disabled_
private

Definition at line 86 of file DuplicateChecker.h.

Referenced by checkDisabled(), and disable().

DuplicateCheckMode edm::DuplicateChecker::duplicateCheckMode_
private

Definition at line 71 of file DuplicateChecker.h.

Referenced by checkDisabled(), checkingAllFiles(), DuplicateChecker(), inputFileOpened(), and isDuplicateAndCheckActive().

bool edm::DuplicateChecker::itIsKnownTheFileHasNoDuplicates_
private

Definition at line 84 of file DuplicateChecker.h.

Referenced by disable(), inputFileClosed(), inputFileOpened(), isDuplicateAndCheckActive(), and noDuplicatesInFile().

std::set<IndexIntoFile::IndexRunLumiEventKey> edm::DuplicateChecker::relevantPreviousEvents_
private

Definition at line 82 of file DuplicateChecker.h.

Referenced by disable(), inputFileClosed(), inputFileOpened(), and isDuplicateAndCheckActive().

Generated for CMSSW Reference Manual by doxygen 1.8.5