cmsHarvester.CMSHarvester Class Reference

CMSHarvester class. More...

Inheritance diagram for cmsHarvester.CMSHarvester:

Public Member Functions
def	__init__
def	cleanup
def	config_file_header
def	db_account_name_cms_cond_dqm_summary
def	db_account_name_cms_cond_globaltag
def	format_conditions_string
def	ident_string
def	option_handler_book_keeping_file
def	option_handler_debug
def	option_handler_force
def	option_handler_frontier_connection
def	option_handler_globaltag
def	option_handler_harvesting_mode
def	option_handler_harvesting_type
def	option_handler_input_Jsonfile
def	option_handler_input_Jsonrunfile
def	option_handler_input_spec
def	option_handler_input_todofile
def	option_handler_no_ref_hists
def	option_handler_quiet
def	option_handler_ref_hist_mapping_file
def	set_output_level
def	time_stamp

Public Attributes
	all_sites_found
	book_keeping_file_name
	book_keeping_file_name_default
	book_keeping_information
	caf_access
	castor_base_dir
	castor_base_dir_default
	castor_prefix
	cmd_line_opts
	crab_submission
	datasets_to_ignore
	datasets_to_use
	force_running
	frontier_connection_name
	frontier_connection_overridden
	globaltag
	globaltag_check_cache
	harvesting_info
	harvesting_mode
	harvesting_mode_default
	harvesting_modes
	harvesting_type
	harvesting_types
	input_method
	input_name
	Jsonfilename
	Jsonlumi
	Jsonrunfilename
	log_level
	logger
	no_matching_site_found_str
	non_t1access
	nr_max_sites
	preferred_site
	ref_hist_mappings
	ref_hist_mappings_file_name
	ref_hist_mappings_file_name_default
	runs_to_ignore
	runs_to_use
	saveByLumiSection
	sites_and_versions_cache
	todofile
	use_ref_hists
	version

Detailed Description

CMSHarvester class.

Class to perform CMS harvesting.

More documentation `obviously' to follow.

Definition at line 295 of file cmsHarvester.py.

Constructor & Destructor Documentation

def cmsHarvester.CMSHarvester.__init__	(		self,
			cmd_line_opts = None
	)

Definition at line 304 of file cmsHarvester.py.

    def __init__(self, cmd_line_opts=None):
        "Initialize class and process command line options."

        self.version = __version__

        # These are the harvesting types allowed. See the head of this
        # file for more information.
        self.harvesting_types = [
            "RelVal",
            "RelValFS",
            "MC",
            "DQMOffline",
            ]

        # These are the possible harvesting modes:
        #   - Single-step: harvesting takes place on-site in a single
        #   step. For each samples only a single ROOT file containing
        #   the harvesting results is returned.
        #   - Single-step-allow-partial: special hack to allow
        #   harvesting of partial statistics using single-step
        #   harvesting on spread-out samples.
        #   - Two-step: harvesting takes place in two steps. The first
        #   step returns a series of monitoring elenent summaries for
        #   each sample. The second step then merges these summaries
        #   locally and does the real harvesting. This second step
        #   produces the ROOT file containing the harvesting results.
        self.harvesting_modes = [
            "single-step",
            "single-step-allow-partial",
            "two-step"
            ]

        # It is possible to specify a GlobalTag that will override any
        # choices (regarding GlobalTags) made by the cmsHarvester.
        # BUG BUG BUG
        # For the moment, until I figure out a way to obtain the
        # GlobalTag with which a given data (!) dataset was created,
        # it is necessary to specify a GlobalTag for harvesting of
        # data.
        # BUG BUG BUG end
        self.globaltag = None

        # It's also possible to switch off the use of reference
        # histograms altogether.
        self.use_ref_hists = True

        # The database name and account are hard-coded. They are not
        # likely to change before the end-of-life of this tool. But of
        # course there is a way to override this from the command
        # line. One can even override the Frontier connection used for
        # the GlobalTag and for the reference histograms
        # independently. Please only use this for testing purposes.
        self.frontier_connection_name = {}
        self.frontier_connection_name["globaltag"] = "frontier://" \
                                                     "FrontierProd/"
        self.frontier_connection_name["refhists"] = "frontier://" \
                                                    "FrontierProd/"
        self.frontier_connection_overridden = {}
        for key in self.frontier_connection_name.keys():
            self.frontier_connection_overridden[key] = False

        # This contains information specific to each of the harvesting
        # types. Used to create the harvesting configuration. It is
        # filled by setup_harvesting_info().
        self.harvesting_info = None

        ###

        # These are default `unused' values that will be filled in
        # depending on the command line options.

        # The type of harvesting we're doing. See
        # self.harvesting_types for allowed types.
        self.harvesting_type = None

        # The harvesting mode, popularly known as single-step
        # vs. two-step. The thing to remember at this point is that
        # single-step is only possible for samples located completely
        # at a single site (i.e. SE).
        self.harvesting_mode = None
        # BUG BUG BUG
        # Default temporarily set to two-step until we can get staged
        # jobs working with CRAB.
        self.harvesting_mode_default = "single-step"
        # BUG BUG BUG end

        # The input method: are we reading a dataset name (or regexp)
        # directly from the command line or are we reading a file
        # containing a list of dataset specifications. Actually we
        # keep one of each for both datasets and runs.
        self.input_method = {}
        self.input_method["datasets"] = {}
        self.input_method["datasets"]["use"] = None
        self.input_method["datasets"]["ignore"] = None
        self.input_method["runs"] = {}
        self.input_method["runs"]["use"] = None
        self.input_method["runs"]["ignore"] = None
        self.input_method["runs"]["ignore"] = None
        # The name of whatever input we're using.
        self.input_name = {}
        self.input_name["datasets"] = {}
        self.input_name["datasets"]["use"] = None
        self.input_name["datasets"]["ignore"] = None
        self.input_name["runs"] = {}
        self.input_name["runs"]["use"] = None
        self.input_name["runs"]["ignore"] = None

        self.Jsonlumi = False
        self.Jsonfilename = "YourJSON.txt"
        self.Jsonrunfilename = "YourJSON.txt"
        self.todofile = "YourToDofile.txt"

        # If this is true, we're running in `force mode'. In this case
        # the sanity checks are performed but failure will not halt
        # everything.
        self.force_running = None

        # The base path of the output dir in CASTOR.
        self.castor_base_dir = None
        self.castor_base_dir_default = "/castor/cern.ch/" \
                                       "cms/store/temp/" \
                                       "dqm/offline/harvesting_output/"

        # The name of the file to be used for book keeping: which
        # datasets, runs, etc. we have already processed.
        self.book_keeping_file_name = None
        self.book_keeping_file_name_default = "harvesting_accounting.txt"

        # The dataset name to reference histogram name mapping is read
        # from a text file. The name of this file is kept in the
        # following variable.
        self.ref_hist_mappings_file_name = None
        # And this is the default value.
        self.ref_hist_mappings_file_name_default = "harvesting_ref_hist_mappings.txt"

        # Hmmm, hard-coded prefix of the CERN CASTOR area. This is the
        # only supported CASTOR area.
        # NOTE: Make sure this one starts with a `/'.
        self.castor_prefix = "/castor/cern.ch"

        # Normally the central harvesting should be done using the
        # `t1access' grid role. To be able to run without T1 access
        # the --no-t1access flag can be used. This variable keeps
        # track of that special mode.
        self.non_t1access = False
        self.caf_access = False
        self.saveByLumiSection = False
        self.crab_submission = False
        self.nr_max_sites = 1

        self.preferred_site = "no preference"

        # This will become the list of datasets and runs to consider
        self.datasets_to_use = {}
        # and this will become the list of datasets and runs to skip.
        self.datasets_to_ignore = {}
        # This, in turn, will hold all book keeping information.
        self.book_keeping_information = {}
        # And this is where the dataset name to reference histogram
        # name mapping is stored.
        self.ref_hist_mappings = {}

        # We're now also allowing run selection. This means we also
        # have to keep list of runs requested and vetoed by the user.
        self.runs_to_use = {}
        self.runs_to_ignore = {}

        # Cache for CMSSW version availability at different sites.
        self.sites_and_versions_cache = {}

        # Cache for checked GlobalTags.
        self.globaltag_check_cache = []

        # Global flag to see if there were any jobs for which we could
        # not find a matching site.
        self.all_sites_found = True

        # Helper string centrally defined.
        self.no_matching_site_found_str = "no_matching_site_found"

        # Store command line options for later use.
        if cmd_line_opts is None:
            cmd_line_opts = sys.argv[1:]
        self.cmd_line_opts = cmd_line_opts

        # Set up the logger.
        log_handler = logging.StreamHandler()
        # This is the default log formatter, the debug option switches
        # on some more information.
        log_formatter = logging.Formatter("%(message)s")
        log_handler.setFormatter(log_formatter)
        logger = logging.getLogger()
        logger.name = "main"
        logger.addHandler(log_handler)
        self.logger = logger
        # The default output mode is quite verbose.
        self.set_output_level("NORMAL")

        #logger.debug("Initialized successfully")

        # End of __init__.