cmsHarvester.CMSHarvester Class Reference

CMSHarvester class. More...

Inheritance diagram for cmsHarvester.CMSHarvester:

Public Member Functions
def	__init__ (self, cmd_line_opts=None)
def	cleanup (self)
def	config_file_header (self)
def	db_account_name_cms_cond_dqm_summary (self)
def	db_account_name_cms_cond_globaltag (self)
def	format_conditions_string (self, globaltag)
def	ident_string (self)
def	option_handler_book_keeping_file (self, option, opt_str, value, parser)
def	option_handler_debug (self, option, opt_str, value, parser)
def	option_handler_force (self, option, opt_str, value, parser)
def	option_handler_frontier_connection (self, option, opt_str, value, parser)
def	option_handler_globaltag (self, option, opt_str, value, parser)
def	option_handler_harvesting_mode (self, option, opt_str, value, parser)
def	option_handler_harvesting_type (self, option, opt_str, value, parser)
def	option_handler_input_Jsonfile (self, option, opt_str, value, parser)
def	option_handler_input_Jsonrunfile (self, option, opt_str, value, parser)
def	option_handler_input_spec (self, option, opt_str, value, parser)
def	option_handler_input_todofile (self, option, opt_str, value, parser)
def	option_handler_no_ref_hists (self, option, opt_str, value, parser)
def	option_handler_quiet (self, option, opt_str, value, parser)
def	option_handler_ref_hist_mapping_file (self, option, opt_str, value, parser)
def	set_output_level (self, output_level)
def	time_stamp (self)

Public Attributes
	all_sites_found
	book_keeping_file_name
	book_keeping_file_name_default
	book_keeping_information
	caf_access
	castor_base_dir
	castor_base_dir_default
	castor_prefix
	cmd_line_opts
	crab_submission
	datasets_to_ignore
	datasets_to_use
	force_running
	frontier_connection_name
	frontier_connection_overridden
	globaltag
	globaltag_check_cache
	harvesting_info
	harvesting_mode
	harvesting_mode_default
	harvesting_modes
	harvesting_type
	harvesting_types
	input_method
	input_name
	Jsonfilename
	Jsonlumi
	Jsonrunfilename
	log_level
	logger
	no_matching_site_found_str
	non_t1access
	nr_max_sites
	preferred_site
	ref_hist_mappings
	ref_hist_mappings_file_name
	ref_hist_mappings_file_name_default
	runs_to_ignore
	runs_to_use
	saveByLumiSection
	sites_and_versions_cache
	todofile
	use_ref_hists
	version

Detailed Description

CMSHarvester class.

Class to perform CMS harvesting.

More documentation `obviously' to follow.

Definition at line 295 of file cmsHarvester.py.

Constructor & Destructor Documentation

__init__()

def cmsHarvester.CMSHarvester.__init__	(		self,
			cmd_line_opts = None
	)

Definition at line 304 of file cmsHarvester.py.

    def __init__(self, cmd_line_opts=None):
        "Initialize class and process command line options."

        self.version = __version__

        # These are the harvesting types allowed. See the head of this
        # file for more information.
        self.harvesting_types = [
            "RelVal",
            "RelValFS",
            "MC",
            "DQMOffline",
            ]

        # These are the possible harvesting modes:
        #   - Single-step: harvesting takes place on-site in a single
        #   step. For each samples only a single ROOT file containing
        #   the harvesting results is returned.
        #   - Single-step-allow-partial: special hack to allow
        #   harvesting of partial statistics using single-step
        #   harvesting on spread-out samples.
        #   - Two-step: harvesting takes place in two steps. The first
        #   step returns a series of monitoring elenent summaries for
        #   each sample. The second step then merges these summaries
        #   locally and does the real harvesting. This second step
        #   produces the ROOT file containing the harvesting results.
        self.harvesting_modes = [
            "single-step",
            "single-step-allow-partial",
            "two-step"
            ]

        # It is possible to specify a GlobalTag that will override any
        # choices (regarding GlobalTags) made by the cmsHarvester.
        # BUG BUG BUG
        # For the moment, until I figure out a way to obtain the
        # GlobalTag with which a given data (!) dataset was created,
        # it is necessary to specify a GlobalTag for harvesting of
        # data.
        # BUG BUG BUG end
        self.globaltag = None

        # It's also possible to switch off the use of reference
        # histograms altogether.
        self.use_ref_hists = True

        # The database name and account are hard-coded. They are not
        # likely to change before the end-of-life of this tool. But of
        # course there is a way to override this from the command
        # line. One can even override the Frontier connection used for
        # the GlobalTag and for the reference histograms
        # independently. Please only use this for testing purposes.
        self.frontier_connection_name = {}
        self.frontier_connection_name["globaltag"] = "frontier://" \
                                                     "FrontierProd/"
        self.frontier_connection_name["refhists"] = "frontier://" \
                                                    "FrontierProd/"
        self.frontier_connection_overridden = {}
        for key in self.frontier_connection_name.keys():
            self.frontier_connection_overridden[key] = False

        # This contains information specific to each of the harvesting
        # types. Used to create the harvesting configuration. It is
        # filled by setup_harvesting_info().
        self.harvesting_info = None

        

        # These are default `unused' values that will be filled in
        # depending on the command line options.

        # The type of harvesting we're doing. See
        # self.harvesting_types for allowed types.
        self.harvesting_type = None

        # The harvesting mode, popularly known as single-step
        # vs. two-step. The thing to remember at this point is that
        # single-step is only possible for samples located completely
        # at a single site (i.e. SE).
        self.harvesting_mode = None
        # BUG BUG BUG
        # Default temporarily set to two-step until we can get staged
        # jobs working with CRAB.
        self.harvesting_mode_default = "single-step"
        # BUG BUG BUG end

        # The input method: are we reading a dataset name (or regexp)
        # directly from the command line or are we reading a file
        # containing a list of dataset specifications. Actually we
        # keep one of each for both datasets and runs.
        self.input_method = {}
        self.input_method["datasets"] = {}
        self.input_method["datasets"]["use"] = None
        self.input_method["datasets"]["ignore"] = None
        self.input_method["runs"] = {}
        self.input_method["runs"]["use"] = None
        self.input_method["runs"]["ignore"] = None
        self.input_method["runs"]["ignore"] = None
        # The name of whatever input we're using.
        self.input_name = {}
        self.input_name["datasets"] = {}
        self.input_name["datasets"]["use"] = None
        self.input_name["datasets"]["ignore"] = None
        self.input_name["runs"] = {}
        self.input_name["runs"]["use"] = None
        self.input_name["runs"]["ignore"] = None

        self.Jsonlumi = False
        self.Jsonfilename = "YourJSON.txt"
        self.Jsonrunfilename = "YourJSON.txt"
        self.todofile = "YourToDofile.txt"

        # If this is true, we're running in `force mode'. In this case
        # the sanity checks are performed but failure will not halt
        # everything.
        self.force_running = None

        # The base path of the output dir in CASTOR.
        self.castor_base_dir = None
        self.castor_base_dir_default = "/castor/cern.ch/" \
                                       "cms/store/temp/" \
                                       "dqm/offline/harvesting_output/"

        # The name of the file to be used for book keeping: which
        # datasets, runs, etc. we have already processed.
        self.book_keeping_file_name = None
        self.book_keeping_file_name_default = "harvesting_accounting.txt"

        # The dataset name to reference histogram name mapping is read
        # from a text file. The name of this file is kept in the
        # following variable.
        self.ref_hist_mappings_file_name = None
        # And this is the default value.
        self.ref_hist_mappings_file_name_default = "harvesting_ref_hist_mappings.txt"

        # Hmmm, hard-coded prefix of the CERN CASTOR area. This is the
        # only supported CASTOR area.
        # NOTE: Make sure this one starts with a `/'.
        self.castor_prefix = "/castor/cern.ch"

        # Normally the central harvesting should be done using the
        # `t1access' grid role. To be able to run without T1 access
        # the --no-t1access flag can be used. This variable keeps
        # track of that special mode.
        self.non_t1access = False
        self.caf_access = False
        self.saveByLumiSection = False
        self.crab_submission = False
        self.nr_max_sites = 1

        self.preferred_site = "no preference"

        # This will become the list of datasets and runs to consider
        self.datasets_to_use = {}
        # and this will become the list of datasets and runs to skip.
        self.datasets_to_ignore = {}
        # This, in turn, will hold all book keeping information.
        self.book_keeping_information = {}
        # And this is where the dataset name to reference histogram
        # name mapping is stored.
        self.ref_hist_mappings = {}

        # We're now also allowing run selection. This means we also
        # have to keep list of runs requested and vetoed by the user.
        self.runs_to_use = {}
        self.runs_to_ignore = {}

        # Cache for CMSSW version availability at different sites.
        self.sites_and_versions_cache = {}

        # Cache for checked GlobalTags.
        self.globaltag_check_cache = []

        # Global flag to see if there were any jobs for which we could
        # not find a matching site.
        self.all_sites_found = True

        # Helper string centrally defined.
        self.no_matching_site_found_str = "no_matching_site_found"

        # Store command line options for later use.
        if cmd_line_opts is None:
            cmd_line_opts = sys.argv[1:]
        self.cmd_line_opts = cmd_line_opts

        # Set up the logger.
        log_handler = logging.StreamHandler()
        # This is the default log formatter, the debug option switches
        # on some more information.
        log_formatter = logging.Formatter("%(message)s")
        log_handler.setFormatter(log_formatter)
        logger = logging.getLogger()
        logger.name = "main"
        logger.addHandler(log_handler)
        self.logger = logger
        # The default output mode is quite verbose.
        self.set_output_level("NORMAL")

        #logger.debug("Initialized successfully")

        # End of __init__.

Member Function Documentation

cleanup()

def cmsHarvester.CMSHarvester.cleanup

(

self

)

Definition at line 508 of file cmsHarvester.py.

Referenced by esMonitoring.FDJsonServer.handle_close().

    def cleanup(self):
        "Clean up after ourselves."

        # NOTE: This is the safe replacement of __del__.

        #self.logger.debug("All done -> shutting down")
        logging.shutdown()

        # End of cleanup.

config_file_header()

def cmsHarvester.CMSHarvester.config_file_header

(

self

)

Definition at line 602 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.ident_string(), join(), and cmsHarvester.CMSHarvester.time_stamp().

    def config_file_header(self):
        "Create a nice header to be used to mark the generated files."

        tmp = []

        time_stamp = self.time_stamp()
        ident_str = self.ident_string()
        tmp.append("# %s" % time_stamp)
        tmp.append("# WARNING: This file was created automatically!")
        tmp.append("")
        tmp.append("# Created by %s" % ident_str)

        header = "\n".join(tmp)

        # End of config_file_header.
        return header

db_account_name_cms_cond_dqm_summary()

def cmsHarvester.CMSHarvester.db_account_name_cms_cond_dqm_summary

(

self

)

See db_account_name_cms_cond_globaltag.

Definition at line 587 of file cmsHarvester.py.

    def db_account_name_cms_cond_dqm_summary(self):
        """See db_account_name_cms_cond_globaltag."""

        account_name = None
        version = self.cmssw_version[6:11]
        if version < "3_4_0":
            account_name = "CMS_COND_31X_DQM_SUMMARY"
        else:
            account_name = "CMS_COND_34X"

        # End of db_account_name_cms_cond_dqm_summary.
        return account_name

db_account_name_cms_cond_globaltag()

def cmsHarvester.CMSHarvester.db_account_name_cms_cond_globaltag

(

self

)

Return the database account name used to store the GlobalTag.

The name of the database account depends (albeit weakly) on
the CMSSW release version.

Definition at line 570 of file cmsHarvester.py.

    def db_account_name_cms_cond_globaltag(self):
        """Return the database account name used to store the GlobalTag.

        The name of the database account depends (albeit weakly) on
        the CMSSW release version.

        """

        # This never changed, unlike the cms_cond_31X_DQM_SUMMARY ->
        # cms_cond_34X_DQM transition.
        account_name = "CMS_COND_31X_GLOBALTAG"

        # End of db_account_name_cms_cond_globaltag.
        return account_name

format_conditions_string()

def cmsHarvester.CMSHarvester.format_conditions_string	(		self,
			globaltag
	)

Create the conditions string needed for `cmsDriver'.

Just glueing the FrontierConditions bit in front of it really.

Definition at line 545 of file cmsHarvester.py.

References spr.find().

    def format_conditions_string(self, globaltag):
        """Create the conditions string needed for `cmsDriver'.

        Just glueing the FrontierConditions bit in front of it really.

        """

        # Not very robust but okay. The idea is that if the user
        # specified (since this cannot happen with GlobalTags coming
        # from DBS) something containing `conditions', they probably
        # know what they're doing and we should not muck things up. In
        # all other cases we just assume we only received the
        # GlobalTag part and we built the usual conditions string from
        # that .
        if globaltag.lower().find("conditions") > -1:
            conditions_string = globaltag
        else:
            conditions_string = "FrontierConditions_GlobalTag,%s" % \
                                globaltag

        # End of format_conditions_string.
        return conditions_string

ident_string()

def cmsHarvester.CMSHarvester.ident_string

(

self

)

Definition at line 533 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.config_file_header().

    def ident_string(self):
        "Spit out an identification string for cmsHarvester.py."

        ident_str = "`cmsHarvester.py " \
                    "version %s': cmsHarvester.py %s" % \
                    (__version__,
                     reduce(lambda x, y: x+' '+y, sys.argv[1:]))

        return ident_str

option_handler_book_keeping_file()

def cmsHarvester.CMSHarvester.option_handler_book_keeping_file	(		self,
			option,
			opt_str,
			value,
			parser
	)

Store the name of the file to be used for book keeping.

The only check done here is that only a single book keeping
file is specified.

Definition at line 962 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.book_keeping_file_name, info(), crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, and cmsHarvester.CMSHarvester.logger.

    def option_handler_book_keeping_file(self, option, opt_str, value, parser):
        """Store the name of the file to be used for book keeping.

        The only check done here is that only a single book keeping
        file is specified.

        """

        file_name = value

        if not self.book_keeping_file_name is None:
            msg = "Only one book keeping file should be specified"
            self.logger.fatal(msg)
            raise Usage(msg)
        self.book_keeping_file_name = file_name

        self.logger.info("Book keeping file to be used: `%s'" % \
                         self.book_keeping_file_name)

        # End of option_handler_book_keeping_file.

option_handler_debug()

def cmsHarvester.CMSHarvester.option_handler_debug	(		self,
			option,
			opt_str,
			value,
			parser
	)

Switch to debug mode.

This both increases the amount of output generated, as well as
changes the format used (more detailed information is given).

Definition at line 660 of file cmsHarvester.py.

References crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, cmsHarvester.CMSHarvester.logger, and cmsHarvester.CMSHarvester.set_output_level().

    def option_handler_debug(self, option, opt_str, value, parser):
        """Switch to debug mode.

        This both increases the amount of output generated, as well as
        changes the format used (more detailed information is given).

        """

        # Switch to a more informative log formatter for debugging.
        log_formatter_debug = logging.Formatter("[%(levelname)s] " \
                                                # NOTE: funcName was
                                                # only implemented
                                                # starting with python
                                                # 2.5.
                                                #"%(funcName)s() " \
                                                #"@%(filename)s:%(lineno)d " \
                                                "%(message)s")
        # Hmmm, not very nice. This assumes there's only a single
        # handler associated with the current logger.
        log_handler = self.logger.handlers[0]
        log_handler.setFormatter(log_formatter_debug)
        self.set_output_level("DEBUG")

        # End of option_handler_debug.

option_handler_force()

def cmsHarvester.CMSHarvester.option_handler_force	(		self,
			option,
			opt_str,
			value,
			parser
	)

Switch on `force mode' in which case we don't brake for nobody.

In so-called `force mode' all sanity checks are performed but
we don't halt on failure. Of course this requires some care
from the user.

Definition at line 696 of file cmsHarvester.py.

References debug, cmsHarvester.CMSHarvester.force_running, crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, and cmsHarvester.CMSHarvester.logger.

    def option_handler_force(self, option, opt_str, value, parser):
        """Switch on `force mode' in which case we don't brake for nobody.

        In so-called `force mode' all sanity checks are performed but
        we don't halt on failure. Of course this requires some care
        from the user.

        """

        self.logger.debug("Switching on `force mode'.")
        self.force_running = True

        # End of option_handler_force.

option_handler_frontier_connection()

def cmsHarvester.CMSHarvester.option_handler_frontier_connection	(		self,
			option,
			opt_str,
			value,
			parser
	)

Override the default Frontier connection string.

Please only use this for testing (e.g. when a test payload has
been inserted into cms_orc_off instead of cms_orc_on).

This method gets called for three different command line
options:
- --frontier-connection,
- --frontier-connection-for-globaltag,
- --frontier-connection-for-refhists.
Appropriate care has to be taken to make sure things are only
specified once.

Definition at line 824 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.frontier_connection_name, cmsHarvester.CMSHarvester.frontier_connection_overridden, relativeConstraints.keys, crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, cmsHarvester.CMSHarvester.logger, and MessageLogger_cfi.warning.

Referenced by cmsHarvester.CMSHarvester.option_handler_no_ref_hists().

                                           value, parser):
        """Override the default Frontier connection string.

        Please only use this for testing (e.g. when a test payload has
        been inserted into cms_orc_off instead of cms_orc_on).

        This method gets called for three different command line
        options:
        - --frontier-connection,
        - --frontier-connection-for-globaltag,
        - --frontier-connection-for-refhists.
        Appropriate care has to be taken to make sure things are only
        specified once.

        """

        # Figure out with which command line option we've been called.
        frontier_type = opt_str.split("-")[-1]
        if frontier_type == "connection":
            # Main option: change all connection strings.
            frontier_types = self.frontier_connection_name.keys()
        else:
            frontier_types = [frontier_type]

        # Make sure that each Frontier connection is specified only
        # once. (Okay, in a bit of a dodgy way...)
        for connection_name in frontier_types:
            if self.frontier_connection_overridden[connection_name] == True:
                msg = "Please specify either:\n" \
                      "  `--frontier-connection' to change the " \
                      "Frontier connection used for everything, or\n" \
                      "either one or both of\n" \
                      "  `--frontier-connection-for-globaltag' to " \
                      "change the Frontier connection used for the " \
                      "GlobalTag and\n" \
                      "  `--frontier-connection-for-refhists' to change " \
                      "the Frontier connection used for the " \
                      "reference histograms."
                self.logger.fatal(msg)
                raise Usage(msg)

        frontier_prefix = "frontier://"
        if not value.startswith(frontier_prefix):
            msg = "Expecting Frontier connections to start with " \
                  "`%s'. You specified `%s'." % \
                  (frontier_prefix, value)
            self.logger.fatal(msg)
            raise Usage(msg)
        # We also kind of expect this to be either FrontierPrep or
        # FrontierProd (but this is just a warning).
        if value.find("FrontierProd") < 0 and \
               value.find("FrontierProd") < 0:
            msg = "Expecting Frontier connections to contain either " \
                  "`FrontierProd' or `FrontierPrep'. You specified " \
                  "`%s'. Are you sure?" % \
                  value
            self.logger.warning(msg)

        if not value.endswith("/"):
            value += "/"

        for connection_name in frontier_types:
            self.frontier_connection_name[connection_name] = value
            self.frontier_connection_overridden[connection_name] = True

            frontier_type_str = "unknown"
            if connection_name == "globaltag":
                frontier_type_str = "the GlobalTag"
            elif connection_name == "refhists":
                frontier_type_str = "the reference histograms"

            self.logger.warning("Overriding default Frontier " \
                                "connection for %s " \
                                "with `%s'" % \
                                (frontier_type_str,
                                 self.frontier_connection_name[connection_name]))

        # End of option_handler_frontier_connection

option_handler_globaltag()

def cmsHarvester.CMSHarvester.option_handler_globaltag	(		self,
			option,
			opt_str,
			value,
			parser
	)

Set the GlobalTag to be used, overriding our own choices.

By default the cmsHarvester will use the GlobalTag with which
a given dataset was created also for the harvesting. The
--globaltag option is the way to override this behaviour.

Definition at line 789 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.globaltag, info(), crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, and cmsHarvester.CMSHarvester.logger.

    def option_handler_globaltag(self, option, opt_str, value, parser):
        """Set the GlobalTag to be used, overriding our own choices.

        By default the cmsHarvester will use the GlobalTag with which
        a given dataset was created also for the harvesting. The
        --globaltag option is the way to override this behaviour.

        """

        # Make sure that this flag only occurred once.
        if not self.globaltag is None:
            msg = "Only one GlobalTag should be specified"
            self.logger.fatal(msg)
            raise Usage(msg)
        self.globaltag = value

        self.logger.info("GlobalTag to be used: `%s'" % \
                         self.globaltag)

        # End of option_handler_globaltag.

option_handler_harvesting_mode()

def cmsHarvester.CMSHarvester.option_handler_harvesting_mode	(		self,
			option,
			opt_str,
			value,
			parser
	)

Set the harvesting mode to be used.

Single-step harvesting can be used for samples that are
located completely at a single site (= SE). Otherwise use
two-step mode.

Definition at line 756 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.harvesting_mode, cmsHarvester.CMSHarvester.harvesting_modes, info(), join(), crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, and cmsHarvester.CMSHarvester.logger.

    def option_handler_harvesting_mode(self, option, opt_str, value, parser):
        """Set the harvesting mode to be used.

        Single-step harvesting can be used for samples that are
        located completely at a single site (= SE). Otherwise use
        two-step mode.

        """

        # Check for valid mode.
        harvesting_mode = value.lower()
        if not harvesting_mode in self.harvesting_modes:
            msg = "Unknown harvesting mode `%s'" % harvesting_mode
            self.logger.fatal(msg)
            self.logger.fatal("  possible modes are: %s" % \
                              ", ".join(self.harvesting_modes))
            raise Usage(msg)

        # Check if we've been given only a single mode, otherwise
        # complain.
        if not self.harvesting_mode is None:
            msg = "Only one harvesting mode should be specified"
            self.logger.fatal(msg)
            raise Usage(msg)
        self.harvesting_mode = harvesting_mode

        self.logger.info("Harvesting mode to be used: `%s'" % \
                         self.harvesting_mode)

        # End of option_handler_harvesting_mode.

option_handler_harvesting_type()

def cmsHarvester.CMSHarvester.option_handler_harvesting_type	(		self,
			option,
			opt_str,
			value,
			parser
	)

Set the harvesting type to be used.

This checks that no harvesting type is already set, and sets
the harvesting type to be used to the one specified. If a
harvesting type is already set an exception is thrown. The
same happens when an unknown type is specified.

Definition at line 712 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.harvesting_type, cmsHarvester.CMSHarvester.harvesting_types, info(), join(), crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, and cmsHarvester.CMSHarvester.logger.

    def option_handler_harvesting_type(self, option, opt_str, value, parser):
        """Set the harvesting type to be used.

        This checks that no harvesting type is already set, and sets
        the harvesting type to be used to the one specified. If a
        harvesting type is already set an exception is thrown. The
        same happens when an unknown type is specified.

        """

        # Check for (in)valid harvesting types.
        # NOTE: The matching is done in a bit of a complicated
        # way. This allows the specification of the type to be
        # case-insensitive while still ending up with the properly
        # `cased' version afterwards.
        value = value.lower()
        harvesting_types_lowered = [i.lower() for i in self.harvesting_types]
        try:
            type_index = harvesting_types_lowered.index(value)
            # If this works, we now have the index to the `properly
            # cased' version of the harvesting type.
        except ValueError:
            self.logger.fatal("Unknown harvesting type `%s'" % \
                              value)
            self.logger.fatal("  possible types are: %s" %
                              ", ".join(self.harvesting_types))
            raise Usage("Unknown harvesting type `%s'" % \
                        value)

        # Check if multiple (by definition conflicting) harvesting
        # types are being specified.
        if not self.harvesting_type is None:
            msg = "Only one harvesting type should be specified"
            self.logger.fatal(msg)
            raise Usage(msg)
        self.harvesting_type = self.harvesting_types[type_index]

        self.logger.info("Harvesting type to be used: `%s'" % \
                         self.harvesting_type)

        # End of option_handler_harvesting_type.

option_handler_input_Jsonfile()

def cmsHarvester.CMSHarvester.option_handler_input_Jsonfile	(		self,
			option,
			opt_str,
			value,
			parser
	)

Definition at line 912 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.Jsonfilename.

    def option_handler_input_Jsonfile(self, option, opt_str, value, parser):

        self.Jsonfilename = value
        # End of option_handler_input_Jsonfile.

option_handler_input_Jsonrunfile()

def cmsHarvester.CMSHarvester.option_handler_input_Jsonrunfile	(		self,
			option,
			opt_str,
			value,
			parser
	)

Definition at line 919 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.Jsonrunfilename.

    def option_handler_input_Jsonrunfile(self, option, opt_str, value, parser):

        self.Jsonrunfilename = value
        # End of option_handler_input_Jsonrunfile.

option_handler_input_spec()

def cmsHarvester.CMSHarvester.option_handler_input_spec	(		self,
			option,
			opt_str,
			value,
			parser
	)

TODO TODO TODO
Document this...

Definition at line 926 of file cmsHarvester.py.

References debug, spr.find(), cmsHarvester.CMSHarvester.input_method, cmsHarvester.CMSHarvester.input_name, crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, cmsHarvester.CMSHarvester.logger, and python.rootplot.root2matplotlib.replace().

    def option_handler_input_spec(self, option, opt_str, value, parser):
        """TODO TODO TODO
        Document this...

        """

        # Figure out if we were called for the `use these' or the
        # `ignore these' case.
        if opt_str.lower().find("ignore") > -1:
            spec_type = "ignore"
        else:
            spec_type = "use"

        # Similar: are we being called for datasets or for runs?
        if opt_str.lower().find("dataset") > -1:
            select_type = "datasets"
        else:
            select_type = "runs"

        if not self.input_method[select_type][spec_type] is None:
            msg = "Please only specify one input method " \
                  "(for the `%s' case)" % opt_str
            self.logger.fatal(msg)
            raise Usage(msg)

        input_method = opt_str.replace("-", "").replace("ignore", "")
        self.input_method[select_type][spec_type] = input_method
        self.input_name[select_type][spec_type] = value

        self.logger.debug("Input method for the `%s' case: %s" % \
                          (spec_type, input_method))

        # End of option_handler_input_spec

option_handler_input_todofile()

def cmsHarvester.CMSHarvester.option_handler_input_todofile	(		self,
			option,
			opt_str,
			value,
			parser
	)

Definition at line 905 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.todofile.

    def option_handler_input_todofile(self, option, opt_str, value, parser):

        self.todofile = value
        # End of option_handler_input_todofile.

option_handler_no_ref_hists()

def cmsHarvester.CMSHarvester.option_handler_no_ref_hists	(		self,
			option,
			opt_str,
			value,
			parser
	)

Definition at line 812 of file cmsHarvester.py.

References crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, cmsHarvester.CMSHarvester.logger, cmsHarvester.CMSHarvester.option_handler_frontier_connection(), cmsHarvester.CMSHarvester.use_ref_hists, and MessageLogger_cfi.warning.

    def option_handler_no_ref_hists(self, option, opt_str, value, parser):
        "Switch use of all reference histograms off."

        self.use_ref_hists = False

        self.logger.warning("Switching off all use of reference histograms")

        # End of option_handler_no_ref_hists.

     

option_handler_quiet()

def cmsHarvester.CMSHarvester.option_handler_quiet	(		self,
			option,
			opt_str,
			value,
			parser
	)

Definition at line 687 of file cmsHarvester.py.

References cmsHarvester.CMSHarvester.set_output_level().

    def option_handler_quiet(self, option, opt_str, value, parser):
        "Switch to quiet mode: less verbose."

        self.set_output_level("QUIET")

        # End of option_handler_quiet.

option_handler_ref_hist_mapping_file()

def cmsHarvester.CMSHarvester.option_handler_ref_hist_mapping_file	(		self,
			option,
			opt_str,
			value,
			parser
	)

Store the name of the file for the ref. histogram mapping.

Definition at line 985 of file cmsHarvester.py.

References info(), crabFunctions.CrabController.logger, o2olib.O2OJobMgr.logger, conddb_version_mgr.conddb_tool.logger, o2olib.O2ORunMgr.logger, cmsHarvester.CMSHarvester.logger, and cmsHarvester.CMSHarvester.ref_hist_mappings_file_name.

    def option_handler_ref_hist_mapping_file(self, option, opt_str, value, parser):
        """Store the name of the file for the ref. histogram mapping.

        """

        file_name = value

        if not self.ref_hist_mappings_file_name is None:
            msg = "Only one reference histogram mapping file " \
                  "should be specified"
            self.logger.fatal(msg)
            raise Usage(msg)
        self.ref_hist_mappings_file_name = file_name

        self.logger.info("Reference histogram mapping file " \
                         "to be used: `%s'" % \
                         self.ref_hist_mappings_file_name)

        # End of option_handler_ref_hist_mapping_file.

set_output_level()

def cmsHarvester.CMSHarvester.set_output_level	(		self,
			output_level
	)

Adjust the level of output generated.

Choices are:
  - normal  : default level of output
  - quiet   : less output than the default
  - verbose : some additional information
  - debug   : lots more information, may be overwhelming

NOTE: The debug option is a bit special in the sense that it
      also modifies the output format.

Definition at line 621 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_debug(), and cmsHarvester.CMSHarvester.option_handler_quiet().

    def set_output_level(self, output_level):
        """Adjust the level of output generated.

        Choices are:
          - normal  : default level of output
          - quiet   : less output than the default
          - verbose : some additional information
          - debug   : lots more information, may be overwhelming

        NOTE: The debug option is a bit special in the sense that it
              also modifies the output format.

        """

        # NOTE: These levels are hooked up to the ones used in the
        #       logging module.
        output_levels = {
            "NORMAL"  : logging.INFO,
            "QUIET"   : logging.WARNING,
            "VERBOSE" : logging.INFO,
            "DEBUG"   : logging.DEBUG
            }

        output_level = output_level.upper()

        try:
            # Update the logger.
            self.log_level = output_levels[output_level]
            self.logger.setLevel(self.log_level)
        except KeyError:
            # Show a complaint
            self.logger.fatal("Unknown output level `%s'" % ouput_level)
            # and re-raise an exception.
            raise Exception

        # End of set_output_level.

time_stamp()

def cmsHarvester.CMSHarvester.time_stamp

(

self

)

Definition at line 520 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.config_file_header().

    def time_stamp(self):
        "Create a timestamp to use in the created config files."

        time_now = datetime.datetime.utcnow()
        # We don't care about the microseconds.
        time_now = time_now.replace(microsecond = 0)
        time_stamp = "%sUTC" % datetime.datetime.isoformat(time_now)

        # End of time_stamp.
        return time_stamp

Member Data Documentation

all_sites_found

cmsHarvester.CMSHarvester.all_sites_found

Definition at line 479 of file cmsHarvester.py.

book_keeping_file_name

cmsHarvester.CMSHarvester.book_keeping_file_name

Definition at line 429 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_book_keeping_file().

book_keeping_file_name_default

cmsHarvester.CMSHarvester.book_keeping_file_name_default

Definition at line 430 of file cmsHarvester.py.

book_keeping_information

cmsHarvester.CMSHarvester.book_keeping_information

Definition at line 461 of file cmsHarvester.py.

caf_access

cmsHarvester.CMSHarvester.caf_access

Definition at line 449 of file cmsHarvester.py.

castor_base_dir

cmsHarvester.CMSHarvester.castor_base_dir

Definition at line 422 of file cmsHarvester.py.

castor_base_dir_default

cmsHarvester.CMSHarvester.castor_base_dir_default

Definition at line 423 of file cmsHarvester.py.

castor_prefix

cmsHarvester.CMSHarvester.castor_prefix

Definition at line 442 of file cmsHarvester.py.

cmd_line_opts

cmsHarvester.CMSHarvester.cmd_line_opts

Definition at line 487 of file cmsHarvester.py.

crab_submission

cmsHarvester.CMSHarvester.crab_submission

Definition at line 451 of file cmsHarvester.py.

datasets_to_ignore

cmsHarvester.CMSHarvester.datasets_to_ignore

Definition at line 459 of file cmsHarvester.py.

datasets_to_use

cmsHarvester.CMSHarvester.datasets_to_use

Definition at line 457 of file cmsHarvester.py.

force_running

cmsHarvester.CMSHarvester.force_running

Definition at line 419 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_force().

frontier_connection_name

cmsHarvester.CMSHarvester.frontier_connection_name

Definition at line 356 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_frontier_connection().

frontier_connection_overridden

cmsHarvester.CMSHarvester.frontier_connection_overridden

Definition at line 361 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_frontier_connection().

globaltag

cmsHarvester.CMSHarvester.globaltag

Definition at line 344 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_globaltag().

globaltag_check_cache

cmsHarvester.CMSHarvester.globaltag_check_cache

Definition at line 475 of file cmsHarvester.py.

harvesting_info

cmsHarvester.CMSHarvester.harvesting_info

Definition at line 368 of file cmsHarvester.py.

harvesting_mode

cmsHarvester.CMSHarvester.harvesting_mode

Definition at line 383 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_harvesting_mode().

harvesting_mode_default

cmsHarvester.CMSHarvester.harvesting_mode_default

Definition at line 387 of file cmsHarvester.py.

harvesting_modes

cmsHarvester.CMSHarvester.harvesting_modes

Definition at line 330 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_harvesting_mode().

harvesting_type

cmsHarvester.CMSHarvester.harvesting_type

Definition at line 377 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_harvesting_type().

harvesting_types

cmsHarvester.CMSHarvester.harvesting_types

Definition at line 311 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_harvesting_type().

input_method

cmsHarvester.CMSHarvester.input_method

Definition at line 394 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_input_spec().

input_name

cmsHarvester.CMSHarvester.input_name

Definition at line 403 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_input_spec().

Jsonfilename

cmsHarvester.CMSHarvester.Jsonfilename

Definition at line 412 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_input_Jsonfile().

Jsonlumi

cmsHarvester.CMSHarvester.Jsonlumi

Definition at line 411 of file cmsHarvester.py.

Jsonrunfilename

cmsHarvester.CMSHarvester.Jsonrunfilename

Definition at line 413 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_input_Jsonrunfile().

log_level

cmsHarvester.CMSHarvester.log_level

Definition at line 648 of file cmsHarvester.py.

logger

cmsHarvester.CMSHarvester.logger

Definition at line 498 of file cmsHarvester.py.

Referenced by looper.Looper.loop(), cmsHarvester.CMSHarvester.option_handler_book_keeping_file(), cmsHarvester.CMSHarvester.option_handler_debug(), cmsHarvester.CMSHarvester.option_handler_force(), cmsHarvester.CMSHarvester.option_handler_frontier_connection(), cmsHarvester.CMSHarvester.option_handler_globaltag(), cmsHarvester.CMSHarvester.option_handler_harvesting_mode(), cmsHarvester.CMSHarvester.option_handler_harvesting_type(), cmsHarvester.CMSHarvester.option_handler_input_spec(), cmsHarvester.CMSHarvester.option_handler_no_ref_hists(), and cmsHarvester.CMSHarvester.option_handler_ref_hist_mapping_file().

no_matching_site_found_str

cmsHarvester.CMSHarvester.no_matching_site_found_str

Definition at line 482 of file cmsHarvester.py.

non_t1access

cmsHarvester.CMSHarvester.non_t1access

Definition at line 448 of file cmsHarvester.py.

nr_max_sites

cmsHarvester.CMSHarvester.nr_max_sites

Definition at line 452 of file cmsHarvester.py.

preferred_site

cmsHarvester.CMSHarvester.preferred_site

Definition at line 454 of file cmsHarvester.py.

ref_hist_mappings

cmsHarvester.CMSHarvester.ref_hist_mappings

Definition at line 464 of file cmsHarvester.py.

ref_hist_mappings_file_name

cmsHarvester.CMSHarvester.ref_hist_mappings_file_name

Definition at line 435 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_ref_hist_mapping_file().

ref_hist_mappings_file_name_default

cmsHarvester.CMSHarvester.ref_hist_mappings_file_name_default

Definition at line 437 of file cmsHarvester.py.

runs_to_ignore

cmsHarvester.CMSHarvester.runs_to_ignore

Definition at line 469 of file cmsHarvester.py.

runs_to_use

cmsHarvester.CMSHarvester.runs_to_use

Definition at line 468 of file cmsHarvester.py.

saveByLumiSection

cmsHarvester.CMSHarvester.saveByLumiSection

Definition at line 450 of file cmsHarvester.py.

sites_and_versions_cache

cmsHarvester.CMSHarvester.sites_and_versions_cache

Definition at line 472 of file cmsHarvester.py.

todofile

cmsHarvester.CMSHarvester.todofile

Definition at line 414 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_input_todofile().

use_ref_hists

cmsHarvester.CMSHarvester.use_ref_hists

Definition at line 348 of file cmsHarvester.py.

Referenced by cmsHarvester.CMSHarvester.option_handler_no_ref_hists().

version

cmsHarvester.CMSHarvester.version

Definition at line 307 of file cmsHarvester.py.

Referenced by validation.Sample.datasetpattern(), and validation.Sample.filename().