fetchall_from_DQM_v2 Namespace Reference

Functions
def	auth_wget
Variables
tuple	all_files = file_list_re.findall(filedir_html)
string	base_url = 'https://cmsweb.cern.ch/dqm/relval/data/browse/ROOT/'
tuple	file_list_re = re.compile(r"<a href='[-./\w]*'>([-./\w]*)<")
list	file_res = [re.compile(r) for r in options.regexp.split(',') + [options.release]]
	Fetch the files, using multi-processing.
tuple	filedir_html = auth_wget(filedir_url)
string	filedir_url = '/'
string	help = 'Fetch data relvals.'
tuple	parser = OptionParser(usage='usage: %prog [options]')
	Define options.
tuple	pool = Pool(options.mthreads)
tuple	release = re.findall('(CMSSW_\d*_\d*_)\d*(?:_[\w\d]*)?', options.release)
list	releasedir = release[0]
string	relvaldir = "RelVal"
	Parse sys.argv.
list	selected_files = [f for f in all_files if all([r.search(f) for r in file_res])]

---

Function Documentation

def fetchall_from_DQM_v2::auth_wget	(		url,
			chunk_size = 1048576
	)

Returns the content of specified URL, which requires authentication.
If the content is bigger than 1MB, then save it to file.

Definition at line 38 of file fetchall_from_DQM_v2.py.

                                      :
    """Returns the content of specified URL, which requires authentication.
    If the content is bigger than 1MB, then save it to file.
    """
    opener = build_opener(X509CertOpen())
    url_file = opener.open(Request(url))
    size = int(url_file.headers["Content-Length"])

    if size < 1048576:   # if File size < 1MB
        filename = basename(url)    #still download
        readed = url_file.read()    ## and then check if its not an empty dir (parent directory)
        if filename != '':
            outfile = open(filename, 'wb')  #then write File to local system
            outfile.write(readed)
        return readed

    filename = basename(url)
    file_id = selected_files.index(filename)

    if isfile("./%s" % filename):
        print '%d. Exsits on disk. Skipping.' % (file_id +1)
        return

    print '%d. Downloading...' % (file_id +1)
    file = open(filename, 'wb')
    # progress = 0
    chunk = url_file.read(chunk_size)
    while chunk:
        file.write(chunk)
        # progress += chunk_size
        chunk = url_file.read(chunk_size)
    print '%d.  Done.' % (file_id +1)
    file.close()

---

Variable Documentation

tuple fetchall_from_DQM_v2::all_files = file_list_re.findall(filedir_html)

Definition at line 123 of file fetchall_from_DQM_v2.py.

Referenced by ContentReader::give_files().

string fetchall_from_DQM_v2::base_url = 'https://cmsweb.cern.ch/dqm/relval/data/browse/ROOT/'

Definition at line 115 of file fetchall_from_DQM_v2.py.

tuple fetchall_from_DQM_v2::file_list_re = re.compile(r"<a href='[-./\w]*'>([-./\w]*)<")

Definition at line 122 of file fetchall_from_DQM_v2.py.

list fetchall_from_DQM_v2::file_res = [re.compile(r) for r in options.regexp.split(',') + [options.release]]

Fetch the files, using multi-processing.

Definition at line 130 of file fetchall_from_DQM_v2.py.

tuple fetchall_from_DQM_v2::filedir_html = auth_wget(filedir_url)

Definition at line 117 of file fetchall_from_DQM_v2.py.

string fetchall_from_DQM_v2::filedir_url = '/'

Definition at line 116 of file fetchall_from_DQM_v2.py.

string fetchall_from_DQM_v2::help = 'Fetch data relvals.'

Definition at line 76 of file fetchall_from_DQM_v2.py.

tuple fetchall_from_DQM_v2::parser = OptionParser(usage='usage: %prog [options]')

Define options.

Definition at line 74 of file fetchall_from_DQM_v2.py.

tuple fetchall_from_DQM_v2::pool = Pool(options.mthreads)

Definition at line 139 of file fetchall_from_DQM_v2.py.

Referenced by FWDialogBuilder::addLabel(), fireworks::boldGC(), CmsShowCommonPopup::CmsShowCommonPopup(), CentralityTableProducer::endJob(), FastTimerService::fillPathMap(), fireworks::italicGC(), stor::ThroughputMonitorCollection::setMemoryPoolPointer(), and CmsShowMainFrame::showFWorksInfo().

tuple fetchall_from_DQM_v2::release = re.findall('(CMSSW_\d*_\d*_)\d*(?:_[\w\d]*)?', options.release)

Definition at line 110 of file fetchall_from_DQM_v2.py.

Referenced by stor::XHTMLMaker::addNode(), stor::XHTMLMaker::addText(), ConfigurationDBHandler::cvt2String(), L1GtTriggerMenuXmlParser::findXMLChild(), XMLDOMBlock::getNewDocument(), L1GtTriggerMenuXmlParser::getXMLAttribute(), L1GtTriggerMenuXmlParser::getXMLTextValue(), MonitorXMLParser::handleElement(), cond::XMLAuthenticationService::XMLAuthenticationService::initialize(), L1GtTriggerMenuXmlParser::initXML(), MonitorXMLParser::load(), MonitorXMLParser::MonitorXMLParser(), MuonAlignmentInputXML::newAlignableMuon(), stor::XHTMLMaker::out(), XMLDOMBlock::parse(), L1GtTriggerMenuXmlParser::parseConditions(), MuonAlignmentInputXML::parseDouble(), SummaryPlotXmlParser::parseXML(), cond::XMLAuthenticationService::XMLAuthenticationService::processFile(), CalibrationXML::readAttribute(), evf::IPCMethod::releaseResources(), edm::RootFile::RootFile(), XMLProcessor::serializeDOM(), stor::XHTMLMaker::start(), cscdqm::XMLFileErrorHandler::warning(), L1GtTriggerMenuXmlParser::workXML(), CalibrationXML::writeAttribute(), stor::XHTMLMaker::XHTMLMaker(), xMLCh2String(), XMLDOMBlock::XMLDOMBlock(), ConfigurationDBHandler::~ConfigurationDBHandler(), LutXml::~LutXml(), MonitorXMLParser::~MonitorXMLParser(), MuonAlignmentInputXML::~MuonAlignmentInputXML(), edm::EventBuffer::OperateBuffer< T >::~OperateBuffer(), edm::SingleConsumerQ::OperateBuffer< T >::~OperateBuffer(), ora::Serializer::~Serializer(), StrX::~StrX(), TagNames::~TagNames(), cscdqm::XercesStringTranscoder::~XercesStringTranscoder(), lhef::XMLSimpleStr::~XMLSimpleStr(), XMLSimpleStr::~XMLSimpleStr(), XMLUniStr::~XMLUniStr(), lhef::XMLUniStr::~XMLUniStr(), and XStr::~XStr().

list fetchall_from_DQM_v2::releasedir = release[0]

Definition at line 113 of file fetchall_from_DQM_v2.py.

string fetchall_from_DQM_v2::relvaldir = "RelVal"

Parse sys.argv.

Check for option errors Use options

Definition at line 106 of file fetchall_from_DQM_v2.py.

list fetchall_from_DQM_v2::selected_files = [f for f in all_files if all([r.search(f) for r in file_res])]

Definition at line 131 of file fetchall_from_DQM_v2.py.