fetchall_from_DQM_v2 Namespace Reference

Functions
def	auth_wget (url, chunk_size=1048576)

Variables
	action
	all_files
	args
	base_url
	default
	dest
	False
	file_list_re
	file_res
	Fetch the files, using multi-processing. More...
	filedir_html
	filedir_url
	help
	mthreads
	options
	Parse sys.argv. More...
	parser
	Define options. More...
	pool
	regexp
	release
	releasedir
	relvaldir
	Check for option errors. More...
	selected_files
	usage

Detailed Description

Script fetches files matching specified RegExps from DQM GUI.

Author:  Albertas Gimbutas,  Vilnius University (LT)
e-mail:  albertasgim@gmail.com

Function Documentation

def fetchall_from_DQM_v2.auth_wget	(		url,
			chunk_size = 1048576
	)

Returns the content of specified URL, which requires authentication.
If the content is bigger than 1MB, then save it to file.

Definition at line 38 of file fetchall_from_DQM_v2.py.

References estimatePileup.basename, and createfilelist.int.

def auth_wget(url, chunk_size=1048576):
    """Returns the content of specified URL, which requires authentication.
    If the content is bigger than 1MB, then save it to file.
    """
    opener = build_opener(X509CertOpen())
    url_file = opener.open(Request(url))
    size = int(url_file.headers["Content-Length"])

    if size < 1048576:   # if File size < 1MB
        filename = basename(url)    #still download
        readed = url_file.read()    ## and then check if its not an empty dir (parent directory)
        if filename != '':
            outfile = open(filename, 'wb')  #then write File to local system
            outfile.write(readed)
        return readed

    filename = basename(url)
    file_id = selected_files.index(filename)

    if isfile("./%s" % filename):
        print '%d. Exsits on disk. Skipping.' % (file_id +1)
        return

    print '%d. Downloading...' % (file_id +1)
    file = open(filename, 'wb')
    # progress = 0
    chunk = url_file.read(chunk_size)
    while chunk:
        file.write(chunk)
        # progress += chunk_size
        chunk = url_file.read(chunk_size)
    print '%d.  Done.' % (file_id +1)
    file.close()

Variable Documentation

fetchall_from_DQM_v2.action

Definition at line 75 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.all_files

Definition at line 123 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.args

Definition at line 90 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.base_url

Definition at line 115 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.default

Definition at line 81 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.dest

Definition at line 75 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.False

Definition at line 87 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.file_list_re

Definition at line 122 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.file_res

Fetch the files, using multi-processing.

Definition at line 130 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.filedir_html

Definition at line 117 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.filedir_url

Definition at line 116 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.help

Definition at line 76 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.mthreads

Definition at line 125 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.options

Parse sys.argv.

Definition at line 90 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.parser

Define options.

Definition at line 74 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.pool

Definition at line 139 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.regexp

Definition at line 92 of file fetchall_from_DQM_v2.py.

Referenced by L1TdeStage2uGT.analyze(), DQMGenericTnPClient.findAllSubdirectories(), HLTMuonRefMethod.findAllSubdirectories(), DQMGenericClient.findAllSubdirectories(), edm.glob2reg(), HLTConfigProvider.matched(), edm.regexMatch(), and HLTConfigProvider.removeVersion().

fetchall_from_DQM_v2.release

Definition at line 91 of file fetchall_from_DQM_v2.py.

Referenced by XMLConfigReader.buildGP(), ConfigurationDBHandler.cvt2String(), edm::DataMixingModule.DataMixingModule(), cms::xerces.dispose(), edm::Worker.doWorkAsync(), SaxToDom.error(), SaxToDom2.error(), L1GtTriggerMenuXmlParser.findXMLChild(), XMLDOMBlock.getNewDocument(), XMLConfigReader.getPatternsVersion(), L1GtTriggerMenuXmlParser.getXMLAttribute(), L1GtTriggerMenuXmlParser.getXMLTextValue(), MonitorXMLParser.handleElement(), cond::XMLAuthenticationService::XMLAuthenticationService.initialize(), L1GtTriggerMenuXmlParser.initXML(), MonitorXMLParser.load(), MonitorXMLParser.MonitorXMLParser(), MuonAlignmentInputXML.newAlignableMuon(), XMLDOMBlock.parse(), L1GtTriggerMenuXmlParser.parseConditions(), MuonAlignmentInputXML.parseDouble(), SummaryPlotXmlParser.parseXML(), cond::XMLAuthenticationService::XMLAuthenticationService.processFile(), HcalItemArrayColl< Item, N >.push_back(), CalibrationXML.readAttribute(), XMLConfigReader.readConfig(), l1t::XmlConfigParser.readContext(), l1t::XmlConfigParser.readHwDescription(), XMLConfigReader.readPatterns(), XMLProcessor.serializeDOM(), HcalItemArrayCollById< Item, N >.setDefault(), SaxToDom.startElement(), SaxToDom2.startElement(), edm::eventsetup::DataKey.swap(), cscdqm::XMLFileErrorHandler.warning(), L1GtTriggerMenuXmlParser.workXML(), CalibrationXML.writeAttribute(), xMLCh2String(), XMLDOMBlock.XMLDOMBlock(), ConfigurationDBHandler.~ConfigurationDBHandler(), cond::persistency::IGTTable.~IGTTable(), LutXml.~LutXml(), MonitorXMLParser.~MonitorXMLParser(), MuonAlignmentInputXML.~MuonAlignmentInputXML(), edm::SingleConsumerQ::OperateBuffer< T >.~OperateBuffer(), cond::persistency::GLOBAL_TAG::Table.~Table(), TagNames.~TagNames(), XMLSimpleStr.~XMLSimpleStr(), lhef::XMLSimpleStr.~XMLSimpleStr(), XMLUniStr.~XMLUniStr(), lhef::XMLUniStr.~XMLUniStr(), and XStr.~XStr().

fetchall_from_DQM_v2.releasedir

Definition at line 113 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.relvaldir

Check for option errors.

Use options

Definition at line 106 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.selected_files

Definition at line 131 of file fetchall_from_DQM_v2.py.

fetchall_from_DQM_v2.usage

Definition at line 74 of file fetchall_from_DQM_v2.py.