Functions
def	auth_wget (url, chunk_size=1048576)

Variables
	action

	all_files

	args

	base_url

	default

	dest

	False

	file_list_re

	file_res
	Fetch the files, using multi-processing. More...

	filedir_html

	filedir_url

	help

	mthreads

	options
	Parse sys.argv. More...

	parser
	Define options. More...

	pool

	regexp

	release

	releasedir

	relvaldir
	Check for option errors. More...

	selected_files

	usage

Detailed Description

Script fetches files matching specified RegExps from DQM GUI.

Author:  Albertas Gimbutas,  Vilnius University (LT)
e-mail:  albertasgim@gmail.com

Function Documentation

◆ auth_wget()

def fetchall_from_DQM_v2.auth_wget	(	url,
		chunk_size = `1048576`
	)

Returns the content of specified URL, which requires authentication.
If the content is bigger than 1MB, then save it to file.

Definition at line 39 of file fetchall_from_DQM_v2.py.

References createfilelist.int, and print().

 def auth_wget(url, chunk_size=1048576):
     """Returns the content of specified URL, which requires authentication.
     If the content is bigger than 1MB, then save it to file.
     """
     opener = build_opener(X509CertOpen())
     url_file = opener.open(Request(url))
     size = int(url_file.headers["Content-Length"])
 
     if size < 1048576:   # if File size < 1MB
         filename = basename(url)    #still download
         readed = url_file.read()    
         if filename != '':
             outfile = open(filename, 'wb')  #then write File to local system
             outfile.write(readed)
         return readed
 
     filename = basename(url)
     file_id = selected_files.index(filename)
 
     if isfile("./%s" % filename):
         print('%d. Exsits on disk. Skipping.' % (file_id +1))
         return
 
     print('%d. Downloading...' % (file_id +1))
     file = open(filename, 'wb')
     # progress = 0
     chunk = url_file.read(chunk_size)
     while chunk:
         file.write(chunk)
         # progress += chunk_size
         chunk = url_file.read(chunk_size)
     print('%d.  Done.' % (file_id +1))
     file.close()
 
 

Variable Documentation

◆ action

fetchall_from_DQM_v2.action

Definition at line 76 of file fetchall_from_DQM_v2.py.

◆ all_files

fetchall_from_DQM_v2.all_files

Definition at line 124 of file fetchall_from_DQM_v2.py.

◆ args

fetchall_from_DQM_v2.args

Definition at line 91 of file fetchall_from_DQM_v2.py.

◆ base_url

fetchall_from_DQM_v2.base_url

Definition at line 116 of file fetchall_from_DQM_v2.py.

◆ default

fetchall_from_DQM_v2.default

Definition at line 82 of file fetchall_from_DQM_v2.py.

◆ dest

fetchall_from_DQM_v2.dest

Definition at line 76 of file fetchall_from_DQM_v2.py.

◆ False

fetchall_from_DQM_v2.False

Definition at line 88 of file fetchall_from_DQM_v2.py.

◆ file_list_re

fetchall_from_DQM_v2.file_list_re

Definition at line 123 of file fetchall_from_DQM_v2.py.

◆ file_res

fetchall_from_DQM_v2.file_res

Fetch the files, using multi-processing.

Definition at line 131 of file fetchall_from_DQM_v2.py.

◆ filedir_html

fetchall_from_DQM_v2.filedir_html

Definition at line 118 of file fetchall_from_DQM_v2.py.

◆ filedir_url

fetchall_from_DQM_v2.filedir_url

Definition at line 117 of file fetchall_from_DQM_v2.py.

◆ help

fetchall_from_DQM_v2.help

Definition at line 77 of file fetchall_from_DQM_v2.py.

◆ mthreads

fetchall_from_DQM_v2.mthreads

Definition at line 126 of file fetchall_from_DQM_v2.py.

◆ options

fetchall_from_DQM_v2.options

Parse sys.argv.

Definition at line 91 of file fetchall_from_DQM_v2.py.

◆ parser

fetchall_from_DQM_v2.parser

Define options.

Definition at line 75 of file fetchall_from_DQM_v2.py.

◆ pool

fetchall_from_DQM_v2.pool

Definition at line 140 of file fetchall_from_DQM_v2.py.

◆ regexp

fetchall_from_DQM_v2.regexp

Definition at line 93 of file fetchall_from_DQM_v2.py.

Referenced by L1TdeStage2uGT.analyze(), compileDMRTrends(), DQMGenericTnPClient.findAllSubdirectories(), HLTMuonRefMethod.findAllSubdirectories(), DQMGenericClient.findAllSubdirectories(), edm.glob2reg(), HLTConfigProvider.matched(), edm.regexMatch(), HLTConfigProvider.removeVersion(), edm.untaintString(), and FWGeometryTableManager.updateFilter().

◆ release

fetchall_from_DQM_v2.release

Definition at line 92 of file fetchall_from_DQM_v2.py.

Referenced by XMLConfigReader.buildGP(), ConfigurationDBHandler.cvt2String(), cms::xerces.dispose(), edm::Worker.doWorkAsync(), SaxToDom.error(), SaxToDom2.error(), L1GtTriggerMenuXmlParser.findXMLChild(), XMLDOMBlock.getNewDocument(), XMLConfigReader.getPatternsVersion(), L1GtTriggerMenuXmlParser.getXMLAttribute(), L1GtTriggerMenuXmlParser.getXMLTextValue(), edm::ConcurrentHadronizerFilter< HAD, DEC >.globalEndRunProduce(), edm::ConcurrentGeneratorFilter< HAD, DEC >.globalEndRunProduce(), MonitorXMLParser.handleElement(), cond::XMLAuthenticationService::XMLAuthenticationService.initialize(), L1GtTriggerMenuXmlParser.initXML(), cond::persistency::GLOBAL_TAG::Table.insert(), MonitorXMLParser.load(), MonitorXMLParser.MonitorXMLParser(), MuonAlignmentInputXML.newAlignableMuon(), XMLDOMBlock.parse(), L1GtTriggerMenuXmlParser.parseConditions(), MuonAlignmentInputXML.parseDouble(), cond::XMLAuthenticationService::XMLAuthenticationService.processFile(), CalibrationXML.readAttribute(), XMLConfigReader.readConfig(), l1t::XmlConfigParser.readContext(), l1t::XmlConfigParser.readHwDescription(), XMLConfigReader.readPatterns(), psClasses.BuildThread.releaseAllLocks(), edm::RepeatingCachedRootSource.RepeatingCachedRootSource(), psClasses.BuildThread.run(), cond::persistency::GLOBAL_TAG::Table.select(), XMLProcessor.serializeDOM(), SaxToDom.startElement(), SaxToDom2.startElement(), cond::persistency::GLOBAL_TAG::Table.update(), cscdqm::XMLFileErrorHandler.warning(), L1GtTriggerMenuXmlParser.workXML(), CalibrationXML.writeAttribute(), xMLCh2String(), XMLDOMBlock.XMLDOMBlock(), ConfigurationDBHandler.~ConfigurationDBHandler(), LutXml.~LutXml(), MonitorXMLParser.~MonitorXMLParser(), MuonAlignmentInputXML.~MuonAlignmentInputXML(), edm::SingleConsumerQ::OperateBuffer< T >.~OperateBuffer(), TagNames.~TagNames(), lhef::XMLSimpleStr.~XMLSimpleStr(), lhef::XMLUniStr.~XMLUniStr(), and XStr.~XStr().

◆ releasedir

fetchall_from_DQM_v2.releasedir

Definition at line 114 of file fetchall_from_DQM_v2.py.

◆ relvaldir

fetchall_from_DQM_v2.relvaldir

Check for option errors.

Use options

Definition at line 107 of file fetchall_from_DQM_v2.py.

◆ selected_files

fetchall_from_DQM_v2.selected_files

Definition at line 132 of file fetchall_from_DQM_v2.py.

◆ usage

fetchall_from_DQM_v2.usage