das-up-to-nevents Namespace Reference

Functions
def	das_do_command (cmd)
def	das_file_data (dataset, opt="")
def	das_file_site (dataset, site)
def	das_lumi_data (dataset, opt="")
def	das_run_data (dataset, opt="")
def	das_run_events_data (dataset, run, opt="")
def	get_lumi_ranges (i)
def	get_url_clean (url)
def	no_intersection ()

Variables
	action
	args
	base_cert_path
	base_cert_url
	Helpers. More...
	best_json
	cert_path
	cert_type
	cert_url
	... More...
	cut
	das_opt
	data_runs
	dataset
	default
	df
	df_r
	df_rs
	events
	files
	golden
	golden_data_runs
	golden_data_runs_tocheck
	golden_flat
	good_lumis
	help
	how
	int
	json_list
	if we have access to eos we get from there ... More...
	latest_json
	lumi_ranges
	lumis
	n_lumis
	None
	on
	opt
	outfile
	parser
	PD
	R
	required
	run_ranges
	site
	str
	sum_events
	testing
	Check if we are in the cms-bot "environment". More...
	threshold
	type
	web_fallback
	year
	get the greatest golden json More...

Function Documentation

das_do_command()

def das-up-to-nevents.das_do_command

(

cmd

)

Definition at line 39 of file das-up-to-nevents.py.

References edm.decode().

Referenced by das_file_data(), das_file_site(), das_lumi_data(), das_run_data(), and das_run_events_data().

def das_do_command(cmd):
    out = subprocess.check_output(cmd, shell=True, executable="/bin/bash").decode('utf8')
    return out.split("\n")

das_file_data()

def das-up-to-nevents.das_file_data	(		dataset,
			opt = ""
	)

Definition at line 50 of file das-up-to-nevents.py.

References das_do_command().

def das_file_data(dataset,opt=""):
    cmd = "dasgoclient --query='file dataset=%s %s| grep file.name, file.nevents'"%(dataset,opt)
    out = das_do_command(cmd)
    out = [np.array(r.split(" "))[[0,3]] for r in out if len(r) > 0]

    df = pd.DataFrame(out,columns=["file","events"])
    df.events = df.events.values.astype(int)
    
    return df

das_file_site()

def das-up-to-nevents.das_file_site	(		dataset,
			site
	)

Definition at line 43 of file das-up-to-nevents.py.

References das_do_command().

def das_file_site(dataset, site):
    cmd = "dasgoclient --query='file dataset=%s site=%s'"%(dataset,site)
    out = das_do_command(cmd)
    df = pd.DataFrame(out,columns=["file"])

    return df

das_lumi_data()

def das-up-to-nevents.das_lumi_data	(		dataset,
			opt = ""
	)

Definition at line 60 of file das-up-to-nevents.py.

References das_do_command().

def das_lumi_data(dataset,opt=""):
    cmd = "dasgoclient --query='file,lumi,run dataset=%s %s'"%(dataset,opt)
    
    out = das_do_command(cmd)
    out = [r.split(" ") for r in out if len(r)>0]
    
    df = pd.DataFrame(out,columns=["file","run","lumis"])
    
    return df

das_run_data()

def das-up-to-nevents.das_run_data	(		dataset,
			opt = ""
	)

Definition at line 79 of file das-up-to-nevents.py.

References das_do_command().

def das_run_data(dataset,opt=""):
    cmd = "dasgoclient --query='run dataset=%s %s '"%(dataset,opt)
    out = das_do_command(cmd)

    return out

das_run_events_data()

def das-up-to-nevents.das_run_events_data	(		dataset,
			run,
			opt = ""
	)

Definition at line 70 of file das-up-to-nevents.py.

References das_do_command(), and int.

def das_run_events_data(dataset,run,opt=""):
    cmd = "dasgoclient --query='file dataset=%s run=%s %s | sum(file.nevents) '"%(dataset,run,opt)
    out = das_do_command(cmd)[0]

    out = [o for o in out.split(" ") if "sum" not in o]
    out = int([r.split(" ") for r in out if len(r)>0][0][0])

    return out

get_lumi_ranges()

def das-up-to-nevents.get_lumi_ranges

(

i

)

Definition at line 32 of file das-up-to-nevents.py.

def get_lumi_ranges(i):
    result = []
    for _, b in itertools.groupby(enumerate(i), lambda pair: pair[1] - pair[0]):
        b = list(b)
        result.append([b[0][1],b[-1][1]]) 
    return result

get_url_clean()

def das-up-to-nevents.get_url_clean

(

url

)

Definition at line 21 of file das-up-to-nevents.py.

def get_url_clean(url):
    
    buffer = BytesIO()
    c = pycurl.Curl()
    c.setopt(c.URL, url)
    c.setopt(c.WRITEDATA, buffer)
    c.perform()
    c.close()
    
    return BeautifulSoup(buffer.getvalue(), "lxml").text

no_intersection()

def das-up-to-nevents.no_intersection

(

)

Definition at line 85 of file das-up-to-nevents.py.

References print().

def no_intersection():
    print("No intersection between:")
    print(" - json   : ", best_json)
    print(" - dataset: ", dataset)
    print("Exiting.")
    sys.exit(1)

Variable Documentation

action

das-up-to-nevents.action

Definition at line 99 of file das-up-to-nevents.py.

args

das-up-to-nevents.args

Definition at line 104 of file das-up-to-nevents.py.

base_cert_path

das-up-to-nevents.base_cert_path

Definition at line 19 of file das-up-to-nevents.py.

base_cert_url

das-up-to-nevents.base_cert_url

Helpers.

Definition at line 18 of file das-up-to-nevents.py.

best_json

das-up-to-nevents.best_json

Definition at line 155 of file das-up-to-nevents.py.

cert_path

das-up-to-nevents.cert_path

Definition at line 132 of file das-up-to-nevents.py.

cert_type

das-up-to-nevents.cert_type

Definition at line 124 of file das-up-to-nevents.py.

cert_url

das-up-to-nevents.cert_url

...

if not we go to the website

Definition at line 146 of file das-up-to-nevents.py.

cut

das-up-to-nevents.cut

Definition at line 204 of file das-up-to-nevents.py.

das_opt

das-up-to-nevents.das_opt

Definition at line 181 of file das-up-to-nevents.py.

data_runs

das-up-to-nevents.data_runs

Definition at line 173 of file das-up-to-nevents.py.

dataset

das-up-to-nevents.dataset

Definition at line 114 of file das-up-to-nevents.py.

default

das-up-to-nevents.default

Definition at line 95 of file das-up-to-nevents.py.

df

das-up-to-nevents.df

Definition at line 199 of file das-up-to-nevents.py.

df_r

das-up-to-nevents.df_r

Definition at line 208 of file das-up-to-nevents.py.

df_rs

das-up-to-nevents.df_rs

Definition at line 202 of file das-up-to-nevents.py.

events

das-up-to-nevents.events

Definition at line 115 of file das-up-to-nevents.py.

files

das-up-to-nevents.files

Definition at line 237 of file das-up-to-nevents.py.

golden

das-up-to-nevents.golden

Definition at line 158 of file das-up-to-nevents.py.

golden_data_runs

das-up-to-nevents.golden_data_runs

Definition at line 174 of file das-up-to-nevents.py.

golden_data_runs_tocheck

das-up-to-nevents.golden_data_runs_tocheck

Definition at line 180 of file das-up-to-nevents.py.

golden_flat

das-up-to-nevents.golden_flat

Definition at line 164 of file das-up-to-nevents.py.

good_lumis

das-up-to-nevents.good_lumis

Definition at line 214 of file das-up-to-nevents.py.

help

das-up-to-nevents.help

Definition at line 95 of file das-up-to-nevents.py.

how

das-up-to-nevents.how

Definition at line 199 of file das-up-to-nevents.py.

int

das-up-to-nevents.int

Definition at line 96 of file das-up-to-nevents.py.

Referenced by das_run_events_data().

json_list

das-up-to-nevents.json_list

if we have access to eos we get from there ...

Definition at line 137 of file das-up-to-nevents.py.

Referenced by querying.factory.object().

latest_json

das-up-to-nevents.latest_json

Definition at line 154 of file das-up-to-nevents.py.

lumi_ranges

das-up-to-nevents.lumi_ranges

Definition at line 240 of file das-up-to-nevents.py.

lumis

das-up-to-nevents.lumis

Definition at line 119 of file das-up-to-nevents.py.

Referenced by edm::RootFile.fillIndexIntoFile(), edm::IndexIntoFile::IndexIntoFileItrImpl.getLumisInRun(), edm::IndexIntoFile::IndexIntoFileItr.getLumisInRun(), edm::IndexIntoFileItrHolder.getLumisInRun(), HcalDbHardcode.makePFCut(), edm::MergeableRunProductMetadata.readRun(), and edm::EventSetupRecordDataGetter.registerLateConsumes().

n_lumis

das-up-to-nevents.n_lumis

Definition at line 215 of file das-up-to-nevents.py.

None

das-up-to-nevents.None

Definition at line 95 of file das-up-to-nevents.py.

on

das-up-to-nevents.on

Definition at line 199 of file das-up-to-nevents.py.

opt

das-up-to-nevents.opt

Definition at line 199 of file das-up-to-nevents.py.

Referenced by PhysicsTools::TreeReader.addBranch(), PhysicsTools::TreeReader.addTypeSingle(), VpspScanAlgorithm.analyse(), TEcnaParHistos.AxisLabelOffset(), TEcnaParHistos.AxisLabelSize(), TEcnaParHistos.AxisTickSize(), TEcnaParHistos.AxisTitleOffset(), TEcnaParHistos.AxisTitleSize(), PedeSteerer.buildMasterSteer(), call_pyupda(), DQMGenericClient.DQMGenericClient(), DQMRivetClient.DQMRivetClient(), fwlite::Event.draw(), HLTGenValClient.makeAllPlots(), TMultiDimFet.MakeHistograms(), CommandLine.parse(), cxxopts::ParseResult.parse(), python.rootplot.core.parse_arguments(), TMultiDimFet.Print(), TMultiDimFet.PrintPolynomialsSpecial(), LegacyIOHelper.save(), PhysicsTools::TreeReader::Value.setOpt(), PhysicsTools::TreeReader.setOptional(), TMultiDimFet.TMultiDimFet(), and TEcnaHistos.TopAxisForHistos().

outfile

das-up-to-nevents.outfile

Definition at line 117 of file das-up-to-nevents.py.

parser

das-up-to-nevents.parser

Definition at line 94 of file das-up-to-nevents.py.

PD

das-up-to-nevents.PD

Definition at line 123 of file das-up-to-nevents.py.

Referenced by HLTScalers.dqmBeginRun().

R

das-up-to-nevents.R

Definition at line 166 of file das-up-to-nevents.py.

required

das-up-to-nevents.required

Definition at line 95 of file das-up-to-nevents.py.

run_ranges

das-up-to-nevents.run_ranges

Definition at line 153 of file das-up-to-nevents.py.

site

das-up-to-nevents.site

Definition at line 118 of file das-up-to-nevents.py.

Referenced by Measurement.constructFromOA(), edm::service::SiteLocalConfigService.getCatalog(), XrdAdaptor::XrdSiteStatisticsInformation.getStatisticsForSite(), XrdAdaptor::Source.getXrootdSite(), XrdAdaptor::Source.getXrootdSiteFromURL(), edm::FileLocator.init(), CocoaToDDLMgr.measurementsAsSpecPars(), edm::service::SiteLocalConfigService.parse(), XrdAdaptor::Source.setXrootdSite(), and edm::service::CondorStatusService.updateImpl().

str

das-up-to-nevents.str

Definition at line 95 of file das-up-to-nevents.py.

sum_events

das-up-to-nevents.sum_events

Definition at line 189 of file das-up-to-nevents.py.

testing

das-up-to-nevents.testing

Check if we are in the cms-bot "environment".

Definition at line 113 of file das-up-to-nevents.py.

threshold

das-up-to-nevents.threshold

Definition at line 116 of file das-up-to-nevents.py.

type

das-up-to-nevents.type

Definition at line 95 of file das-up-to-nevents.py.

web_fallback

das-up-to-nevents.web_fallback

Definition at line 133 of file das-up-to-nevents.py.

year

das-up-to-nevents.year

get the greatest golden json

Definition at line 122 of file das-up-to-nevents.py.