mps_update Namespace Reference

Functions
def	fill_time_info (mps_index, status, cpu_time)

Variables
	condor_h
	condor_q
	cpu_time
string	disabled = "DISABLED" if "DISABLED" in lib.JOBSTATUS[mps_index] else ""
	end
	check for orphaned jobs More...
dictionary	htcondor_jobstatus
	mapping of HTCondor status codes to MPS status More...
	job_id
dictionary	job_status = {}
	deal with submitted jobs by looking into output of shell (condor_q) More...
	lib = mpslib.jobdatabase()
	collect submitted jobs (use 'in' to handle composites, e.g. More...
dictionary	mps_index = submitted_jobs.get(job_id, -1)
	status = htcondor_jobstatus[status]
bool	submitted = True
dictionary	submitted_jobs = {}
dictionary	submitted_jobs_copy = { k:v for k,v in submitted_jobs.items() }
	loop over remaining jobs to see whether they are done More...
	userlog = os.path.join("jobData", lib.JOBDIR[mps_index], "HTCJOB")

Function Documentation

fill_time_info()

def mps_update.fill_time_info	(		mps_index,
			status,
			cpu_time
	)

Fill timing info in the database for `mps_index`.

Arguments:
- `mps_index`: index in the MPS database
- `status`: job status
- `cpu_time`: extracted CPU timing information

Definition at line 10 of file mps_update.py.

References createfilelist.int, and str.

def fill_time_info(mps_index, status, cpu_time):
    """Fill timing info in the database for `mps_index`.

    Arguments:
    - `mps_index`: index in the MPS database
    - `status`: job status
    - `cpu_time`: extracted CPU timing information
    """

    cpu_time = int(round(cpu_time))  # care only about seconds for now
    if status in ("RUN", "DONE"):
        if cpu_time > 0:
            diff = cpu_time - lib.JOBRUNTIME[mps_index]
            lib.JOBRUNTIME[mps_index] = cpu_time
            lib.JOBHOST[mps_index] = "+"+str(diff)
            lib.JOBINCR[mps_index] = diff
        else:
            lib.JOBRUNTIME[mps_index] = 0
            lib.JOBINCR[mps_index] = 0

Variable Documentation

condor_h

mps_update.condor_h

Initial value:

=  subprocess.check_output(["condor_history", job_id, "-limit", "1",
                                        "-userlog", userlog,
                                        "-af:j", "JobStatus", "RemoteSysCpu"],
                                       stderr = subprocess.STDOUT).decode()

Definition at line 101 of file mps_update.py.

condor_q

mps_update.condor_q

Initial value:

=  subprocess.check_output(["condor_q", "-af:j",
                                        "JobStatus", "RemoteSysCpu"],
                                       stderr = subprocess.STDOUT).decode()

Definition at line 64 of file mps_update.py.

cpu_time

mps_update.cpu_time

Definition at line 68 of file mps_update.py.

disabled

string mps_update.disabled = "DISABLED" if "DISABLED" in lib.JOBSTATUS[mps_index] else ""

Definition at line 75 of file mps_update.py.

end

mps_update.end

check for orphaned jobs

Definition at line 79 of file mps_update.py.

htcondor_jobstatus

dictionary mps_update.htcondor_jobstatus

Initial value:

=  {"1": "PEND", # Idle
                      "2": "RUN",  # Running
                      "3": "EXIT", # Removed
                      "4": "DONE", # Completed
                      "5": "PEND", # Held
                      "6": "RUN",  # Transferring output
                      "7": "PEND"}

mapping of HTCondor status codes to MPS status

Definition at line 34 of file mps_update.py.

job_id

mps_update.job_id

Definition at line 68 of file mps_update.py.

job_status

dictionary mps_update.job_status = {}

deal with submitted jobs by looking into output of shell (condor_q)

Definition at line 63 of file mps_update.py.

lib

mps_update.lib = mpslib.jobdatabase()

collect submitted jobs (use 'in' to handle composites, e.g.

DISABLEDFETCH)

Definition at line 45 of file mps_update.py.

mps_index

mps_update.mps_index = submitted_jobs.get(job_id, -1)

Definition at line 73 of file mps_update.py.

status

dictionary mps_update.status = htcondor_jobstatus[status]

Definition at line 68 of file mps_update.py.

Referenced by XMLProcessor._toXMLCh(), addFilesWithFork(), cms::cuda::HostAllocator< T, FLAGS >.allocate(), ESTimingTask.analyze(), CSCTFAnalyzer.analyze(), TKStatus.analyze(), BeamSpotProblemMonitor.analyze(), SiPixelGenErrorDBObjectReader.analyze(), BasicHepMCValidation.analyze(), DTNoiseCalibration.analyze(), TotemRPDQMSource.analyze(), WriteL1TriggerObjectsTxt.analyze(), CounterChecker.Analyze(), L1TCSCTF.analyze(), L1Validator.analyze(), CTPPSDiamondDQMSource.analyze(), HLTriggerJSONMonitoring.analyze(), FWCaloDataHistProxyBuilder.assertCaloDataSlice(), edm::EventProcessor.beginLumiAsync(), edm::EventProcessor.beginRunAsync(), EcalBoundaryInfoCalculator< EBDetId >.boundaryRecHits(), gen::AMPTHadronizer.build_ampt(), TtFullHadHypKinFit.buildHypo(), TtSemiLepHypHitFit.buildHypo(), TtSemiLepHypKinFit.buildHypo(), CaloGeometryHelper.buildNeighbourArray(), hgcal::backend.buildSlinkTrailer(), PFMETBenchmark.calculateQuantities(), CastorChannelStatus.CastorChannelStatus(), EcalRecHitWorkerRecover.checkChannelStatus(), pat::UserData.checkDictionaries(), dqm::DQMFileSaverOnline.checkError(), CTPPSDiamondDQMSource.checkEventNumber(), edm.checkForModuleDependencyCorrectness(), lhef::LHEEvent.checkHepMCTree(), HFTimingTrust.checkHFTimErr(), DAQSource.checkNext(), SiPixelFedCablingTree.checkNumbering(), EcalBoundaryInfoCalculator< EBDetId >.checkRecHitHasDeadNeighbour(), EgammaHLTNxNClusterProducer.checkStatusOfEcalRecHit(), HLTEcalResonanceFilter.checkStatusOfEcalRecHit(), HLTRegionalEcalResonanceFilter.checkStatusOfEcalRecHit(), DataModeFRDStriped.checksumValid(), l1tVertexFinder::VertexNTupler::GenParticlesBranchData.clear(), tensorflow.closeSession(), edm::service.cmssw_stacktrace_fork(), compare_l1candidate(), dtCalibration::DTVDriftSegment.compute(), HFShower.compute(), HDShower.compute(), EMShower.compute(), SiPixelUtility.computeErrorCode(), EcalHitMaker.configureGeometry(), edm::EventProcessor.continueLumiAsync(), ErrorCheckerBase.conversionError(), dtCalibration::DTTTrigMatchRPhi.correction(), dtCalibration::DTTTrigFillWithAverage.correction(), dtCalibration::DTTTrigConstantShift.correction(), dtCalibration::DTT0FillChamberFromDB.correction(), dtCalibration::DTT0FillDefaultFromDB.correction(), dtCalibration::DTT0AbsoluteReferenceCorrection.correction(), dtCalibration::DTT0FEBPathCorrection.correction(), dtCalibration::DTTTrigT0SegCorrection.correction(), dtCalibration::DTT0ChamberReferenceCorrection.correction(), dtCalibration::DTT0WireInChamberReferenceCorrection.correction(), dtCalibration::DTTTrigResidualCorrection.correction(), TMatacq.countBadPulses(), tensorflow.createSession(), CUDAService.CUDAService(), DDIsValid(), cms::cuda::HostAllocator< T, FLAGS >.deallocate(), gen::TauolappInterface.decay(), LzmaFile.DecodeAll(), LzmaFile.DecodeBuffer(), dtCalibration::DTT0AbsoluteReferenceCorrection.defaultT0(), dtCalibration::DTT0ChamberReferenceCorrection.defaultT0(), dtCalibration::DTT0FEBPathCorrection.defaultT0(), dtCalibration::DTT0WireInChamberReferenceCorrection.defaultT0(), Demangle.Demangle(), EcalHitMaker.diagonalEdge(), doBZ2Compression(), CompareAlignments.doComparison(), edm::EDLooperBase.doDuringLoop(), PulseChiSqSNNLS.DoFit(), SiPixelGainCalibrationAnalysis.doFits(), EmissionVetoHook1.doVetoFSREmission(), EmissionVetoHook1.doVetoISREmission(), doZlibCompression(), SiPixelLorentzAnglePCLHarvesterMCS.dqmEndJob(), TKStatus.dumpTkDcsStatus(), PFECALHashNavigator.ecalNeighbArray(), DTT0Correction.endJob(), DTVDriftWriter.endJob(), DTTTrigCorrection.endJob(), edm::EventProcessor.endOfLoop(), TMatacq.enterdata(), ErrorChecker.errorDetId(), ErrorCheckerPhase0.errorDetId(), ErrorChecker.errorDetIdSimple(), ESRecHitAnalyticAlgo.EvalAmplitude(), ESRecHitFitAlgo.EvalAmplitude(), ESRecHitSimAlgo.evalAmplitude(), PhysicsTools::MVAComputer.evalInternal(), ExternalLHEProducer.executeScript(), EcalHitMaker.fastInsideCell(), MuScleFitPlotter.fillGen(), Py8toJetInput.fillJetAlgoInput(), LHETablesProducer.fillLHEObjectTable(), PFElectronTranslator.fillMVAValueMap(), TrackingQualityChecker.fillTrackingStatus(), TrackingQualityChecker.fillTrackingStatusAtLumi(), PythiaFilterTTBar.filter(), PythiaFilterIsolatedTrack.filter(), EcalDeadCellBoundaryEnergyFilter.filter(), Phase2Tracker::Phase2TrackerFEDBuffer.findChannels(), GenParticlesHelper.findDescendents(), HepMCValidationHelper.findFSRPhotons(), GenHFHadronMatcher.findInMothers(), L1GtVhdlWriterCore.findObjectType(), GenParticlesHelper.findParticles(), edm::Path.finished(), UnbinnedLikelihoodFit.fit(), SiPixelLAHarvestMCS::FitFPixMuH.fit(), TtFullHadKinFitter::KinFit.fit(), BSFitter.Fit_d0phi(), for(), EcalLaserCondTools.from_hdf_to_db(), Phase2Tracker::Phase2TrackerFEDHeader.frontendStatus(), EcalBoundaryInfoCalculator< EBDetId >.gapRecHits(), GaussianTailNoiseGenerator.generate(), gen::BaseHadronizer.generateLHE(), pat::PATObject< reco::Muon >.genParticleById(), DTTtrig.get(), DTMtime.get(), Py8toJetInput.getAncestor(), dtCalibration::DTTTrigFillWithAverage.getAverage(), ESTrivialConditionRetriever.getChannelStatusFromConfiguration(), EcalTrivialConditionRetriever.getChannelStatusFromConfiguration(), EcalDeadCellTriggerPrimitiveFilter.getChannelStatusMaps(), EcalDeadCellDeltaRFilter.getChannelStatusMaps(), ErrorCheckerBase.getConversionErrorTypeAndIssueLogMessage(), pat::EventHypothesis.getDemangledSymbol(), SiStripUtility.getDetectorStatusColor(), cscdqm::Summary.GetEfficiencyHW(), PixelSLinkDataInputSource.getEventNumberFromFillWords(), HLTTauMCProducer.getGenDecayProducts(), CrossingPtBasedLinearizationPointFinder.getLinearizationPoint(), TrackingUtility.getMEStatus(), SiStripUtility.getMEStatus(), TrackingUtility.getMEStatusColor(), SiStripUtility.getMEStatusColor(), DTDeadFlagHandler.getNewObjects(), DTPerformanceHandler.getNewObjects(), DTRangeT0Handler.getNewObjects(), DTReadOutMappingHandler.getNewObjects(), DTStatusFlagHandler.getNewObjects(), DTT0Handler.getNewObjects(), DTTPGParametersHandler.getNewObjects(), DAQSource.getNextDataBlock(), FedRawDataInputSource.getNextEvent(), cscdqm::Summary.GetReportingArea(), HepMCFileReader.getStatsFromTuple(), SiPixelUtility.getStatus(), SiPixelUtility.getStatusColor(), EcalTrivialConditionRetriever.getTrgChannelStatusFromConfiguration(), edm::EventProcessor.globalEndLumiAsync(), edm::EventProcessor.globalEndRunAsync(), lhef::H5Handler.H5Handler(), XrdAdaptor::Source.handle(), edm::EventProcessor.handleNextEventForStreamAsync(), XrdAdaptor::RequestManager.handleOpen(), XrdAdaptor::ClientRequest.HandleResponse(), SendMonitoringInfoHandler.HandleResponse(), QueryAttrHandler.HandleResponse(), DelayedClose.HandleResponseWithHosts(), XrdAdaptor::RequestManager::OpenHandler.HandleResponseWithHosts(), egamma.hasActiveHcal(), GenParticlesHelper.hasAncestor(), CalorimetryManager.HDShowerSimulation(), Generator.HepMC2G4(), HcalHFStatusBitFromRecHits.hfSetFlagFromRecHits(), PFBadHcalPseudoClusterProducer.init(), fireworks::Context.initEveElements(), TTUBasicConfig.initialise(), RBCBasicConfig.initialise(), LogicTool< GenLogic >.initialise(), RPCWheel.initialise(), RBCEmulator.initialise(), TTUEmulator.initialise(), XrdAdaptor::RequestManager.initialize(), Pythia8HepMC3Hadronizer.initializeForExternalPartons(), Pythia8Hadronizer.initializeForExternalPartons(), Pythia8HepMC3Hadronizer.initializeForInternalPartons(), Pythia8Hadronizer.initializeForInternalPartons(), edm::ParameterDescriptionCases< T >.insert(), edm::ParameterSwitchBase.insertAndCheckLabels(), HcalObjRepresent.isBitSet(), MBUEandQCDValidation.isCharged(), HepMCValidationHelper.isChargedLepton(), EcalDeadCellDeltaRFilter.isCloseToBadEcalChannel(), AnomalousECALVariables.isDeadEcalCluster(), GenParticlesHelper.isDirect(), MBUEandQCDValidation.isNeutral(), MBUEandQCDValidation.isNeutrino(), HepMCValidationHelper.isNeutrino(), npstat.isNonDecreasing(), npstat.isNonIncreasing(), cscdqm::Summary.IsPhysicsReady(), isRocmDeviceSupported(), npstat.isStrictlyDecreasing(), AbsHcalFunctor.isStrictlyDecreasing(), npstat.isStrictlyIncreasing(), AbsHcalFunctor.isStrictlyIncreasing(), FFTJetProducer.iterateJetReconstruction(), tensorflow.loadGraphDef(), tensorflow.loadMetaGraphDef(), LzmaDec_DecodeToBuf(), LzmaDec_DecodeToDic(), LzmaDecode(), main(), gen::TauolappInterface.make_simple_tau_event(), lhef::LHEEvent.makeHepMCParticle(), gen.GeneratorAnalyzer.GeneratorAnalyzer.makeMCInfo(), EcalUncalibRecHitMultiFitAlgo.makeRecHit(), reco::MCMatchSelector< T1, T2 >.MCMatchSelector(), CompareAlignments.MergeRootfile(), PulseChiSqSNNLS.Minimize(), CaloGeometryHelper.move(), FedRawDataInputSource.nextEvent(), ESRecHitSimAlgo.oldEvalAmplitude(), ESRecHitSimAlgo.oldreconstruct(), edm::storage::XrdFile.open(), XrdAdaptor::RequestManager::OpenHandler.open(), TEcnaRead.OpenRootFile(), TEcnaRun.OpenRootFile(), TSCBLBuilderNoMaterial.operator()(), GenJetParticleSelector.operator()(), FastTimerService::ResourcesPerPath.operator+=(), operator<<(), trigger::TriggerObject.particle(), Generator.particleAssignDaughters(), FFTJetCorrectionProducer.performPileupSubtraction(), pos::PixelDetectorConfig.PixelDetectorConfig(), PixelTemplateSmearerBase.PixelTemplateSmearerBase(), SymmetryFit.pol2_from_pol2(), SymmetryFit.pol2_from_pol3(), FastTimerService.postPathEvent(), edm::OccurrenceTraits< EventPrincipal, BranchActionStreamBegin >.postPathSignal(), HepMCFileReader.printEvent(), TMatacq.printitermatacqData(), RBCChamberORLogic.process(), RPCWheel.process(), edm::EventProcessor.processEventWithLooper(), TtSemiLepKinFitProducer< LeptonCollection >.produce(), PseudoTopProducer.produce(), EcalRecHitParametersGPUESProducer.produce(), PFElectronTranslator.produce(), RPCUnpackingModule.produce(), CSCTFUnpacker.produce(), PFPhotonTranslator.produce(), Onia2MuMuPAT.produce(), FlavorHistoryProducer.produce(), l1t::GenToInputProducer.produce(), RPCTechnicalTrigger.produce(), FFTJetProducer.produce(), HcalHardcodeCalibrations.produceChannelQuality(), EcalTrivialConditionRetriever.produceEcalDAQTowerStatus(), EcalTrivialConditionRetriever.produceEcalDCSTowerStatus(), ecaldqm::CalibrationSummaryClient.producePlots(), ecaldqm::SummaryClient.producePlots(), hgcal::HGCalFrameGenerator.produceSlinkEvent(), read_processor_model(), edm::EventProcessor.readAndMergeRunEntriesAsync(), StoreEcalCondition.readEcalChannelStatusFromFile(), StoreESCondition.readESChannelStatusFromFile(), DTReadOutMapping.readOutToGeometry(), FedRawDataInputSource.readSupervisor(), DAQSource.readSupervisor(), edm::storage::Storage.readv(), EcalDCSTowerStatusXMLTranslator.readXML(), EcalDAQTowerStatusXMLTranslator.readXML(), ESRecHitSimAlgo.reconstruct(), ESRecHitAnalyticAlgo.reconstruct(), ESRecHitFitAlgo.reconstruct(), edm::EventProcessor.releaseBeginRunResources(), XrdAdaptor::RequestManager.requestFailure(), FastTimerService::ResourcesPerPath.reset(), gen::Pythia6Hadronizer.residualDecay(), TEcnaRootFile.ReStart(), L1GtVhdlWriterCore.returnConditionsOfOneClass(), dqmservices::TriggerSelector::TreeElement.returnStatus(), ROCmService.ROCmService(), Pythia8PartonSelector.run(), RawToDigiConverter.run(), tensorflow.run(), ecaldqm::RawDataTask.runOnRawData(), BaseHiGenEvtSelector.selectParticle(), PartonHadronDecayGenEvtSelector.selectParticle(), HadronDecayGenEvtSelector.selectParticle(), HcalSourcePositionData.set(), CSCDDUHeader.setBOEStatus(), DTStatusFlag.setCellDead(), DTDeadFlag.setCellDead_HV(), DTDeadFlag.setCellDead_RO(), DTDeadFlag.setCellDead_TP(), DTDeadFlag.setCellDiscCat(), DTStatusFlag.setCellFEMask(), DTStatusFlag.setCellNoHV(), DTStatusFlag.setCellNoise(), DTStatusFlag.setCellTDCMask(), DTStatusFlag.setCellTrigMask(), ESRecHitSimAlgo.setChannelStatus(), ESRecHitAnalyticAlgo.setChannelStatus(), ESRecHitFitAlgo.setChannelStatus(), sistrip::FEDFEHeader.setChannelStatus(), sistrip::FEDAPVErrorHeader.setChannelStatus(), sistrip::FEDFullDebugHeader.setChannelStatus(), FWLegoViewBase.setContext(), EcalDeadCellTriggerPrimitiveFilter.setEvtRecHitstatus(), EcalDeadCellTriggerPrimitiveFilter.setEvtTPstatus(), EcalRecHitWorkerSimple.setFlagBits(), ESDigitizer.setGain(), EcalTBHodoscopePlaneRawHits.setHit(), reco::CSCHaloData.SetHLTBit(), FFTJetProducer.setJetStatusBit(), CSCDDUHeader.setOutputPathStatus(), reco::SuperCluster.setPreshowerPlanesStatus(), L1Analysis::L1AnalysisCSCTF.SetStatus(), reco::PFCandidateElectronExtra.setStatus(), reco::PFCandidateEGammaExtra.setStatus(), MonDelaysTTDat.setTaskStatus(), MonPedestalsOnlineDat.setTaskStatus(), MonPedestalOffsetsDat.setTaskStatus(), MonLaserBlueDat.setTaskStatus(), MonLaserRedDat.setTaskStatus(), MonLaserGreenDat.setTaskStatus(), MonPNPedDat.setTaskStatus(), MonLaserIRedDat.setTaskStatus(), MonLed2Dat.setTaskStatus(), MonLed1Dat.setTaskStatus(), MonMemChConsistencyDat.setTaskStatus(), MonCrystalConsistencyDat.setTaskStatus(), MonPedestalsDat.setTaskStatus(), MonMemTTConsistencyDat.setTaskStatus(), MonTTConsistencyDat.setTaskStatus(), MonTestPulseDat.setTaskStatus(), MonPNRedDat.setTaskStatus(), MonPNIRedDat.setTaskStatus(), MonPNLed2Dat.setTaskStatus(), MonPNBlueDat.setTaskStatus(), MonPNMGPADat.setTaskStatus(), MonPNGreenDat.setTaskStatus(), MonPNLed1Dat.setTaskStatus(), ITimingDat.setTaskStatus(), HBHETimingShapedFlagSetter.SetTimingShapedFlags(), CSCDDUHeader.setTTSStatus(), StatusSelector.StatusSelector(), edm::EventProcessor.streamBeginRunAsync(), edm::EventProcessor.streamEndLumiAsync(), SymmetryFit.symmetryChi2(), TEcnaRootFile.TEcnaRootFile(), cms::cudatest.testDevices(), cms::rocmtest.testDevices(), CSCFitAFEBThr.ThresholdNoise(), HBHETimingShapedFlagSetter.timingStatus(), TMatacq.TMatacq(), sistrip.toString(), HLTPerformanceInfo.totalPathCPUTime(), HLTPerformanceInfo.totalPathTime(), toTString(), edm.typeDemangle(), pat::UserData.typeNameFor(), HcaluLUTTPGCoder.update(), edm::service::CondorStatusService.updateChirpImpl(), PulseChiSqSNNLS.updateCov(), FWGUIManager.updateStatus(), CmsShowMainFrame.updateStatusBar(), CrossingPtBasedLinearizationPointFinder.useAllTracks(), cscdqm::Summary.Write(), npstat::LinInterpolatedTableND< Numeric, Axis >.write(), EPOS::IO_EPOS.write_event(), gs.writeCompressedStringArchive(), and gs.writeStringArchive().

submitted

bool mps_update.submitted = True

Definition at line 50 of file mps_update.py.

submitted_jobs

dictionary mps_update.submitted_jobs = {}

Definition at line 48 of file mps_update.py.

submitted_jobs_copy

dictionary mps_update.submitted_jobs_copy = { k:v for k,v in submitted_jobs.items() }

loop over remaining jobs to see whether they are done

Definition at line 93 of file mps_update.py.

userlog

mps_update.userlog = os.path.join("jobData", lib.JOBDIR[mps_index], "HTCJOB")

Definition at line 100 of file mps_update.py.