CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
List of all members | Public Member Functions | Public Attributes | Private Attributes
CrabHelper.CrabHelper Class Reference
Inheritance diagram for CrabHelper.CrabHelper:

Public Member Functions

def __init__
 
def cert_info
 
def check_crabtask
 
def crab
 
def crab_config_filename
 
def crab_config_filepath
 
def crab_taskname
 
def create_crab_config
 
def fill_options_from_crab_config
 
def submit_crab_task
 
def voms_proxy_create
 
def voms_proxy_time_left
 
def write_crabConfig
 

Public Attributes

 crab_config
 
 crabFunctions
 

Private Attributes

 _cert_info
 
 _crab
 

Detailed Description

Definition at line 12 of file CrabHelper.py.

Constructor & Destructor Documentation

def CrabHelper.CrabHelper.__init__ (   self)

Definition at line 14 of file CrabHelper.py.

14 
15  def __init__(self):
16  # perform imports only when creating instance. This allows to use the classmethods e.g.for
17  # CLI construction before crab is sourced.
18  self.crabFunctions = import_module('CalibMuon.DTCalibration.Workflow.Crabtools.crabFunctions')
19  # cached member variables
20  self._crab = None
21  self._cert_info = None

Member Function Documentation

def CrabHelper.CrabHelper.cert_info (   self)

Definition at line 213 of file CrabHelper.py.

References CrabHelper.CrabHelper._cert_info, CrabHelper.CrabHelper.voms_proxy_create(), and CrabHelper.CrabHelper.voms_proxy_time_left().

213 
214  def cert_info(self):
215  if not self._cert_info:
216  if not self.voms_proxy_time_left() > 0:
217  warn_msg = "No valid proxy, a default proxy without a specific"
218  warn_msg = "VOGroup will be used"
219  self.voms_proxy_create()
220  log.warning(warn_msg)
221  self._cert_info = self.crabFunctions.CertInfo()
222  return self._cert_info
def CrabHelper.CrabHelper.check_crabtask (   self)

Definition at line 50 of file CrabHelper.py.

References CrabHelper.CrabHelper.crab_config_filepath(), DTWorkflow.DTWorkflow.get_output_files(), tools.getTerminalSize(), join(), DTWorkflow.DTWorkflow.local_path(), print(), sistrip::SpyUtilities.range(), and tools.stdinWait().

Referenced by DTWorkflow.DTWorkflow.check().

50 
51  def check_crabtask(self):
52  print(self.crab_config_filepath)
53  task = self.crabFunctions.CrabTask(crab_config = self.crab_config_filepath,
54  initUpdate = False)
55  if self.options.no_exec:
56  log.info("Nothing to check in no-exec mode")
57  return True
58  for n_check in range(self.options.max_checks):
59  task.update()
60  if task.state in ( "COMPLETED"):
61  print("Crab task complete. Getting output locally")
62  output_path = os.path.join( self.local_path, "unmerged_results" )
63  self.get_output_files(task, output_path)
64  return True
65  if task.state in ("SUBMITFAILED", "FAILED"):
66  print("Crab task failed")
67  return False
68  possible_job_states = ["nUnsubmitted",
69  "nIdle",
70  "nRunning",
71  "nTransferring",
72  "nCooloff",
73  "nFailed",
74  "nFinished",
75  "nComplete" ]
76 
77  jobinfos = ""
78  for jobstate in possible_job_states:
79  njobs_in_state = getattr(task, jobstate)
80  if njobs_in_state > 0:
81  jobinfos+="%s: %d " % (jobstate[1:], njobs_in_state)
82 
83  #clear line for reuse
84  sys.stdout.write("\r")
85  sys.stdout.write("".join([" " for i in range(tools.getTerminalSize()[0])]))
86  sys.stdout.write("\r")
87  prompt_text = "Check (%d/%d). Task state: %s (%s). Press q and enter to stop checks: " % (n_check,
88  self.options.max_checks, task.state, jobinfos)
89  user_input = tools.stdinWait(prompt_text, "", self.options.check_interval)
90  if user_input in ("q","Q"):
91  return False
92  print("Task not completed after %d checks (%d minutes)" % ( self.options.max_checks,
93  int( self.options.check_interval / 60. )))
94  return False
sistrip::SpyUtilities::range
const uint16_t range(const Frame &aFrame)
Definition: SiStripSpyUtilities.cc:76
tools.getTerminalSize
def getTerminalSize
Definition: tools.py:96
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
tools.stdinWait
def stdinWait
Definition: tools.py:70
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
def CrabHelper.CrabHelper.crab (   self) 
Retuns a CrabController instance from cache or creates new
   on on first call

Definition at line 202 of file CrabHelper.py.

References CrabHelper.CrabHelper._crab.

202 
203  def crab(self):
204  """ Retuns a CrabController instance from cache or creates new
205  on on first call """
206  if self._crab is None:
207  if self.cert_info.voGroup:
208  self._crab = self.crabFunctions.CrabController(voGroup = self.cert_info.voGroup)
209  else:
210  self._crab = self.crabFunctions.CrabController()
211  return self._crab
def CrabHelper.CrabHelper.crab_config_filename (   self)

Definition at line 224 of file CrabHelper.py.

References CrabHelper.CrabHelper.crab_taskname(), cmsswPreprocessor.CmsswPreprocessor.options, DTCalibrationWorker.DTCalibrationWorker.options, DTWorkflow.DTWorkflow.options, TestProcess.TestProcess.options, confdb.HLTProcess.options, VisualizationOptions.options(), production_tasks.Task.options, fireworks::OptionNode.options(), edmIntegrityCheck.IntegrityCheck.options, Config.Process.options, and validateAlignments.ValidationJobMultiIOV.options.

Referenced by CrabHelper.CrabHelper.crab_config_filepath(), CrabHelper.CrabHelper.fill_options_from_crab_config(), and CrabHelper.CrabHelper.write_crabConfig().

224 
225  def crab_config_filename(self):
226  if hasattr(self.options, "crab_config_path"):
227  return self.options.crab_config_path
228  return 'crab_%s_cfg.py' % self.crab_taskname
def CrabHelper.CrabHelper.crab_config_filepath (   self)

Definition at line 230 of file CrabHelper.py.

References CrabHelper.CrabHelper.crab_config_filename(), and DTWorkflow.DTWorkflow.local_path().

Referenced by CrabHelper.CrabHelper.check_crabtask(), DTWorkflow.DTWorkflow.prepare_common_write(), DTVdriftWorkflow.DTvdriftWorkflow.prepare_meantimer_dump(), and DTVdriftWorkflow.DTvdriftWorkflow.prepare_segment_dump().

230 
231  def crab_config_filepath(self):
232  base_path = os.path.join( self.options.working_dir,self.local_path)
233  return os.path.join( base_path, self.crab_config_filename)
def CrabHelper.CrabHelper.crab_taskname (   self)

Definition at line 235 of file CrabHelper.py.

References cmsswPreprocessor.CmsswPreprocessor.options, DTCalibrationWorker.DTCalibrationWorker.options, DTWorkflow.DTWorkflow.options, TestProcess.TestProcess.options, confdb.HLTProcess.options, VisualizationOptions.options(), production_tasks.Task.options, fireworks::OptionNode.options(), edmIntegrityCheck.IntegrityCheck.options, Config.Process.options, validateAlignments.ValidationJobMultiIOV.options, and str.

Referenced by CrabHelper.CrabHelper.crab_config_filename().

235 
236  def crab_taskname(self):
237  taskname = self.options.label + "_" + self.options.workflow + "_"
238  if hasattr( self.options, "workflow_mode"):
239  taskname+= self.options.workflow_mode + "_"
240  taskname += "run_" + str(self.options.run) + "_v" + str(self.options.trial)
241  return taskname
str
#define str(s)
Definition: TestProcessor.cc:54
def CrabHelper.CrabHelper.create_crab_config (   self) 
Create a crab config object dependent on the chosen command option

Definition at line 120 of file CrabHelper.py.

Referenced by CrabHelper.CrabHelper.submit_crab_task().

120 
121  def create_crab_config(self):
122  """ Create a crab config object dependent on the chosen command option"""
123  from CalibMuon.DTCalibration.Workflow.Crabtools.crabConfigParser import CrabConfigParser
124  self.crab_config = CrabConfigParser()
125  """ Fill common options in crab config """
126  ### General section
127  self.crab_config.add_section('General')
128  if "/" in self.crab_taskname:
129  raise ValueError( 'Sample contains "/" which is not allowed' )
130  self.crab_config.set( 'General', 'requestName', self.crab_taskname )
131  self.crab_config.set( 'General', 'workArea', self.local_path)
132  if self.options.no_log:
133  self.crab_config.set( 'General', 'transferLogs', 'False' )
134  else:
135  self.crab_config.set( 'General', 'transferLogs', 'True' )
136  ### JobType section
137  self.crab_config.add_section('JobType')
138  self.crab_config.set( 'JobType', 'pluginName', 'Analysis' )
139  self.crab_config.set( 'JobType', 'psetName', self.pset_path )
140  self.crab_config.set( 'JobType', 'outputFiles', self.output_files)
141  if self.input_files:
142  self.crab_config.set( 'JobType', 'inputFiles', self.input_files)
143  ### Data section
144  self.crab_config.add_section('Data')
145  self.crab_config.set('Data', 'inputDataset', self.options.datasetpath)
146  # set job splitting options
147  if self.options.datasettype =="MC":
148  self.crab_config.set('Data', 'splitting', 'FileBased')
149  self.crab_config.set('Data', 'unitsPerJob', str(self.options.filesPerJob) )
150  else:
151  self.crab_config.set('Data', 'splitting', 'LumiBased')
152  self.crab_config.set('Data', 'unitsPerJob', str(self.options.lumisPerJob) )
153  if self.options.runselection:
154  self.crab_config.set( "Data",
155  "runRange",
156  ",".join( self.options.runselection )
157  )
158  # set output path in compliance with crab3 structure
159  self.crab_config.set('Data', 'publication', False)
160  self.crab_config.set('Data', 'outputDatasetTag', self.remote_out_path["outputDatasetTag"])
161  self.crab_config.set('Data', 'outLFNDirBase', self.remote_out_path["outLFNDirBase"] )
162 
163  # set site section options
164  self.crab_config.add_section('Site')
165  self.crab_config.set('Site', 'storageSite', self.options.output_site)
166  self.crab_config.set('Site', 'whitelist', self.options.ce_white_list)
167  self.crab_config.set('Site', 'blacklist', self.options.ce_black_list)
168 
169  #set user section options if necessary
170 # if self.cert_info.voGroup or self.cert_info.voRole:
171 # self.crab_config.add_section('User')
172 # if self.cert_info.voGroup:
173 # self.crab_config.set('User', "voGroup", self.cert_info.voGroup)
174 # if self.cert_info.voRole:
175 # self.crab_config.set('User', "voRole", self.cert_info.voRole)
176  log.debug("Created crab config: %s " % self.crab_config_filename)
CrabConfigParser
This module extends the python configparser to create crab3 config files.
join
static std::string join(char **cmd)
Definition: RemoteFile.cc:19
str
#define str(s)
Definition: TestProcessor.cc:54
def CrabHelper.CrabHelper.fill_options_from_crab_config (   self)

Definition at line 189 of file CrabHelper.py.

References CrabHelper.CrabHelper.crab_config_filename(), cmsswPreprocessor.CmsswPreprocessor.options, DTCalibrationWorker.DTCalibrationWorker.options, DTWorkflow.DTWorkflow.options, TestProcess.TestProcess.options, confdb.HLTProcess.options, VisualizationOptions.options(), production_tasks.Task.options, fireworks::OptionNode.options(), edmIntegrityCheck.IntegrityCheck.options, Config.Process.options, validateAlignments.ValidationJobMultiIOV.options, and submitPVValidationJobs.split().

189 
190  def fill_options_from_crab_config(self):
191  crabtask = CrabTask( crab_config = self.crab_config_filename )
192  splitinfo = crabtask.crabConfig.Data.outputDatasetTag.split("_")
193  run, trial = splitinfo[0].split("Run")[-1], splitinfo[1].split("v")[-1]
194  if not self.options.run:
195  self.options.run = int(run)
196  self.options.trail = int(trial)
197  if not hasattr(self.options, "datasetpath"):
198  self.options.datasetpath = crabtask.crabConfig.Data.inputDataset
199  if not hasattr(self.options, "label"):
200  self.options.label = crabtask.crabConfig.General.requestName.split("_")[0]
def CrabHelper.CrabHelper.submit_crab_task (   self)

Definition at line 22 of file CrabHelper.py.

References CrabHelper.CrabHelper.create_crab_config(), and CrabHelper.CrabHelper.write_crabConfig().

Referenced by DTWorkflow.DTWorkflow.submit().

22 
23  def submit_crab_task(self):
24  # create a crab config
25  log.info("Creating crab config")
26  self.create_crab_config()
27  #write crab config
28  full_crab_config_filename = self.write_crabConfig()
29  if self.options.no_exec:
30  log.info("Runing with option no-exec exiting")
31  return True
32  #submit crab job
33  log.info("Submitting crab job")
34  self.crab.submit(full_crab_config_filename)
35  log.info("crab job submitted. Waiting 120 seconds before first status call")
36  time.sleep( 120 )
37 
38  task = self.crabFunctions.CrabTask(crab_config = full_crab_config_filename)
39  task.update()
40  if task.state =="UNKNOWN":
41  time.sleep( 30 )
42  task.update()
43  success_states = ( 'QUEUED', 'SUBMITTED', "COMPLETED", "FINISHED")
44  if task.state in success_states:
45  log.info("Job in state %s" % task.state )
46  return True
47  else:
48  log.error("Job submission not successful, crab state:%s" % task.state)
49  raise RuntimeError("Job submission not successful, crab state:%s" % task.state)
def CrabHelper.CrabHelper.voms_proxy_create (   self,
  passphrase = None 
)

Definition at line 105 of file CrabHelper.py.

Referenced by CrabHelper.CrabHelper.cert_info().

105 
106  def voms_proxy_create(self, passphrase = None):
107  voms = 'cms'
108  if passphrase:
109  p = subprocess.Popen(['voms-proxy-init', '--voms', voms, '--valid', '192:00'],
110  stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
111  stdout = p.communicate(input=passphrase+'\n')[0]
112  retcode = p.returncode
113  if not retcode == 0:
114  raise ProxyError('Proxy initialization command failed: %s'%stdout)
115  else:
116  retcode = subprocess.call(['voms-proxy-init', '--voms', voms, '--valid', '192:00'])
117  if not retcode == 0:
118  raise ProxyError('Proxy initialization command failed.')
119 
CrabHelper.ProxyError
Exception for the VOMS proxy.
Definition: CrabHelper.py:243
def CrabHelper.CrabHelper.voms_proxy_time_left (   self)

Definition at line 95 of file CrabHelper.py.

Referenced by CrabHelper.CrabHelper.cert_info().

95 
96  def voms_proxy_time_left(self):
97  process = subprocess.Popen(['voms-proxy-info', '-timeleft'],
98  stdout=subprocess.PIPE,
99  stderr=subprocess.STDOUT)
100  stdout = process.communicate()[0]
101  if process.returncode != 0:
102  return 0
103  else:
104  return int(stdout)
def CrabHelper.CrabHelper.write_crabConfig (   self) 
Write crab config file in working dir with label option as name

Definition at line 177 of file CrabHelper.py.

References CrabHelper.CrabHelper.crab_config_filename(), and DTWorkflow.DTWorkflow.local_path().

Referenced by CrabHelper.CrabHelper.submit_crab_task().

177 
178  def write_crabConfig(self):
179  """ Write crab config file in working dir with label option as name """
180  base_path = os.path.join( self.options.working_dir,self.local_path)
181  filename = os.path.join( base_path, self.crab_config_filename)
182  if not os.path.exists(base_path):
183  os.makedirs(base_path)
184  if os.path.exists(filename):
185  raise IOError("file %s alrady exits"%(filename))
186  self.crab_config.writeCrabConfig(filename)
187  log.info( 'created crab config file %s'%filename )
188  return filename

Member Data Documentation

CrabHelper.CrabHelper._cert_info
private

Definition at line 20 of file CrabHelper.py.

Referenced by CrabHelper.CrabHelper.cert_info().

CrabHelper.CrabHelper._crab
private

Definition at line 19 of file CrabHelper.py.

Referenced by CrabHelper.CrabHelper.crab().

CrabHelper.CrabHelper.crab_config

Definition at line 123 of file CrabHelper.py.

CrabHelper.CrabHelper.crabFunctions

Definition at line 17 of file CrabHelper.py.

Generated for CMSSW Reference Manual by doxygen 1.8.5