CMS 3D CMS Logo
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
List of all members | Public Member Functions | Public Attributes | Private Member Functions
crabFunctions.CrabController Class Reference

The CrabController class. More...

Public Member Functions

def __init__
 The constructor. More...
 
def callCrabCommand
 Call crab command in a new process and return result dict. More...
 
def checkusername
 Returns the hn name for a user with valid proxy. More...
 
def checkwrite
 Check if crab can write to specified site. More...
 
def commandlineOptions
 Populates an existing optparse parser or returns a new one with options for crab functions. More...
 
def crabFolders
 Return list of all crab folders in workin area (default cwd) More...
 
def getlog
 Call crab getlog. More...
 
def readCrabConfig
 Read a crab config and return python object. More...
 
def report
 Call crab report command and return path to lumiSummary. More...
 
def resubmit
 Resubmit all failed tasks in job or specified list of jobs in task. More...
 
def status
 Check crab status. More...
 
def submit
 Check if crab can write to specified site. More...
 

Public Attributes

 crab_q
 
 debug
 
 dry_run
 
 logger
 
 username
 
 voGroup
 
 workingArea
 

Private Member Functions

def _prepareFoldername
 Add crab_ to Foldername if needed. More...
 

Detailed Description

The CrabController class.

This class can be used to manage Analyses using crab3

Definition at line 31 of file crabFunctions.py.

Constructor & Destructor Documentation

def crabFunctions.CrabController.__init__ (   self,
  debug = 0,
  logger = None,
  workingArea = None,
  voGroup = None,
  username = None 
)

The constructor.

Parameters
self,:The object pointer.
self,:A previously defined logger. Crab log messages will use this logger as their parent logger.

Definition at line 36 of file crabFunctions.py.

36 
37  def __init__(self, debug=0, logger = None , workingArea = None, voGroup = None, username = None):
38 
39  setConsoleLogLevel(LOGLEVEL_MUTE)
40  self.debug = debug
41  if workingArea is not None:
42  self.workingArea = workingArea
43  else:
44  self.workingArea = os.getcwd()
45  self.dry_run = False
46  if voGroup is not None:
47  self.voGroup = voGroup
48  else:
49  self.voGroup = "dcms"
50  if username is not None:
51  self.username = username
52  else:
53  self.username = None
54 
55  if logger is not None:
56  self.logger = logger.getChild("CrabController")
57  else:
58  # add instance logger as logger to root
59  self.logger = logging.getLogger("CrabController")
60  # check if handlers are present for root logger
61  # we assume that the default logging is not configured
62  # if handler is present
63  if len(logging.getLogger().handlers) < 1 :
64  ch = logging.FileHandler('crabController.log', mode='a', encoding=None, delay=False)
65  ch.setLevel(logging.DEBUG)
66  # create formatter
67  formatter = logging.Formatter( '%(asctime)s - %(name)s - %(levelname)s - %(message)s' )
68  # add formatter to ch
69  ch.setFormatter(formatter)
70  self.logger.addHandler(ch)
71 
self.crab_q = Queue()
crabFunctions.CrabController.__init__
def __init__
The constructor.
Definition: crabFunctions.py:36

Member Function Documentation

def crabFunctions.CrabController._prepareFoldername (   self,
  name 
)
private

Add crab_ to Foldername if needed.

Parameters
getlog(self,name)

Definition at line 255 of file crabFunctions.py.

Referenced by crabFunctions.CrabController.getlog(), crabFunctions.CrabController.report(), and crabFunctions.CrabController.resubmit().

255 
256  def _prepareFoldername(self, name):
257  if name.startswith("crab_"):
258  crabfolder = '%s'%name
259  else:
260  crabfolder = "crab_%s "%name
return crabfolder.strip()
crabFunctions.CrabController._prepareFoldername
def _prepareFoldername
Add crab_ to Foldername if needed.
Definition: crabFunctions.py:255
def crabFunctions.CrabController.callCrabCommand (   self,
  crabArgs 
)

Call crab command in a new process and return result dict.

Parameters
selfThe object pointer
crabArgsA list of arguments for crab beginning with the command

Definition at line 189 of file crabFunctions.py.

References crabFunctions.CrabController.crab_q.

Referenced by crabFunctions.CrabController.getlog(), crabFunctions.CrabController.report(), crabFunctions.CrabController.resubmit(), crabFunctions.CrabController.status(), and crabFunctions.CrabController.submit().

189 
190  def callCrabCommand( self, crabArgs ):
191  crabCommandProcessArgs = (self.crab_q, crabArgs)
192  p = Process(target=crabCommandProcess, args=(crabCommandProcessArgs))
193  p.start()
194  res = self.crab_q.get()
195  p.join()
196  return res
crabFunctions.CrabController.callCrabCommand
def callCrabCommand
Call crab command in a new process and return result dict.
Definition: crabFunctions.py:189
def crabFunctions.CrabController.checkusername (   self)

Returns the hn name for a user with valid proxy.

self: CrabController

Parameters
self,:The object pointer.
Returns
users hypernews name

Definition at line 141 of file crabFunctions.py.

References crabFunctions.CrabController.username.

Referenced by crabFunctions.CrabController.checkwrite().

141 
142  def checkusername(self):
143  #depreceated string: cmd = 'crab checkHNname --voGroup=dcms'
144  #~ cmd = 'crab checkusername --voGroup=dcms'
145  try:
146  username = os.environ["CERNUSERNAME"]
147  return username
148  except:pass
149  res = crabCommand('checkusername')
150  try:
151  self.username = res['username']
152  return res['username']
153  except:
154  return "noHNname"
crabFunctions.CrabController.checkusername
def checkusername
Returns the hn name for a user with valid proxy.
Definition: crabFunctions.py:141
def crabFunctions.CrabController.checkwrite (   self,
  site = 'T2_DE_RWTH',
  path = 'noPath' 
)

Check if crab can write to specified site.

self: CrabController

Parameters
self,:The object pointer. site string
siteThe Site symbol [default:T2_DE_RWTH] path string
pathlfn path to check write permission in. see twiki WorkBookCRAB3Tutorial
Returns
boolean which is True if user can write to site and False otherwise

Definition at line 81 of file crabFunctions.py.

References crabFunctions.CrabController.checkusername(), crabFunctions.CrabController.username, and crabFunctions.CrabController.voGroup.

81 
82  def checkwrite(self,site='T2_DE_RWTH',path='noPath'):
83  if self.username is None: self.checkusername()
84  try:
85  self.logger.info( "Checking if user can write to /store/user/%s on site %s with voGroup %s"%(self.username,site , self.voGroup) )
86  if not 'noPath' in path:
87  res = crabCommand('checkwrite','--site',site,'--voGroup',self.voGroup,'--lfn', path)
88  else:
89  res = crabCommand('checkwrite','--site',site,'--voGroup',self.voGroup)
90  if res['status'] == 'SUCCESS':
91  self.logger.info("Checkwrite was sucessfully called.")
92  return True
93  else:
94  self.logger.error( "The crab checkwrite command failed for site: %s"%site )
95  return False
96  except:
97  self.logger.error( 'Unable to perform crab checkwrite')
98  return False
crabFunctions.CrabController.checkusername
def checkusername
Returns the hn name for a user with valid proxy.
Definition: crabFunctions.py:141
crabFunctions.CrabController.checkwrite
def checkwrite
Check if crab can write to specified site.
Definition: crabFunctions.py:81
def crabFunctions.CrabController.commandlineOptions (   self,
  parser = optparse.OptionParser( 'usage: %prog' ) 
)

Populates an existing optparse parser or returns a new one with options for crab functions.

This functions populates a previously created (or new) instance of a optparse parser object with options needed by crab functions. It is possible to add three kinds of options:

Definition at line 272 of file crabFunctions.py.

272 
273  def commandlineOptions(self,parser = optparse.OptionParser( 'usage: %prog' )):
274  # we first need to call parse_args with a dummy string at the beginning to
275  # check for existing options later
276  (currentoptions, args ) = parser.parse_args([" "])
277 
278  # The following block shows how variables should be added, where
279  # conflicts are possible if the option is overridden by this function
280  # they raise a value error
281  #try:
282  # parser.add_option( '--someOption',metavar='DIR',default=None,
283  # help='Dummy option for future integration')
284  #except OptionConflictError as e:
285  # conditionalLog(crablog,"There are conflicts extending the optparse options object",'error')
286  # conditionalLog(crablog,e.strerror,'error')
287 
288  # options where it is checked if they exists and new options are added
289  # otherwise
290  if not hasattr(currentoptions, 'dry_run'):
291  parser.add_option( '--dry-run', action='store_true', default=False,
292  help='Do everything except calling CRAB or registering samples to the database.' )
293  if not hasattr(currentoptions, 'workingArea'):
294  parser.add_option( '--workingArea',metavar='DIR',default=os.getcwd(),help='The area (full or relative path) where to create the CRAB project directory. '
295  'If the area doesn\'t exist, CRAB will try to create it using the mkdir command' \
296  ' (without -p option). Defaults to the current working directory.' )
297 
298 
299  # Some options can be added without expected trouble with other parser
300  # parts, simply because it is quite fixed what they represent.
301  # those variables should be added here and will throw no exception if
302  # they already exist in the parser
303  #parser.set_conflict_handler('resolve')
304  #parser.add_option( '--someOption',metavar='DIR',default=None,
305  # help='Dummy option for future integration')
306 
307  return parser
308 
309 
crabFunctions.CrabController.commandlineOptions
def commandlineOptions
Populates an existing optparse parser or returns a new one with options for crab functions.
Definition: crabFunctions.py:272
def crabFunctions.CrabController.crabFolders (   self)

Return list of all crab folders in workin area (default cwd)

Parameters
selfThe object pointer

Definition at line 247 of file crabFunctions.py.

References crabFunctions.CrabController.workingArea.

247 
248  def crabFolders(self):
249  results = []
250  dirlist = [ x for x in os.listdir( self.workingArea ) if (x.startswith('crab_') and os.path.isdir( os.path.join(self.workingArea,x) ) )]
251  return dirlist
crabFunctions.CrabController.crabFolders
def crabFolders
Return list of all crab folders in workin area (default cwd)
Definition: crabFunctions.py:247
def crabFunctions.CrabController.getlog (   self,
  name 
)

Call crab getlog.

Parameters
self,:The object pointer. name string

Definition at line 201 of file crabFunctions.py.

References crabFunctions.CrabController._prepareFoldername(), and crabFunctions.CrabController.callCrabCommand().

201 
202  def getlog(self, name):
203  foldername = self._prepareFoldername( name)
204  try:
205  #res = crabCommand('--quiet','status', dir = 'crab_%s' % name)
206  res = self.callCrabCommand( ('getlog', '%s' % foldername) )
207  return res['success'], res['failed']
208  except:
209  self.logger.error("Error calling crab getlog for %s" %foldername)
210  return {}, {}
crabFunctions.CrabController.getlog
def getlog
Call crab getlog.
Definition: crabFunctions.py:201
crabFunctions.CrabController.callCrabCommand
def callCrabCommand
Call crab command in a new process and return result dict.
Definition: crabFunctions.py:189
crabFunctions.CrabController._prepareFoldername
def _prepareFoldername
Add crab_ to Foldername if needed.
Definition: crabFunctions.py:255
def crabFunctions.CrabController.readCrabConfig (   self,
  name 
)

Read a crab config and return python object.

Parameters
self,:The object pointer.
nameThe sample name (crab request name)

Definition at line 228 of file crabFunctions.py.

228 
229  def readCrabConfig( self, name ):
230  try:
231  if os.path.exists(name):
232  pset = name
233  else:
234  pset = 'crab_%s_cfg.py' % name
235  with open( pset, 'r') as cfgfile:
236  cfo = imp.load_source("pycfg", pset, cfgfile )
237  config = cfo.config
238  del cfo
239  return config
240  except:
241  return False
crabFunctions.CrabController.readCrabConfig
def readCrabConfig
Read a crab config and return python object.
Definition: crabFunctions.py:228
def crabFunctions.CrabController.report (   self,
  name 
)

Call crab report command and return path to lumiSummary.

Parameters
selfThe object pointer
nameThe crab3 request name, a.k.a the sample name

Definition at line 215 of file crabFunctions.py.

References crabFunctions.CrabController._prepareFoldername(), and crabFunctions.CrabController.callCrabCommand().

215 
216  def report(self, name):
217  foldername = self._prepareFoldername( name)
218  try:
219  res = self.callCrabCommand( ('report', '%s' % foldername) )
220  return res['analyzedLumis']
221  except:
222  self.logger.error("Error calling crab report for %s" %foldername)
223 
crabFunctions.CrabController.callCrabCommand
def callCrabCommand
Call crab command in a new process and return result dict.
Definition: crabFunctions.py:189
crabFunctions.CrabController._prepareFoldername
def _prepareFoldername
Add crab_ to Foldername if needed.
Definition: crabFunctions.py:255
crabFunctions.CrabController.report
def report
Call crab report command and return path to lumiSummary.
Definition: crabFunctions.py:215
def crabFunctions.CrabController.resubmit (   self,
  name,
  joblist = None 
)

Resubmit all failed tasks in job or specified list of jobs in task.

self: CrabController

Parameters
self,:The object pointer. name string
nameThe crab3 request name, a.k.a the sample name joblist list of strings
joblistThe crab3 request name, a.k.a the sample name

Definition at line 122 of file crabFunctions.py.

References crabFunctions.CrabController._prepareFoldername(), crabFunctions.CrabController.callCrabCommand(), crabFunctions.CrabController.dry_run, and crabFunctions.CrabController.workingArea.

122 
123  def resubmit(self,name,joblist = None):
124  if self.dry_run:
125  self.logger.info('Dry-run: Created config file. ')
126  return {}
127  #~ if joblist is not None:
128  #~ jobstring ="%s"%','.join(joblist)
129  #~ cmd = ('resubmit','--wait', '--jobids=',jobstring, os.path.join(self.workingArea,self._prepareFoldername(name)) )
130  if False:
131  pass
132  else:
133  cmd = ('resubmit','--wait', os.path.join(self.workingArea,self._prepareFoldername(name)) )
134  res = self.callCrabCommand( cmd )
135  self.logger.info("crab resumbit called for task %s"%name)
return res
crabFunctions.CrabController.resubmit
def resubmit
Resubmit all failed tasks in job or specified list of jobs in task.
Definition: crabFunctions.py:122
crabFunctions.CrabController.callCrabCommand
def callCrabCommand
Call crab command in a new process and return result dict.
Definition: crabFunctions.py:189
crabFunctions.CrabController._prepareFoldername
def _prepareFoldername
Add crab_ to Foldername if needed.
Definition: crabFunctions.py:255
def crabFunctions.CrabController.status (   self,
  name 
)

Check crab status.

self: CrabController

Parameters
self,:The object pointer. name string
nameThe crab3 request name, a.k.a the sample name

Definition at line 161 of file crabFunctions.py.

References crabFunctions.CrabController.callCrabCommand(), crabFunctions.CrabController.dry_run, and print().

161 
162  def status(self,name):
163  if self.dry_run:
164  self.logger.info('Dry-run: Created config file. crab command would have been: %s'%cmd)
165  else:
166  try:
167  if not "crab_" in name:
168  callname = "crab_" + name
169  else:
170  callname = name
171  res = self.callCrabCommand( ('status', '--long', callname) )
172  #print res
173  if 'taskFailureMsg' in res and 'jobs' in res:
174  return res['status'], res['jobs'], res['taskFailureMsg']
175  elif 'jobs' in res and 'taskFailureMsg' not in res:
176  return res['status'], res['jobs'],None
177  elif 'jobs' not in res and 'taskFailureMsg' in res:
178  return res['status'], {},res['taskFailureMsg']
179  else:
180  return res['status'],{},None
181  except Exception as e:
182  print(e)
183  self.logger.error("Can not run crab status request")
184  return "NOSTATE",{},None
crabFunctions.CrabController.callCrabCommand
def callCrabCommand
Call crab command in a new process and return result dict.
Definition: crabFunctions.py:189
print
void print(TMatrixD &m, const char *label=nullptr, bool mathematicaFormat=false)
Definition: Utilities.cc:47
crabFunctions.CrabController.status
def status
Check crab status.
Definition: crabFunctions.py:161
def crabFunctions.CrabController.submit (   self,
  name 
)

Check if crab can write to specified site.

Parameters
self,:The object pointer. name string
nameThe crab3 config file name

Definition at line 104 of file crabFunctions.py.

References crabFunctions.CrabController.callCrabCommand(), ALIUtils.debug, crabFunctions.CrabController.debug, DTTTrigCorrectionFirst.debug, DTTPDeadWriter.debug, DTT0CalibrationRMS.debug, DTTTrigWriter.debug, DTNoiseComputation.debug, DTT0Calibration.debug, DTTTrigCalibration.debug, DTTMax.debug, DTVDriftCalibration.debug, crabFunctions.CrabController.dry_run, and str.

104 
105  def submit(self,name):
106  if self.dry_run:
107  res = self.callCrabCommand(('submit', '--dryrun', name))
108  self.logger.info('Dry-run: You may check the created config and sandbox')
109  else:
110  res = self.callCrabCommand(('submit','--wait' , name))
111  self.logger.info("crab sumbit called for task %s"%name)
112  if self.debug > 1:
113  self.logger.info(str(res))
return res
crabFunctions.CrabController.submit
def submit
Check if crab can write to specified site.
Definition: crabFunctions.py:104
crabFunctions.CrabController.callCrabCommand
def callCrabCommand
Call crab command in a new process and return result dict.
Definition: crabFunctions.py:189
str
#define str(s)
Definition: TestProcessor.cc:53

Member Data Documentation

crabFunctions.CrabController.crab_q

Definition at line 71 of file crabFunctions.py.

Referenced by crabFunctions.CrabController.callCrabCommand().

crabFunctions.CrabController.debug

Definition at line 39 of file crabFunctions.py.

Referenced by util.rrapi.RRApi.dprint(), rrapi.RRApi.dprint(), pkg.AbstractPkg.generate(), rrapi.RRApi.get(), util.rrapi.RRApi.get(), pkg.AbstractPkg.get_kwds(), runTauIdMVA.TauIDEmbedder.loadMVA_WPs_run2_2017(), runTauIdMVA.TauIDEmbedder.runTauID(), crabFunctions.CrabController.submit(), and pkg.AbstractPkg.write().

crabFunctions.CrabController.dry_run

Definition at line 44 of file crabFunctions.py.

Referenced by crabFunctions.CrabController.resubmit(), crabFunctions.CrabController.status(), and crabFunctions.CrabController.submit().

crabFunctions.CrabController.logger

Definition at line 55 of file crabFunctions.py.

Referenced by o2olib.O2OJobMgr.runManager().

crabFunctions.CrabController.username

Definition at line 50 of file crabFunctions.py.

Referenced by crabFunctions.CrabController.checkusername(), and crabFunctions.CrabController.checkwrite().

crabFunctions.CrabController.voGroup

Definition at line 46 of file crabFunctions.py.

Referenced by crabFunctions.CrabController.checkwrite().

crabFunctions.CrabController.workingArea

Definition at line 41 of file crabFunctions.py.

Referenced by crabFunctions.CrabController.crabFolders(), and crabFunctions.CrabController.resubmit().

Generated for CMSSW Reference Manual by doxygen 1.8.5