CMS 3D CMS Logo

 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Pages
Classes | Functions | Variables
upload_popcon Namespace Reference

Classes

class  CERNSSOError
 
class  DropBox
 
class  HTTP
 
class  HTTPError
 

Functions

def _getCERNSSOCookies
 
def addToTarFile
 
def getInput
 
def getInputChoose
 
def getInputRepeat
 
def getInputWorkflow
 

Variables

string __author__ = 'Miguel Ojeda'
 
string __copyright__ = 'Copyright 2012, CERN CMS'
 
list __credits__ = ['Giacomo Govi', 'Salvatore Di Guida', 'Miguel Ojeda', 'Andreas Pfeiffer']
 
string __email__ = 'mojedasa@cern.ch'
 
string __license__ = 'Unknown'
 
string __maintainer__ = 'Miguel Ojeda'
 
int __version__ = 6
 
string defaultBackend = 'online'
 
string defaultHostname = 'cms-conddb-prod.cern.ch'
 
string defaultNetrcHost = 'DropBox'
 
string defaultTemporaryFile = 'upload.tar.bz2'
 
string defaultUrlTemplate = 'https://%s/dropBox/'
 
string defaultWorkflow = 'offline'
 

Function Documentation

def upload_popcon._getCERNSSOCookies (   url,
  secureTarget = True,
  secure = True 
)
private
Returns the required CERN SSO cookies for a URL using Kerberos.

They can be used with any HTTP client (libcurl, wget, urllib...).

If you wish to make secure SSL connections to the CERN SSO
(i.e. verify peers/hosts), you may need to install the CERN-CA-certs package.
Use secure == False to skip this (i.e. this is the same as curl
-k/--insecure). Not recommended: tell users to install them or use lxplus6.

The same way, if you have a self-signed certificate in your target URL
you can use secureTarget == False as well. Note that this option
is provided in order to be able to use a secure SSL connection to CERN SSO,
even if the connection to your target URL is not secure. Note that
you will probably need the CERN-CA-certs package after you get a certificate
signed by the CERN CA (https://cern.ch/ca), even if you did not need it
for the CERN SSO.

Note that this method *does* a query to the given URL if successful.

This was implemented outside the HTTP class for two main reasons:

    * The only thing needed to use CERN SSO is the cookies, therefore
      this function is useful alone as well (e.g. as a simple replacement
      of the cern-get-sso-cookie script or as a Python port of
      the WWW::CERNSSO::Auth Perl package -- this one does not write
      any file and can be used in-memory, by the way).

    * We do not need to use the curl handler of the HTTP class.
      This way we do not overwrite any options in that one and we use
      only a temporary one here for getting the cookie.

TODO: Support also Certificate/Key authentication.
TODO: Support also Username/Password authentication.
TODO: Review the error paths.
TODO: Why PERLSESSID was used in the original code?
TODO: Retry if timeouts are really common (?)

Definition at line 53 of file upload_popcon.py.

Referenced by upload_popcon.HTTP.addCERNSSOCookies().

53 
54 def _getCERNSSOCookies(url, secureTarget = True, secure = True):
55  '''Returns the required CERN SSO cookies for a URL using Kerberos.
56 
57  They can be used with any HTTP client (libcurl, wget, urllib...).
58 
59  If you wish to make secure SSL connections to the CERN SSO
60  (i.e. verify peers/hosts), you may need to install the CERN-CA-certs package.
61  Use secure == False to skip this (i.e. this is the same as curl
62  -k/--insecure). Not recommended: tell users to install them or use lxplus6.
63 
64  The same way, if you have a self-signed certificate in your target URL
65  you can use secureTarget == False as well. Note that this option
66  is provided in order to be able to use a secure SSL connection to CERN SSO,
67  even if the connection to your target URL is not secure. Note that
68  you will probably need the CERN-CA-certs package after you get a certificate
69  signed by the CERN CA (https://cern.ch/ca), even if you did not need it
70  for the CERN SSO.
71 
72  Note that this method *does* a query to the given URL if successful.
73 
74  This was implemented outside the HTTP class for two main reasons:
75 
76  * The only thing needed to use CERN SSO is the cookies, therefore
77  this function is useful alone as well (e.g. as a simple replacement
78  of the cern-get-sso-cookie script or as a Python port of
79  the WWW::CERNSSO::Auth Perl package -- this one does not write
80  any file and can be used in-memory, by the way).
81 
82  * We do not need to use the curl handler of the HTTP class.
83  This way we do not overwrite any options in that one and we use
84  only a temporary one here for getting the cookie.
85 
86  TODO: Support also Certificate/Key authentication.
87  TODO: Support also Username/Password authentication.
88  TODO: Review the error paths.
89  TODO: Why PERLSESSID was used in the original code?
90  TODO: Retry if timeouts are really common (?)
91  '''
92 
93  def perform():
94  response = cStringIO.StringIO()
95  curl.setopt(curl.WRITEFUNCTION, response.write)
96  curl.perform()
97  code = curl.getinfo(curl.RESPONSE_CODE)
98  response = response.getvalue()
99  effectiveUrl = curl.getinfo(curl.EFFECTIVE_URL)
100  return (code, response, effectiveUrl)
101 
102  # These constants and the original code came from the official CERN
103  # cern-get-sso-cookie script and WWW::CERNSSO::Auth Perl package.
104  VERSION = '0.4.2'
105  CERN_SSO_CURL_USER_AGENT_KRB = 'curl-sso-kerberos/%s' % VERSION
106  CERN_SSO_CURL_AUTHERR = 'HTTP Error 401.2 - Unauthorized'
107  CERN_SSO_CURL_ADFS_EP = '/adfs/ls/auth'
108  CERN_SSO_CURL_ADFS_SIGNIN = 'wa=wsignin1.0'
109  CERN_SSO_CURL_CAPATH = '/etc/pki/tls/certs'
110 
111  logging.debug('secureTarget = %s', secureTarget)
112  logging.debug('secure = %s', secure)
113 
114  curl = pycurl.Curl()
115 
116  # Store the cookies in memory, which we will retreive later on
117  curl.setopt(curl.COOKIEFILE, '')
118 
119  # This should not be needed, but sometimes requests hang 'forever'
120  curl.setopt(curl.TIMEOUT, 10)
121  curl.setopt(curl.CONNECTTIMEOUT, 10)
122 
123  # Ask curl to use Kerberos5 authentication
124  curl.setopt(curl.USERAGENT, CERN_SSO_CURL_USER_AGENT_KRB)
125  curl.setopt(curl.HTTPAUTH, curl.HTTPAUTH_GSSNEGOTIATE)
126  curl.setopt(curl.USERPWD, ':')
127 
128  # Follow location (and send the password along to other hosts,
129  # although we do not really send any password)
130  curl.setopt(curl.FOLLOWLOCATION, 1)
131  curl.setopt(curl.UNRESTRICTED_AUTH, 1)
132 
133  # We do not need the headers
134  curl.setopt(curl.HEADER, 0)
135 
136  # The target server has a valid certificate
137  if secureTarget:
138  curl.setopt(curl.SSL_VERIFYPEER, 1)
139  curl.setopt(curl.SSL_VERIFYHOST, 2)
140  curl.setopt(curl.CAPATH, CERN_SSO_CURL_CAPATH)
141  else:
142  curl.setopt(curl.SSL_VERIFYPEER, 0)
143  curl.setopt(curl.SSL_VERIFYHOST, 0)
144 
145  # Fetch the url
146  logging.debug('Connecting to %s', url)
147  curl.setopt(curl.URL, url)
148  (code, response, effectiveUrl) = perform()
149 
150  if CERN_SSO_CURL_ADFS_EP not in effectiveUrl:
151  raise CERNSSOError('Not behind SSO or we already have the cookie.')
152 
153  # Do the manual redirection to the IDP
154  logging.debug('Redirected to IDP %s', effectiveUrl)
155 
156  # The CERN SSO servers have a valid certificate
157  if secure:
158  curl.setopt(curl.SSL_VERIFYPEER, 1)
159  curl.setopt(curl.SSL_VERIFYHOST, 2)
160  curl.setopt(curl.CAPATH, CERN_SSO_CURL_CAPATH)
161  else:
162  curl.setopt(curl.SSL_VERIFYPEER, 0)
163  curl.setopt(curl.SSL_VERIFYHOST, 0)
164 
165  curl.setopt(curl.URL, effectiveUrl)
166  (code, response, effectiveUrl) = perform()
167 
168  if CERN_SSO_CURL_AUTHERR in response:
169  raise CERNSSOError('Authentication error: Redirected to IDP Authentication error %s' % effectiveUrl)
170 
171  match = re.search('form .+?action="([^"]+)"', response)
172  if not match:
173  raise CERNSSOError('Something went wrong: could not find the expected redirection form (do you have a valid Kerberos ticket? -- see klist and kinit).')
174 
175  # Do the JavaScript redirection via the form to the SP
176  spUrl = match.groups()[0]
177  logging.debug('Redirected (via form) to SP %s', spUrl)
178 
179  formPairs = re.findall('input type="hidden" name="([^"]+)" value="([^"]+)"', response)
180 
181  # Microsoft ADFS produces broken encoding in auth forms:
182  # '<' and '"' are encoded as '&lt;' and '&quot;' however
183  # '>' is *not* encoded. Does not matter here though, we just decode.
184  htmlParser = HTMLParser.HTMLParser()
185  formPairs = [(x[0], htmlParser.unescape(x[1])) for x in formPairs]
186 
187  # The target server has a valid certificate
188  if secureTarget:
189  curl.setopt(curl.SSL_VERIFYPEER, 1)
190  curl.setopt(curl.SSL_VERIFYHOST, 2)
191  curl.setopt(curl.CAPATH, CERN_SSO_CURL_CAPATH)
192  else:
193  curl.setopt(curl.SSL_VERIFYPEER, 0)
194  curl.setopt(curl.SSL_VERIFYHOST, 0)
195 
196  curl.setopt(curl.URL, spUrl)
197  curl.setopt(curl.POSTFIELDS, urllib.urlencode(formPairs))
198  curl.setopt(curl.POST, 1)
199  (code, response, effectiveUrl) = perform()
200 
201  if CERN_SSO_CURL_ADFS_SIGNIN in effectiveUrl:
202  raise CERNSSOError('Something went wrong: still on the auth page.')
203 
204  # Return the cookies
205  return curl.getinfo(curl.INFO_COOKIELIST)
206 
def _getCERNSSOCookies
def upload_popcon.addToTarFile (   tarFile,
  fileobj,
  arcname 
)

Definition at line 403 of file upload_popcon.py.

Referenced by upload_popcon.DropBox.uploadFile().

404 def addToTarFile(tarFile, fileobj, arcname):
405  tarInfo = tarFile.gettarinfo(fileobj = fileobj, arcname = arcname)
406  tarInfo.mode = 0o400
407  tarInfo.uid = tarInfo.gid = tarInfo.mtime = 0
408  tarInfo.uname = tarInfo.gname = 'root'
409  tarFile.addfile(tarInfo, fileobj)
410 
def upload_popcon.getInput (   default,
  prompt = '' 
)
Like raw_input() but with a default and automatic strip().

Definition at line 551 of file upload_popcon.py.

Referenced by getInputChoose(), and getInputWorkflow().

552 def getInput(default, prompt = ''):
553  '''Like raw_input() but with a default and automatic strip().
554  '''
555 
556  answer = raw_input(prompt)
557  if answer:
558  return answer.strip()
559 
560  return default.strip()
561 
def upload_popcon.getInputChoose (   optionsList,
  default,
  prompt = '' 
)
Makes the user choose from a list of options.

Definition at line 575 of file upload_popcon.py.

References getInput().

576 def getInputChoose(optionsList, default, prompt = ''):
577  '''Makes the user choose from a list of options.
578  '''
579 
580  while True:
581  index = getInput(default, prompt)
582 
583  try:
584  return optionsList[int(index)]
585  except ValueError:
586  logging.error('Please specify an index of the list (i.e. integer).')
587  except IndexError:
588  logging.error('The index you provided is not in the given list.')
589 
def upload_popcon.getInputRepeat (   prompt = '')
Like raw_input() but repeats if nothing is provided and automatic strip().

Definition at line 590 of file upload_popcon.py.

591 def getInputRepeat(prompt = ''):
592  '''Like raw_input() but repeats if nothing is provided and automatic strip().
593  '''
594 
595  while True:
596  answer = raw_input(prompt)
597  if answer:
598  return answer.strip()
599 
600  logging.error('You need to provide a value.')
601 
602 
603 
604 
def upload_popcon.getInputWorkflow (   prompt = '')
Like getInput() but tailored to get target workflows (synchronization options).

Definition at line 562 of file upload_popcon.py.

References getInput().

563 def getInputWorkflow(prompt = ''):
564  '''Like getInput() but tailored to get target workflows (synchronization options).
565  '''
566 
567  while True:
568  workflow = getInput(defaultWorkflow, prompt)
569 
570  if workflow in frozenset(['offline', 'hlt', 'express', 'prompt', 'pcl']):
571  return workflow
572 
573  logging.error('Please specify one of the allowed workflows. See above for the explanation on each of them.')
574 

Variable Documentation

string upload_popcon.__author__ = 'Miguel Ojeda'

Definition at line 5 of file upload_popcon.py.

string upload_popcon.__copyright__ = 'Copyright 2012, CERN CMS'

Definition at line 6 of file upload_popcon.py.

list upload_popcon.__credits__ = ['Giacomo Govi', 'Salvatore Di Guida', 'Miguel Ojeda', 'Andreas Pfeiffer']

Definition at line 7 of file upload_popcon.py.

string upload_popcon.__email__ = 'mojedasa@cern.ch'

Definition at line 10 of file upload_popcon.py.

string upload_popcon.__license__ = 'Unknown'

Definition at line 8 of file upload_popcon.py.

string upload_popcon.__maintainer__ = 'Miguel Ojeda'

Definition at line 9 of file upload_popcon.py.

int upload_popcon.__version__ = 6

Definition at line 11 of file upload_popcon.py.

string upload_popcon.defaultBackend = 'online'

Definition at line 28 of file upload_popcon.py.

string upload_popcon.defaultHostname = 'cms-conddb-prod.cern.ch'

Definition at line 29 of file upload_popcon.py.

string upload_popcon.defaultNetrcHost = 'DropBox'

Definition at line 32 of file upload_popcon.py.

string upload_popcon.defaultTemporaryFile = 'upload.tar.bz2'

Definition at line 31 of file upload_popcon.py.

string upload_popcon.defaultUrlTemplate = 'https://%s/dropBox/'

Definition at line 30 of file upload_popcon.py.

string upload_popcon.defaultWorkflow = 'offline'

Definition at line 33 of file upload_popcon.py.