BeautifulSoup::BeautifulSoup Class Reference

Inheritance diagram for BeautifulSoup::BeautifulSoup:

List of all members.

Public Member Functions
def	__init__
def	__init__
def	extractCharsetFromMeta
def	extractCharsetFromMeta
Public Attributes
	declaredHTMLEncoding
	originalEncoding
Static Public Attributes
tuple	CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
list	NESTABLE_BLOCK_TAGS = ['blockquote', 'div', 'fieldset', 'ins', 'del']
list	NESTABLE_INLINE_TAGS
dictionary	NESTABLE_LIST_TAGS
dictionary	NESTABLE_TABLE_TAGS
tuple	NESTABLE_TAGS
list	NON_NESTABLE_BLOCK_TAGS = ['address', 'form', 'p', 'pre']
tuple	PRESERVE_WHITESPACE_TAGS = set(['pre', 'textarea'])
dictionary	QUOTE_TAGS = {'script' : None, 'textarea' : None}
tuple	RESET_NESTING_TAGS
tuple	SELF_CLOSING_TAGS

---

Detailed Description

This parser knows the following facts about HTML:

* Some tags have no closing tag and should be interpreted as being
  closed as soon as they are encountered.

* The text inside some tags (ie. 'script') may contain tags which
  are not really part of the document and which should be parsed
  as text, not tags. If you want to parse the text as tags, you can
  always fetch it and parse it explicitly.

* Tag nesting rules:

  Most tags can't be nested at all. For instance, the occurance of
  a <p> tag should implicitly close the previous <p> tag.

   <p>Para1<p>Para2
    should be transformed into:
   <p>Para1</p><p>Para2

  Some tags can be nested arbitrarily. For instance, the occurance
  of a <blockquote> tag should _not_ implicitly close the previous
  <blockquote> tag.

   Alice said: <blockquote>Bob said: <blockquote>Blah
    should NOT be transformed into:
   Alice said: <blockquote>Bob said: </blockquote><blockquote>Blah

  Some tags can be nested, but the nesting is reset by the
  interposition of other tags. For instance, a <tr> tag should
  implicitly close the previous <tr> tag within the same <table>,
  but not close a <tr> tag in another table.

   <table><tr>Blah<tr>Blah
    should be transformed into:
   <table><tr>Blah</tr><tr>Blah
    but,
   <tr>Blah<table><tr>Blah
    should NOT be transformed into
   <tr>Blah<table></tr><tr>Blah

Differing assumptions about tag nesting rules are a major source
of problems with the BeautifulSoup class. If BeautifulSoup is not
treating as nestable a tag your page author treats as nestable,
try ICantBelieveItsBeautifulSoup, MinimalSoup, or
BeautifulStoneSoup before writing your own subclass.

Definition at line 1447 of file BeautifulSoup.py.

---

Constructor & Destructor Documentation

def BeautifulSoup::BeautifulSoup::__init__	(		self,
			args,
			kwargs
	)

Definition at line 1495 of file BeautifulSoup.py.

                                       :
        if not kwargs.has_key('smartQuotesTo'):
            kwargs['smartQuotesTo'] = self.HTML_ENTITIES
        kwargs['isHTML'] = True
        BeautifulStoneSoup.__init__(self, *args, **kwargs)

def BeautifulSoup::BeautifulSoup::__init__	(		self,
			args,
			kwargs
	)

Definition at line 1495 of file BeautifulSoup.py.

                                       :
        if not kwargs.has_key('smartQuotesTo'):
            kwargs['smartQuotesTo'] = self.HTML_ENTITIES
        kwargs['isHTML'] = True
        BeautifulStoneSoup.__init__(self, *args, **kwargs)

---

Member Function Documentation

def BeautifulSoup::BeautifulSoup::extractCharsetFromMeta	(		self,
			attrs
	)

Beautiful Soup can detect a charset included in a META tag,
try to convert the document to that charset, and re-parse the
document from the beginning.

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1553 of file BeautifulSoup.py.

                                           :
        """Beautiful Soup can detect a charset included in a META tag,
        try to convert the document to that charset, and re-parse the
        document from the beginning."""
        httpsEquiv = None
        contentType = None
        contentTypeIndex = None
        tagNeedsEncodingSubstitution = False

        for i in range(0, len(attrs)):
            key, value = attrs[i]
            key = key.lower()
            if key == 'https-equiv':
                httpsEquiv = value
            elif key == 'content':
                contentType = value
                contentTypeIndex = i

        if httpsEquiv and contentType: # It's an interesting meta tag.
            match = self.CHARSET_RE.search(contentType)
            if match:
                if (self.declaredHTMLEncoding is not None or
                    self.originalEncoding == self.fromEncoding):
                    # An HTML encoding was sniffed while converting
                    # the document to Unicode, or an HTML encoding was
                    # sniffed during a previous pass through the
                    # document, or an encoding was specified
                    # explicitly and it worked. Rewrite the meta tag.
                    def rewrite(match):
                        return match.group(1) + "%SOUP-ENCODING%"
                    newAttr = self.CHARSET_RE.sub(rewrite, contentType)
                    attrs[contentTypeIndex] = (attrs[contentTypeIndex][0],
                                               newAttr)
                    tagNeedsEncodingSubstitution = True
                else:
                    # This is our first pass through the document.
                    # Go through it again with the encoding information.
                    newCharset = match.group(3)
                    if newCharset and newCharset != self.originalEncoding:
                        self.declaredHTMLEncoding = newCharset
                        self._feed(self.declaredHTMLEncoding)
                        raise StopParsing
                    pass
        tag = self.unknown_starttag("meta", attrs)
        if tag and tagNeedsEncodingSubstitution:
            tag.containsSubstitutions = True

def BeautifulSoup::BeautifulSoup::extractCharsetFromMeta	(		self,
			attrs
	)

Beautiful Soup can detect a charset included in a META tag,
try to convert the document to that charset, and re-parse the
document from the beginning.

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1553 of file BeautifulSoup.py.

                                           :
        """Beautiful Soup can detect a charset included in a META tag,
        try to convert the document to that charset, and re-parse the
        document from the beginning."""
        httpsEquiv = None
        contentType = None
        contentTypeIndex = None
        tagNeedsEncodingSubstitution = False

        for i in range(0, len(attrs)):
            key, value = attrs[i]
            key = key.lower()
            if key == 'https-equiv':
                httpsEquiv = value
            elif key == 'content':
                contentType = value
                contentTypeIndex = i

        if httpsEquiv and contentType: # It's an interesting meta tag.
            match = self.CHARSET_RE.search(contentType)
            if match:
                if (self.declaredHTMLEncoding is not None or
                    self.originalEncoding == self.fromEncoding):
                    # An HTML encoding was sniffed while converting
                    # the document to Unicode, or an HTML encoding was
                    # sniffed during a previous pass through the
                    # document, or an encoding was specified
                    # explicitly and it worked. Rewrite the meta tag.
                    def rewrite(match):
                        return match.group(1) + "%SOUP-ENCODING%"
                    newAttr = self.CHARSET_RE.sub(rewrite, contentType)
                    attrs[contentTypeIndex] = (attrs[contentTypeIndex][0],
                                               newAttr)
                    tagNeedsEncodingSubstitution = True
                else:
                    # This is our first pass through the document.
                    # Go through it again with the encoding information.
                    newCharset = match.group(3)
                    if newCharset and newCharset != self.originalEncoding:
                        self.declaredHTMLEncoding = newCharset
                        self._feed(self.declaredHTMLEncoding)
                        raise StopParsing
                    pass
        tag = self.unknown_starttag("meta", attrs)
        if tag and tagNeedsEncodingSubstitution:
            tag.containsSubstitutions = True

---

Member Data Documentation

tuple BeautifulSoup::BeautifulSoup::CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M) [static]

Definition at line 1551 of file BeautifulSoup.py.

BeautifulSoup::BeautifulSoup::declaredHTMLEncoding

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1555 of file BeautifulSoup.py.

list BeautifulSoup::BeautifulSoup::NESTABLE_BLOCK_TAGS = ['blockquote', 'div', 'fieldset', 'ins', 'del'] [static]

Definition at line 1518 of file BeautifulSoup.py.

list BeautifulSoup::BeautifulSoup::NESTABLE_INLINE_TAGS [static]

Initial value:

['span', 'font', 'q', 'object', 'bdo', 'sub', 'sup',
                            'center']

Definition at line 1512 of file BeautifulSoup.py.

dictionary BeautifulSoup::BeautifulSoup::NESTABLE_LIST_TAGS [static]

Initial value:

{ 'ol' : [],
                           'ul' : [],
                           'li' : ['ul', 'ol'],
                           'dl' : [],
                           'dd' : ['dl'],
                           'dt' : ['dl'] }

Definition at line 1521 of file BeautifulSoup.py.

dictionary BeautifulSoup::BeautifulSoup::NESTABLE_TABLE_TAGS [static]

Initial value:

{'table' : [],
                           'tr' : ['table', 'tbody', 'tfoot', 'thead'],
                           'td' : ['tr'],
                           'th' : ['tr'],
                           'thead' : ['table'],
                           'tbody' : ['table'],
                           'tfoot' : ['table'],
                           }

Definition at line 1529 of file BeautifulSoup.py.

tuple BeautifulSoup::BeautifulSoup::NESTABLE_TAGS [static]

Initial value:

buildTagMap([], NESTABLE_INLINE_TAGS, NESTABLE_BLOCK_TAGS,
                                NESTABLE_LIST_TAGS, NESTABLE_TABLE_TAGS)

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Reimplemented in BeautifulSoup::ICantBelieveItsBeautifulSoup, and BeautifulSoup::MinimalSoup.

Definition at line 1547 of file BeautifulSoup.py.

list BeautifulSoup::BeautifulSoup::NON_NESTABLE_BLOCK_TAGS = ['address', 'form', 'p', 'pre'] [static]

Definition at line 1538 of file BeautifulSoup.py.

BeautifulSoup::BeautifulSoup::originalEncoding

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1555 of file BeautifulSoup.py.

tuple BeautifulSoup::BeautifulSoup::PRESERVE_WHITESPACE_TAGS = set(['pre', 'textarea']) [static]

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1505 of file BeautifulSoup.py.

dictionary BeautifulSoup::BeautifulSoup::QUOTE_TAGS = {'script' : None, 'textarea' : None} [static]

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1507 of file BeautifulSoup.py.

tuple BeautifulSoup::BeautifulSoup::RESET_NESTING_TAGS [static]

Initial value:

buildTagMap(None, NESTABLE_BLOCK_TAGS, 'noscript',
                                     NON_NESTABLE_BLOCK_TAGS,
                                     NESTABLE_LIST_TAGS,
                                     NESTABLE_TABLE_TAGS)

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Reimplemented in BeautifulSoup::MinimalSoup.

Definition at line 1542 of file BeautifulSoup.py.

tuple BeautifulSoup::BeautifulSoup::SELF_CLOSING_TAGS [static]

Initial value:

buildTagMap(None,
                                    ['br' , 'hr', 'input', 'img', 'meta',
                                    'spacer', 'link', 'frame', 'base'])

Reimplemented from BeautifulSoup::BeautifulStoneSoup.

Definition at line 1501 of file BeautifulSoup.py.