Black Hat Python, 2nd Edition by Justin Seitz & Tim Arnold

Author:Justin Seitz & Tim Arnold [Justin Seitz]
Language: eng
Format: epub
Publisher: No Starch Press
Published: 2021-04-12T16:00:00+00:00

HTMLParser 101

In the example in this section, we used the requests and lxml packages to make HTTP requests and parse the resulting content. But what if you are unable to install the packages and therefore must rely on the standard library? As we noted in the beginning of this chapter, you can use urllib for making your requests, but you’ll need to set up your own parser with the standard library html.parser.HTMLParser.

There are three primary methods you can implement when using the HTMLParser class: handle_starttag, handle_endtag, and handle_data. The handle_starttag function will be called anytime an opening HTML tag is encountered, and the opposite is true for the handle_endtag function, which gets called each time a closing HTML tag is encountered. The handle_data function gets called when there is raw text between tags. The function prototypes for each function are slightly different, as follows:

handle_starttag(self, tag, attributes) handle_endttag(self, tag) handle_data(self, data)

Here’s a quick example to highlight this:

<title>Python rocks!</title> handle_starttag => tag variable would be "title" handle_data => data variable would be "Python rocks!" handle_endtag => tag variable would be "title"

With this very basic understanding of the HTMLParser class, you can do things like parse forms, find links for spidering, extract all of the pure text for data-mining purposes, or find all of the images in a page.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.