## 5 Openers and Handlers 

When you fetch a URL you use an opener (an instance of the perhaps confusingly-named urllib.request. OpenerDirector ). Normally we have been using the default opener - via urlopen - but you can create custom openers. Openers use handlers. All the “heavy lifting” is done by the handlers. Each handler knows how to open URLs for a particular URL scheme (http, ftp, etc.), or how to handle an aspect of URL opening, for example HTTP redirections or HTTP cookies. 

You will want to create openers if you want to fetch URLs with specific handlers installed, for example to get an opener that handles cookies, or to get an opener that does not handle redirections. 

To create an opener, instantiate an $_{OpenerDirector}$, and then call . add_handler(some_handler_instance) repeatedly. 

Alternatively, you can use build_opener , which is a convenience function for creating opener objects with a single function call. build_opener adds several handlers by default, but provides a quick way to add more and/or override the default handlers. 

Other sorts of handlers you might want to can handle proxies, authentication, and other common but slightly spe- cialised situations. 

install_opener can be used to make an opener object the (global) default opener. This means that calls to urlopen will use the opener you have installed. 

Opener objects have an$_{open}$method, which can be called directly to fetch urls in the same way as the$_{urlopen}$ function: there’s no need to call$_{install_opener}$ , except as a convenience. 

## 6 Basic Authentication 

To illustrate creating and installing a handler we will use the HTTPBasicAuthHandler . For a more detailed discussion of this subject – including an explanation of how Basic Authentication works - see the Basic Authentication Tutorial. 

When authentication is required, the server sends a header (as well as the 401 error code) requesting authentication. This specifies the authentication scheme and a ‘realm’. The header looks like: WWW-Authenticate: SCHEME realm="REALM" . 

e.g. 

WWW-Authenticate: Basic realm="cPanel Users" 

The client should then retry the request with the appropriate name and password for the realm included as a header in the request. This is ‘basic authentication’. In order to simplify this process we can create an instance of$_{HTTPBasicAuthHandler}$ and an opener to use this handler. 

The$_{HTTPBasicAuthHandler}$ uses an object called a password manager to handle the mapping of URLs and realms to passwords and usernames. If you know what the realm is (from the authentication header sent by the server), then you can use a$_{HTTPPasswordMgr}$ . Frequently one doesn’t care what the realm is. In that case, it is convenient to use$_{HTTPPasswordMgrWithDefaultRealm}$. This allows you to specify a default username and password for a URL. This will be supplied in the absence of you providing an alternative combination for a specific realm. We indicate this by providing$_{None}$ as the realm argument to the$_{add_password}$method. 

The top-level URL is the first URL that requires authentication. URLs “deeper” than the URL you pass to .add_password() will also match. 

# create a password manager password_mgr=urllib.request.HTTPPasswordMgrWithDefaultRealm() # Add the username and password. # If we knew the realm, we could use it instead of None. top_level_url ="http://example.com/foo/ " 

(continues on next page) 

