Using Python urllib and urllib2 for Data Transfer, Headers, Cookies, and Proxy Configuration

Python provides two modules, urllib and urllib2 , for making URL requests; the main difference is that urllib2 can accept a Request object to set request headers, while urllib works with plain URLs and offers urlencode for building query strings.

Typical usage combines both modules: encode form data with urllib.urlencode , pass the resulting string as the data argument to a urllib2.Request , and then open the request with urllib2.urlopen to retrieve the response.

To send JSON data, first serialize the Python object with json.dumps , then include it as the request body and set the appropriate Content-Type: application/json header.

Custom headers can be added via Request.add_header (or add_headers in the example). Common headers include User-Agent for client identification and Content-Type with values such as application/xml , application/json , or application/x-www-form-urlencoded depending on the API.

Cookies are automatically handled by urllib2 , but for explicit control you can use a HTTPCookieProcessor with build_opener to store cookies in a file and simulate login sessions.

Advanced features like proxy configuration, timeout, redirection handling, and HTTP error processing are achieved by creating a custom opener with urllib2.build_opener and adding appropriate handlers (e.g., ProxyHandler , HTTPHandler , HTTPSHandler ).

Once the custom opener is installed via urllib2.install_opener , subsequent calls to urllib2.urlopen will use the configured settings, allowing consistent proxy usage and other HTTP options across the application.

For further details and complete examples, refer to the original tutorial linked at the end of the article.