Problems with twisted.web.client

1. Poorly implemented and tested
2. Inflexible API:
   1. can't get headers (without mucking around with url parsing and custom factories)
   2. can't stream request/response
   3. can't override protocol behavior without duplicating lots of code
3. doesn't support lots of http features, eg:
   1. caching
   2. pipelining
   3. connection reuse
   4. chunked (or any other) encoding
   5. byte ranges
   6. AND SO ON FOR EVER
4. hard to test (doesn't parameterize reactor, messes around with global state)

Replacement API

• Introduce a low-level implementation of a good generic request/response interface
• the interface will have a single method for submitting a request and getting back a response (or an error)
  • errors can indicate:
    • request could not be sent at all (eg, connection does not exist)
    • maybe some other stuff (dns?)
  • response includes all aspects of the http response
    • status
    • headers
    • streaming data/response body
    • trailers hahahahahahahahahahahahahahahaha
• First implementation will be a very simple HTTP protocol implementation without support for features such as persistent connections or pipelining
  • the class can be improved later to add these features without changing the basic request/response api
• Higher level wrappers (with the same request/response API) will add features like:
  • transparent redirect handling
  • authentication
  • caching/etags/if-modified-since/etc
  • connection management (pools, reconnect, etc)
  • ssl something? certificate/hostname verification etc
  • etc

Plan

1. Implement minimal, good, well-tested (etc) HTTP client protocol and the basic request/response API from above (maybe based on `twisted.web2.client`)
2. Support timeouts: #3450, #4330, probably others
3. Implement a high-level API with minimal additional functionality
   1. Provide a file-based message body producer
4. Allow binding to a particular local interface (handled in #3450)
5. Make requests over IPv6 when necessary/appropriate
6. Support HTTP proxies
7. Implement other HTTP protocol and high-level features
   1. Provide a multipart/form-data body producer
   2. Support HTTPS URLs and certificate verification
   3. Implement a cookie-aware `Agent` wrapper
   4. Implement a gzip-aware Agent wrapper (sending and receiving?)
   5. Implement an authentication wrapper
   6. Implement a caching wrapper
   7. high-level access to persistent connections
      1. low-level protocol support for controlling whether the connection is re-used or not
   8. Support pipelined requests in the high-level API
   9. ''100 Continue'' support
   10. Redirect following (perhaps multiple policies, dealing with POST/PUT/etc redirects)
   11. Port the XML-RPC client to use Agent
8. Implement HTTP header parsing to provide access to structured header values (based on twisted.web2.http_headers)