Problems with twisted.web.client

  1. Poorly implemented and tested
  2. Inflexible API:
    1. can't get headers (without mucking around with url parsing and custom factories)
    2. can't stream request/response
    3. can't override protocol behavior without duplicating lots of code
  3. doesn't support lots of http features, eg:
    1. caching
    2. pipelining
    3. connection reuse
    4. chunked (or any other) encoding
    5. byte ranges
  4. hard to test (doesn't parameterize reactor, messes around with global state)

Replacement API

  • Introduce a low-level implementation of a good generic request/response interface
  • the interface will have a single method for submitting a request and getting back a response (or an error)
    • errors can indicate:
      • request could not be sent at all (eg, connection does not exist)
      • maybe some other stuff (dns?)
    • response includes all aspects of the http response
      • status
      • headers
      • streaming data/response body
      • trailers hahahahahahahahahahahahahahahaha
  • First implementation will be a very simple HTTP protocol implementation without support for features such as persistent connections or pipelining
    • the class can be improved later to add these features without changing the basic request/response api
  • Higher level wrappers (with the same request/response API) will add features like:
    • transparent redirect handling
    • authentication
    • caching/etags/if-modified-since/etc
    • connection management (pools, reconnect, etc)
    • ssl something? certificate/hostname verification etc
    • etc


  1. Implement minimal, good, well-tested (etc) HTTP client protocol and the basic request/response API from above (maybe based on `twisted.web2.client`)
  2. Support timeouts: #3450, #4330, probably others
  3. Implement a high-level API with minimal additional functionality
    1. Provide a file-based message body producer
  4. Allow binding to a particular local interface (handled in #3450)
  5. Make requests over IPv6 when necessary/appropriate
  6. Support HTTP proxies
  7. Implement other HTTP protocol and high-level features
    1. Provide a multipart/form-data body producer
    2. Support HTTPS URLs and certificate verification
    3. Implement a cookie-aware `Agent` wrapper
    4. Implement a gzip-aware Agent wrapper (sending and receiving?)
    5. Implement an authentication wrapper
    6. Implement a caching wrapper
    7. high-level access to persistent connections
      1. low-level protocol support for controlling whether the connection is re-used or not
    8. Support pipelined requests in the high-level API
    9. ''100 Continue'' support
    10. Redirect following (perhaps multiple policies, dealing with POST/PUT/etc redirects)
    11. Port the XML-RPC client to use Agent
  8. Implement HTTP header parsing to provide access to structured header values (based on twisted.web2.http_headers)
Last modified 8 years ago Last modified on 03/01/14 13:44:46