Version 13 (modified by exarkun, 6 years ago) (diff)

a few more features that we should have

Problems with twisted.web.client

  1. Poorly implemented and tested
  2. Inflexible API:
    1. can't get headers (without mucking around with url parsing and custom factories)
    2. can't stream request/response
    3. can't override protocol behavior without duplicating lots of code
  3. doesn't support lots of http features, eg:
    1. caching
    2. pipelining
    3. connection reuse
    4. chunked (or any other) encoding
    5. byte ranges
  4. hard to test (doesn't parameterize reactor, messes around with global state)

Replacement API

  • Introduce a low-level implementation of a good generic request/response interface
  • the interface will have a single method for submitting a request and getting back a response (or an error)
    • errors can indicate:
      • request could not be sent at all (eg, connection does not exist)
      • maybe some other stuff (dns?)
    • response includes all aspects of the http response
      • status
      • headers
      • streaming data/response body
      • trailers hahahahahahahahahahahahahahahaha
  • First implementation will be a very simple HTTP protocol implementation without support for features such as persistent connections or pipelining
    • the class can be improved later to add these features without changing the basic request/response api
  • Higher level wrappers (with the same request/response API) will add features like:
    • transparent redirect handling
    • caching/etags/if-modified-since/etc
    • connection management (pools, reconnect, etc)
    • ssl something? certificate/hostname verification etc
    • etc


  1. Implement minimal, good, well-tested (etc) HTTP client protocol and the basic request/response API from above (maybe based on `twisted.web2.client`)
  2. Support timeouts: #3450, #4330, probably others
  3. Implement a high-level API with minimal additional functionality
    1. Provide a file-based message body producer
    2. Provide a multipart/form-data body producer
    3. Support HTTPS URLs and certificate verification
    4. Implement a cookie-aware Agent wrapper
  4. Support HTTP proxies
  5. Implement other HTTP protocol and high-level features
    1. high-level access to persistent connections
      1. low-level protocol support for controlling whether the connection is re-used or not
    2. 100 Continue support
  6. Implement HTTP header parsing to provide access to structured header values (based on twisted.web2.http_headers)