Version 5 (modified by exarkun, 6 years ago) (diff)


Problems with twisted.web.client

  1. Poorly implemented and tested
  2. Inflexible API:
    1. can't get headers (without mucking around with url parsing and custom factories)
    2. can't stream request/response
    3. can't override protocol behavior without duplicating lots of code
  3. doesn't support lots of http features, eg:
    1. caching
    2. pipelining
    3. connection reuse
    4. chunked (or any other) encoding
    5. byte ranges
  4. hard to test (doesn't parameterize reactor, messes around with global state)

Replacement API

  • Introduce a low-level implementation of a good generic request/response interface
  • the interface will have a single method for submitting a request and getting back a response (or an error)
    • errors can indicate:
      • request could not be sent at all (eg, connection does not exist)
      • maybe some other stuff (dns?)
    • response includes all aspects of the http response
      • status
      • headers
      • streaming data/response body
      • trailers hahahahahahahahahahahahahahahaha
  • First implementation will be a very simple HTTP protocol implementation without support for features such as persistent connections or pipelining
    • the class can be improved later to add these features without changing the basic request/response api
  • Higher level wrappers (with the same request/response API) will add features like:
    • transparent redirect handling
    • caching/etags/if-modified-since/etc
    • connection management (pools, reconnect, etc)
    • ssl something? certificate/hostname verification etc
    • etc


  1. Implement minimal, good, well-tested (etc) HTTP client protocol and the basic request/response API from above (maybe based on `twisted.web2.client`)
  2. Support timeouts: #3450, probably others
  3. Implement a high-level API with minimal additional functionality
  4. Implement other HTTP protocol and high-level features
    1. persistent connections
  5. Implement HTTP header parsing to provide access to structured header values (based on twisted.web2.http_headers)