Problems with twisted.web.client
- Poorly implemented and tested
- Inflexible API:
- can't get headers (without mucking around with url parsing and custom factories)
- can't stream request/response
- can't override protocol behavior without duplicating lots of code
- doesn't support lots of http features, eg:
- caching
- pipelining
- connection reuse
- chunked (or any other) encoding
- byte ranges
- AND SO ON FOR EVER
- hard to test (doesn't parameterize reactor, messes around with global state)
Replacement API
- Introduce a low-level implementation of a good generic request/response interface
- the interface will have a single method for submitting a request and getting back a response (or an error)
- errors can indicate:
- request could not be sent at all (eg, connection does not exist)
- maybe some other stuff (dns?)
- response includes all aspects of the http response
- status
- headers
- streaming data/response body
- trailers hahahahahahahahahahahahahahahaha
- First implementation will be a very simple HTTP protocol implementation without support for features such as persistent connections or pipelining
- the class can be improved later to add these features without changing the basic request/response api
- Higher level wrappers (with the same request/response API) will add features like:
- transparent redirect handling
- caching/etags/if-modified-since/etc
- connection management (pools, reconnect, etc)
- ssl something? certificate/hostname verification etc
- etc
Plan
- Implement minimal, good, well-tested (etc) HTTP client protocol and the basic request/response API from above (maybe based on `twisted.web2.client`)
- Support timeouts: #3450, probably others
- Implement a high-level API with minimal additional functionality
- Implement other HTTP protocol and high-level features
- persistent connections
- Implement HTTP header parsing to provide access to structured header values (based on
twisted.web2.http_headers
)
Download in other formats: