Problems with twisted.web.client
- Poorly implemented and tested
- Inflexible API:
- can't get headers (without mucking around with url parsing and custom factories)
- can't stream request/response
- can't override protocol behavior without duplicating lots of code
- doesn't support lots of http features, eg:
- caching
- pipelining
- connection reuse
- chunked (or any other) encoding
- byte ranges
- AND SO ON FOR EVER
- hard to test (doesn't parameterize reactor, messes around with global state)
Replacement API
- Introduce a low-level implementation of a good generic request/response interface
- the interface will have a single method for submitting a request and getting back a response (or an error)
- errors can indicate:
- request could not be sent at all (eg, connection does not exist)
- maybe some other stuff (dns?)
- response includes all aspects of the http response
- status
- headers
- streaming data/response body
- trailers hahahahahahahahahahahahahahahaha
- errors can indicate:
- First implementation will be a very simple HTTP protocol implementation without support for features such as persistent connections or pipelining
- the class can be improved later to add these features without changing the basic request/response api
- Higher level wrappers (with the same request/response API) will add features like:
- transparent redirect handling
- authentication
- caching/etags/if-modified-since/etc
- connection management (pools, reconnect, etc)
- ssl something? certificate/hostname verification etc
- etc
Plan
- Implement minimal, good, well-tested (etc) HTTP client protocol and the basic request/response API from above (maybe based on `twisted.web2.client`)
- Support timeouts: #3450, #4330, probably others
- Implement a high-level API with minimal additional functionality
- Allow binding to a particular local interface (handled in #3450)
- Make requests over IPv6 when necessary/appropriate
- Support HTTP proxies
- Implement other HTTP protocol and high-level features
- Provide a multipart/form-data body producer
- Support HTTPS URLs and certificate verification
- Implement a cookie-aware `Agent` wrapper
- Implement a gzip-aware
Agent
wrapper (sending and receiving?) - Implement an authentication wrapper
- Implement a caching wrapper
- high-level access to persistent connections
- Support pipelined requests in the high-level API
- ''100 Continue'' support
- Redirect following (perhaps multiple policies, dealing with POST/PUT/etc redirects)
- Port the XML-RPC client to use Agent
- Implement HTTP header parsing to provide access to structured header values (based on
twisted.web2.http_headers
)
Last modified 8 years ago
Last modified on 03/01/14 13:44:46