[Twisted-Python] Couched 0.1: CouchDB twisted client

glyph at divmod.com glyph at divmod.com
Fri Sep 5 21:12:13 EDT 2008


On 5 Sep, 11:25 pm, thostrup at gmail.com wrote:
>2008/9/5 Jean-Paul Calderone <exarkun at divmod.com>
>>On Fri, 5 Sep 2008 09:48:08 +0200, Henrik Thostrup Jensen <
>>thostrup at gmail.com> wrote:
>>>[snip]
>>>Twisted web is currently not an option (I need request pipelining and 
>>>http
>>>1.1).

Please feel free to use twisted.web2 if it is meeting your needs 
currently.  However, our goal remains to focus maintenance effort on 
twisted.web and provide a gradual, compatible evolution to a better 
webserver.  The HTTP 1.1 features that you need will eventually be added 
to twisted.web, and I hope that the split with paisley can be resolved 
at that point.

I personally don't use couchdb or have any interest in its bindings, but 
it sounds like you have some other misconceptions about twisted and its 
development process which I'd like to correct.
>>Which other HTTP 1.1 features do you need?

>Furthermore I need the server to support large uploads (last i checked 
>the
>640K limitation was still there) and downloads, and with reasonable
>efficiancy.

640k limitation?  In twisted.web?  As far as I know there's no such 
thing; the only 640k limitation I know of anywhere in Twisted is in PB, 
and that's just on individual message size.  A brief skim through the 
twisted.web client and server code doesn't reveal any obvious size 
limitations that I can see.  (Is there some 640k limitation in the 'cgi' 
module?)

If there is such a limitation it should be more clearly documented 
somewhere.

True, twisted.web doesn't provide a graceful way for resources to handle 
streaming file uploads as they arrive, but for dedicated servers it is 
certainly *possible*; Request.handleContentChunk says it's "not for 
users", but you could easily write a few simple patches which would make 
it "for users", or just subclass it and ignore the docstring.  Nothing 
in twisted.web is really that hard to fix.
>Ticket #1956 fixes some of these things. I'm not quite sure how it 
>handles
>errors (which can happen both ways), but I've just had a quick look. 
>It's
>been a whlie since i've had a look at all this. I write a bit about it,
>which I've placed here: http://www.cs.aau.dk/~htj/prodcons.txt

Ticket #1956 is still kind of stuck defining the problem.

If you have comments or thoughts on #1956, please feel free to add them 
as comments.  Text files lying around the web are not as helpful as 
discussions on tickets.

Your text file suffers from the same problems as the existing discussion 
on the ticket, though: a lack of concrete use-cases and example code. 
There are lots of proposed solutions but not a lot of proposed problems. 
(I am certainly a guilty party here, I really should describe the 
problem in more detail.)  We all feel there is some inelegance, but 
there is no example of some code that we can all agree is gross and 
would be fixed by some better API.

Please come up with an example - the shorter the better - of a 
deficiency in the streaming API and put it on the ticket.  Hopefully 
that will further the discussion there.
>IMHO, getting (and deciding on) a proper producer/consumer interface 
>need to
>happen, to make whatever web things twisted will have usefull. Changing 
>the
>producer/consumer might be massive API break, however it is a needed 
>one.

No.

1. We will never break compatibility with an API this old and this 
stable.  When we do start to move over to a new API, the transition will 
be particularly gradual and slow given the core nature of this API and 
the fact that pretty much everything uses it.  Even if it were only 
around for one release, see 
http://twistedmatrix.com/trac/wiki/CompatibilityPolicy

2. It's not clear what the new API would be.  Every suggestion proposed 
thus far has some issues, and there's no clear winner.  Unless someone 
takes the time to really thoroughly explain the problem and describe 
what effect the solution would have, with examples, then we're not going 
to make any progress.

3. There isn't anything that you *can't do* with 
ITransport/IProducer/IConsumer/IProtocol.  It can be slightly awkward 
sometimes, but I have yet to see an example where you really can't make 
something work.  For example, IConsumer doesn't have .finish(), but 
that's fine, because you probably mean IProtocol.connectionLost(...) 
anyway.  Some users of these interfaces don't provide nice ways to get 
data in and out, and yes, a uniform and consistent way to deal with 
slinging data between endpoints (for example, a standard, featureful 
object to remix a 'protocol' into a 'transport' and vice versa) would 
probably help with that.  Given that the deficiency creates momentary 
inconveniences rather than impossibilities, there's no way we're going 
to break *every single twisted program ever* just to make it slightly 
more convenient.

4. Given that the semantics are already there, it will be very easy to 
evolve new code to call old interfaces, provide adapters which wrap 
around things and generally make it easy to have some hypothetical shiny 
new interface cooperate just fine with the crufty old ones.




More information about the Twisted-Python mailing list