<br><br><div class="gmail_quote">On Thu, Mar 5, 2009 at 7:45 AM, Jean-Paul Calderone <span dir="ltr"><<a href="mailto:exarkun@divmod.com">exarkun@divmod.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">On Thu, 05 Mar 2009 16:20:27 +0100, Markus Wanner <markus@bluegap.ch> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<br>
I'm trying to stream longish data via web2, but experience sudden stalls<br>
in data transfer, followed by a connection abort after a certain<br>
timeout. I can't completely reproduce the issue, yet, but figured that<br>
the size of the blocks I'm returning via the stream's read method<br>
affects the failure rate (an stream.IByteStream implementation).<br>
<br>
I've tried with chunk sizes between 1 MiB down to 256 bytes. At that<br>
rate, I suddenly get the following unhandled errors, which don't seem to<br>
have to do much with my code.<br>
</blockquote>
<br></div>
I haven't looked at how web2 handles IByteStream providers, but my first<br>
guess would be that this is an example of a somewhat common bug where<br>
Deferreds are chained to an arbitrary length based on application data<br>
and when there's too much application data, the chain gets too long (a<br>
limit imposed by how much recursion is possible when unwinding the chain),<br>
and you get this failure.</blockquote><div><br>I think this probably has something to do with the fact the IByteStream.read may optionally return a Deferred making it quite easy to write code that accidentally chain Deferreds to infinity and beyond. Such as if you're calling IByteStream.read in the callback of another Deferred and returning the result. Sometimes the read method will return a Deferred, sometimes it'll just return your data. Without reading the implementation of the provider you're using it's not easy to know which it'll do when.<br>
<br>While the specific bug you're encountering may or may not be in your application code (I haven't read it, please post a minimal example if you can) it illuminates a fundamental flaw of the twisted.web2.stream APIs. (It's a flaw we understand, and there is a ticket I can't find right now that documents how it should work there may even be a branch dialtone worked on that could possibly fix it.)<br>
<br>If this email doesn't help you find your problem please do post a minimal example and I promise I'll read it today.<br><br>-David<br></div></div><br>