<div>Hi,</div><div><br></div><div>I have written a small utility function to replace "twisted.web.client.getPage", to be able to read the response header.</div><div><br></div><div>I have to say that the ever improving documentation made it quite easy for me to do it using the new twisted.web.client.Agent, so well done to all!</div>
<div><br></div><div>Since my wrapper works quite well, I decided to add gzip response support, as it's another feature lacking from the original getPage. Again, it was quite simple and it looks it works quite well, in proof of concept scenario.</div>
<div><br></div><div>Then it came my dilemma. What I'm doing now is a synchronous decompression as shown below:</div><div><br></div><div>compressedstream = StringIO.StringIO(inzip) </div><div>gzipper = gzip.GzipFile(fileobj=compressedstream)</div>
<div>_data = gzipper.read()</div><div>return _data</div><div><br></div><div>This works quite well, but I wanted to add support for arbitrary large compressed responses, and I wanted to ask your opinion on the best approach for this:</div>
<div><br></div><div>-a separate thread? it has it's limit, as it's not scaling well at all, but, in the possible scenario of a getPage usege shouldn't be a big issue (i.e. not many concurrent calls)</div><div>
<br></div><div>-a Producer/consumer? That sounded like the modern twisted way of doing it, but I didn't manage to be able to implement it properly, as I could create a proper "consumer" class by looking to the example in the documentation...</div>
<div><br></div><div>-twisted.python.zipstream.DeflatedZipFileEntry?</div><div>I found this and seemed a potential way of doing it, with may be the use of inline generators?</div><div>But then, I thought, is it a too complex approach for a simple problem? </div>
<div><br></div><div>I guess that decompressing data in twisted should be a fairly common task, but I have not found a sample that looked like the "best" way for doing it, so... here is this email</div><div><br></div>
<div>Thanks for you help, and I'll be happy to post the final code for future reference if anyone is interested</div><div><br></div><div>Michele</div>