<html><body>On 01:53 pm, daniel@keystonewood.com wrote:<br />>On Apr 5, 2007, at 5:53 AM, glyph@divmod.com wrote:<br />>> >>I'm afraid that the feature you want doesn't make any sense and  is,  in <br />>>a<br />>> >>broad sense, impossible.<br />>><br />>> >Maybe it's impossible for you to see things the way I see them   because <br />>>you<br />>> >have become drunk on Twisted Kool-Aide.<br />>><br />>>You are making it sound like you are bringing a fresh new idea to  the <br />>>discussion here, which I've never heard before and am unwilling  to <br />>>consider because I'm inflexible in my thinking.  That's not  what's <br />>>happening.<br />><br />>I'm sorry I wrote that...it was inflammatory and did not bring any  value to <br />>the conversation. Please accept my apology.<br /><br />Thank you, I very much appreciate the sentiment!  I'm glad to see you also only quoted the actually useful / productive parts of my response too ;).<br /><br />>The external "blocking" resource is just a shell script that takes  some <br />>time t run. It does not acquire any shared resources that would  result in <br />>dead lock and it will always return (maybe with an error,  but it will <br />>return) unless something terrible happens (e.g. plug is  pulled on server, <br />>fire, etc.).<br /><br />I thought I understood what was going on, but now I'm confused again.  Why do you need mutual exclusion at all if it doesn't acquire any shared resources?  Couldn't you just run it concurrently?<br /><br />>It would be more maintainable because it would look just like normal <br />>sequential python code:<br /><br />Yes, it would *look* like sequential python code.  But it wouldn't be :).  There's a heck of a lot that can happen in acquire(); your whole application could run for ten minutes on that one line of code.  Worst of all, it would only happen in extreme situations, so testing or debugging issues that are caused by it becomes even more difficult.<br /><br /><snip blocking code><br /><br />>This is very simple and very easy to maintain. It could be written  with <br />>inlineCallbacks fairly easily as well:<br />><br />>yield lock.acquire()<br />>try:<br />>     yield process.check_call(...)<br />>     yeild process.check_call(...)<br />>finally:<br />>     lock.release()<br />><br />>That's pretty nice (so nice I might just rewrite my code that way).<br /><br />I'm glad you think so.  I was originally not too happy about inlineCallbacks (its predecessors did not do so well) but I keep seeing examples like this which it makes look much nicer.<br /><br />>My complaint is that the code must have knowledge of the twisted  environment <br />>(why else would it yield the result of process.check_call ()?). I do not <br />>really see the conceptual difference between these two  code blocks except <br />>one yields to and one calls into the reactor event  loop. Is there some <br />>other inherent problem with the first example? Of  course you need to make <br />>sure that the code inside the try/finally  block does not try to acquire the <br />>lock again, but that's a basic  concurrency problem which can even happen in <br />>the next example.<br /><br />This is really the key thing.  If you're running your code in the Twisted environment, and you want it to be correct, it really must know about the Twisted environment.  The simple presence of the 'yield' keyword at every level where a Deferred is being returned forces you to acknowledge, "yes, I know that a context switch may occur here".  Without it, any function could suddenly and radically change the assumptions that all of its callers were allowed to make.<br /><br />>Moving on, in a fully deferred world we have this:<br /><br /><snip ugly stuff><br /><br />>... you get the picture.<br />><br />>Notice the code to acquire/release the lock--there are three  different <br />>calls to lock.release() in there, and they all must be  carefully sorted out <br />>to make sure that exactly one of them will be  called in any given scenario <br />>--that's hard to maintain.<br /><br />There are other ways to deal with that.  maybeDeferred, for example, will make sure you always get a Deferred back and that it looks vaguely correct.<br /><br />>Right, that would work and that's exactly what subprocess.check_call () (the <br />>real python built-in version) would do. Unfortunately twisted  does not work <br />>with the subprocess module--spawnProcess() is the only  alternative I found <br />>that actually works and that means I have to use  a deferred.<br /><br />The only thing I have to say about that is:<br />    http://twistedmatrix.com/trac/ticket/733<br /><br />>>Another solution here would be for Twisted to have a nice  convenience API <br />>>for dispatching tasks to a process pool.  Right now  setting up a process <br />>>pool is conceptually easy but mechanically  difficult; you have to do a lot <br />>>of typing and make a lot of  irrelevant decisions (AMP or PB or pickle? <br />>>stdio or sockets?).<br />><br />>That sounds nice.<br /><br />Something I'd be doing in my copious spare time, if I had any.<br /><br />>I understand that PB is fully symmetrical. In my case I am only using  half <br />>(client makes request, server responds). Would it make sense to  relax the <br />>constraints when PB is used in this way?<br /><br />I don't know if it would be feasible to do the work required for PB, due to other, less fundamental implementation issues.  However, it was a design goal of AMP that it be possible to implement a "naive", only-a-few-lines-of-Python version for drop-in ease-of-use comparable to XMLRPC while still providing the actual "good" version in Twisted itself.  I have heard rumors to the effect that Eric Mangold actually wrote such a thing, but I don't know where it is.<br /><br />>>#2545.<br />><br />>This looks very interesting. I'll try to help out with this effort if  I can <br />>find some time.<br /><br />Thanks.<br /><br />>Thanks for taking time to read my ramblings and understand the  problems <br />>that I am having (even if we don't quite agree on the  simplest solutions). <br />>Your input is valuable, and I am indebted to you  for providing free support <br />>in your spare time.<br /><br />Thanks very much for taking the time to acknowledge this.  You leave me here with the impression that writing these emails was time well spent.  And, thanks in advance for working on any of those tickets I gave you links to ;-).<br /><br /></body></html>