From wolfgang.kde at rohdewald.de Mon Sep 1 05:39:25 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 01 Sep 2014 13:39:25 +0200 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved Message-ID: <1574968.iq743TZKTO@i5> Hi, while trying to port jelly to Python 3 (banana tests already pass with Python 3), I found a bug in jelly. It has already been reported 3 years ago with ticket 4935. And there is a fix attached to that ticket. This fix resolves the problem for me. Why has this fix never been included? It is necessary for porting to Python 3. -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 1 05:57:22 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 01 Sep 2014 11:57:22 -0000 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved In-Reply-To: <1574968.iq743TZKTO@i5> References: <1574968.iq743TZKTO@i5> Message-ID: <20140901115722.20413.1823536452.divmod.xquotient.1771@top> On 11:39 am, wolfgang.kde at rohdewald.de wrote: >Hi, > >while trying to port jelly to Python 3 (banana tests already pass with >Python 3), > >I found a bug in jelly. It has already been reported 3 years ago with >ticket 4935. > >And there is a fix attached to that ticket. This fix resolves the >problem for me. > >Why has this fix never been included? It is necessary for porting to >Python 3. Perhaps because the ticket was never given the "review" keyword - so no one ever noticed that there was a potential fix attached to it. See https://twistedmatrix.com/trac/wiki/ReviewProcess Jean-Paul >-- >Wolfgang > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python at twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From grigorescu_ciprian at yahoo.com Mon Sep 1 06:24:43 2014 From: grigorescu_ciprian at yahoo.com (Grigorescu Ciprian) Date: Mon, 1 Sep 2014 05:24:43 -0700 Subject: [Twisted-Python] Using spawnProcess Message-ID: <1409574283.53606.YahooMailNeo@web141705.mail.bf1.yahoo.com> Hello, I have wrote a custom protocol(inherit from SMTPClient) and factory(inherit from ClientFactory) in order to send emails from multiple ips(multiple connections, factories). Now I want to run this pice of code multiple times using spawnProcess to send emails simultaneous, each process having "n" connections from "n" ips. How can i use spawnProcess? Or is another way to send emails in a parallel fashion with twisted? Maybe another approach? Thank you, -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 1 06:31:16 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 01 Sep 2014 14:31:16 +0200 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved In-Reply-To: <20140901115722.20413.1823536452.divmod.xquotient.1771@top> References: <1574968.iq743TZKTO@i5> <20140901115722.20413.1823536452.divmod.xquotient.1771@top> Message-ID: <4619137.bWkyiyDUYJ@i5> Am Montag, 1. September 2014, 11:57:22 schrieb exarkun at twistedmatrix.com: > >Why has this fix never been included? It is necessary for porting to > >Python 3. > > Perhaps because the ticket was never given the "review" keyword - so no > one ever noticed that there was a potential fix attached to it. So I should formally be able to review this since I am not the author. What I can say for certain is that the fix looks OK, and porting to Python3 depends on it. Could somebody else please review this ticket 4935? -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 1 06:52:56 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 01 Sep 2014 12:52:56 -0000 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved In-Reply-To: <4619137.bWkyiyDUYJ@i5> References: <1574968.iq743TZKTO@i5> <20140901115722.20413.1823536452.divmod.xquotient.1771@top> <4619137.bWkyiyDUYJ@i5> Message-ID: <20140901125256.20413.1175149262.divmod.xquotient.1784@top> On 12:31 pm, wolfgang.kde at rohdewald.de wrote: >Am Montag, 1. September 2014, 11:57:22 schrieb >exarkun at twistedmatrix.com: >> >Why has this fix never been included? It is necessary for porting to >> >Python 3. >> >>Perhaps because the ticket was never given the "review" keyword - so >>no >>one ever noticed that there was a potential fix attached to it. > >So I should formally be able to review this since I am not the author. Hi Wolfgang, Your input on the issue would be much appreciated, yes. However, since neither you nor the contributor has commit access, you can't actually grant the ticket a passing review. >What I can say for certain is that the fix looks OK, and porting to >Python3 depends on it. > >Could somebody else please review this ticket 4935? I took a look at the patch. There are a couple issues: 1) The tests are not written as unit tests which integrate into the existing test suite. They need to be rewritten so that `trial twisted` will run them. 2) It wasn't immediately obvious to me how the issue is fixed by the patch. It would be nice to either comment on the ticket explaining how the change fixes the problem or add such information in a comment near the code that is changing. Thanks, Jean-Paul From exarkun at twistedmatrix.com Mon Sep 1 06:55:39 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 01 Sep 2014 12:55:39 -0000 Subject: [Twisted-Python] Using spawnProcess In-Reply-To: <1409574283.53606.YahooMailNeo@web141705.mail.bf1.yahoo.com> References: <1409574283.53606.YahooMailNeo@web141705.mail.bf1.yahoo.com> Message-ID: <20140901125539.20413.1865087905.divmod.xquotient.1789@top> On 12:24 pm, grigorescu_ciprian at yahoo.com wrote: >Hello, >I have wrote a custom protocol(inherit from SMTPClient) and >factory(inherit from ClientFactory) in order to send emails from >multiple ips(multiple connections, factories). > >Now I want to run this pice of code multiple times using spawnProcess >to send emails simultaneous, each process having "n" connections from >"n" ips. > >How can i use spawnProcess? Or is another way to send emails in a >parallel fashion with twisted? Maybe another approach? You don't need multiple processes to send email concurrently. If you were to use `twisted.mail.smtp.sendmail`, then you could do this by just making multiple calls to `sendmail`. You don't have to wait for one email to be completely sent before you start trying to send the next one. For example, to send two emails concurrently: from sys import stdout from twisted.mail.smtp import sendmail from twisted.internet.defer import gatherResults from twisted.internet.task import react from twisted.python.log import startLogging def main(): return gatherResults([ sendmail(...), sendmail(...), ]) startLogging(stdout) react(main, []) You should be able to do something similar with the API you've constructed (or perhaps you can just use `sendmail` instead). Jean-Paul From wolfgang.kde at rohdewald.de Mon Sep 1 16:24:33 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Tue, 02 Sep 2014 00:24:33 +0200 Subject: [Twisted-Python] what is a non-class class? Message-ID: <1522503.xpp1Bo2WKl@i5> >From spread/jelly.py in _unjelly_method: - if type(im_class) is not types.ClassType: raise InsecureJelly("Method found with non-class class.") Same problem in _unjelly_instance What is a non-class class or rather what sort of types is meant to be insecure? Suggestions for a better error message instead of "non-class class"? This should work with both old style and new style classes, right now it works only for old style and fails for new style. Working with Python3 gives bonus points. Current source code does not have test cases for both _unjelly_instance and _unjelly_method using new style classes. My patch for ticket 4935 has a such a test case for _unjelly_method which fails as expected. If only user defined classes are accepted as secure, see also https://stackoverflow.com/questions/14612865/how-to-check-if-object-is-instance-of-new-style-user-defined-class Would it be acceptable to instead exclude an explicit list of basic types as that stackoverflow answer suggests? if im_class in (int, long, float, bool, list, set, frozenset, dict, type(None), str, unicode, tuple, what else?): raise InsecureJelly https://twistedmatrix.com/documents/current/api/twisted.spread.jelly.InsecureJelly.html does not cover this. Python 2.7.6: >>> class A(object): ... pass ... >>> type(A) >>> type(int) >>> import types >>> types.ClassType >>> class B: ... pass ... >>> type(B) Python 3.3: >>> class A: ... pass ... >>> type(A) >>> type(int) -- Wolfgang From adi at roiban.ro Tue Sep 2 03:08:49 2014 From: adi at roiban.ro (Adi Roiban) Date: Tue, 2 Sep 2014 10:08:49 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures Message-ID: Hi, While using spawnProcess on Linux I found out that when an invalid executable is called there is a corner case in which a zombie process is left until main process exists and can not be closed. I wrote a test for this but I was not able to reproduce this error in isolation, event if I run the test for 10000 times. reapProcess will always succeed from the first call. For the production code I can always reproduce the problem. Inspecting the execution thread I found out that all pipes are closed but spawned process is not closed yet. Due to this Process.maybeCallProcessEnded() will call self.reapProcess(). In my case, os.waitpid(pid, os.WNOHANG) return 0, and self.reapProcess() will just ignore this case. Process handlers is still registered in reapProcessHandlers but reapAllProcesses is no longer called. If a add a reactor.callLater(self.reapProcess) the next call to os.waitpid will succeed and the forked process is closed. ------------ What am I doing wrong? Why reapAllProcesses is no longer called, even if reapProcessHandlers is not empty? My process protocol code is here https://gist.github.com/adiroiban/bac493f00ce5e94738ce but from what I see, this should happen for any protocol. Many thanks, -- Adi Roiban From jmp at editshare.com Tue Sep 2 05:05:33 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Tue, 02 Sep 2014 07:05:33 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: Message-ID: <5405A47D.3020106@editshare.com> On 09/02/2014 05:08 AM, Adi Roiban wrote: > Hi, > > While using spawnProcess on Linux I found out that when an invalid > executable is called there is a corner case in which a zombie process > is left until main process exists and can not be closed. > > I wrote a test for this but I was not able to reproduce this error in > isolation, event if I run the test for 10000 times. reapProcess will > always succeed from the first call. > > For the production code I can always reproduce the problem. > > Inspecting the execution thread I found out that all pipes are closed > but spawned process is not closed yet. Due to this > Process.maybeCallProcessEnded() will call self.reapProcess(). > > In my case, os.waitpid(pid, os.WNOHANG) return 0, and > self.reapProcess() will just ignore this case. We encountered this problem in our code too. We worked around it with the following code, which basically monkey-patches Twisted to "try again later" when waitpid returns 0. (Most of the code below is just copied from _BaseProcess; the important part is the "elif pid == 0" branch.) ----- """Workarounds for problems with Twisted.""" import errno import os from twisted.python import log from twisted.internet.process import ( _BaseProcess, reapAllProcesses, unregisterReapProcessHandler ) def workaround_reapProcess(reactor): """Install a workaround for unsticking reapProcess. Sometimes when a child process takes too long to die that reapProcess doesn't catch it in time. We add a hack where we add a timeout to the reactor to try again later. """ def reapProcess(self): """ Try to reap a process (without blocking) via waitpid. This is called when sigchild is caught or a Process object loses its "connection" (stdout is closed) This ought to result in reaping all zombie processes, since it will be called twice as often as it needs to be. (Unfortunately, this is a slightly experimental approach, since UNIX has no way to be really sure that your process is going to go away w/o blocking. I don't want to block.) """ try: try: pid, status = os.waitpid(self.pid, os.WNOHANG) except OSError, e: if e.errno == errno.ECHILD: # no child process pid = None else: raise except: log.msg('Failed to reap %d:' % self.pid) log.err() pid = None status = None if pid: self.processEnded(status) unregisterReapProcessHandler(pid, self) elif pid == 0: # Twisted seems to get stuck if pid is 0, which means that # the child process hasn't changed status, but if called # after SIGCHLD probably means that the child process is # in the process of dying, but hasn't quite died yet. # We'll try to kick the reactor to reap the processes # again in a bit. # # We're testing specifically against 0 because pid may # also be None in an error case. def unstick(): reapAllProcesses() reactor.callLater(1, unstick) _BaseProcess.reapProcess = reapProcess ---- To use this, import your reactor and then call workaround_reapProcess(reactor). Now that two of us have seen the same problem, we should probably file a ticket in the bug tracker. --Justin From 4kir4.1i at gmail.com Tue Sep 2 06:21:24 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Tue, 02 Sep 2014 16:21:24 +0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures References: <5405A47D.3020106@editshare.com> Message-ID: <87egvuqkl7.fsf@gmail.com> Justin Mazzola Paluska writes: Unrelated. > def unstick(): > reapAllProcesses() > reactor.callLater(1, unstick) > Why do you use `unstick` instead of `reapAllProcesses` directly. Do you expect that `reapAllProcesses` name will be rebound in the future? -- Akira From jmp at editshare.com Tue Sep 2 06:44:50 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Tue, 2 Sep 2014 08:44:50 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <87egvuqkl7.fsf@gmail.com> References: <5405A47D.3020106@editshare.com> <87egvuqkl7.fsf@gmail.com> Message-ID: On Sep 2, 2014 8:22 AM, "Akira Li" <4kir4.1i at gmail.com> wrote: > > Justin Mazzola Paluska writes: > > Unrelated. > > > def unstick(): > > reapAllProcesses() > > reactor.callLater(1, unstick) > > > > Why do you use `unstick` instead of `reapAllProcesses` directly. > Do you expect that `reapAllProcesses` name will be rebound in the future? I had a bunch of debug spew in unstick when I was developing this monkey patch. I left the indirection in in case I needed to add the debug spew back. --Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Tue Sep 2 07:31:00 2014 From: adi at roiban.ro (Adi Roiban) Date: Tue, 2 Sep 2014 14:31:00 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <5405A47D.3020106@editshare.com> References: <5405A47D.3020106@editshare.com> Message-ID: Problem solved... see below On 2 September 2014 12:05, Justin Mazzola Paluska wrote: > On 09/02/2014 05:08 AM, Adi Roiban wrote: [snip] > if pid: > self.processEnded(status) > unregisterReapProcessHandler(pid, self) > elif pid == 0: > # Twisted seems to get stuck if pid is 0, which means that > # the child process hasn't changed status, but if called > # after SIGCHLD probably means that the child process is > # in the process of dying, but hasn't quite died yet. > # We'll try to kick the reactor to reap the processes > # again in a bit. > # > # We're testing specifically against 0 because pid may > # also be None in an error case. > def unstick(): > reapAllProcesses() > reactor.callLater(1, unstick) > > _BaseProcess.reapProcess = reapProcess > > ---- > > To use this, import your reactor and then call > workaround_reapProcess(reactor). > > Now that two of us have seen the same problem, we should probably file a > ticket in the bug tracker. > --Justin My quick fix was to only call reactor.callLater(self.reapProcess) and not to reap all processes ---------- I dig deeper and I found out that since I was using reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not installed. I have switched to using just reactor.run() and the process is now killed. Thanks! -- Adi Roiban From jmp at editshare.com Tue Sep 2 07:38:55 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Tue, 02 Sep 2014 09:38:55 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> Message-ID: <5405C86F.2000609@editshare.com> On 09/02/2014 09:31 AM, Adi Roiban wrote: > Problem solved... see below > > On 2 September 2014 12:05, Justin Mazzola Paluska wrote: >> On 09/02/2014 05:08 AM, Adi Roiban wrote: > [snip] > >> if pid: >> self.processEnded(status) >> unregisterReapProcessHandler(pid, self) >> elif pid == 0: >> # Twisted seems to get stuck if pid is 0, which means that >> # the child process hasn't changed status, but if called >> # after SIGCHLD probably means that the child process is >> # in the process of dying, but hasn't quite died yet. >> # We'll try to kick the reactor to reap the processes >> # again in a bit. >> # >> # We're testing specifically against 0 because pid may >> # also be None in an error case. >> def unstick(): >> reapAllProcesses() >> reactor.callLater(1, unstick) >> >> _BaseProcess.reapProcess = reapProcess >> >> ---- >> >> To use this, import your reactor and then call >> workaround_reapProcess(reactor). >> >> Now that two of us have seen the same problem, we should probably file a >> ticket in the bug tracker. >> --Justin > My quick fix was to only call reactor.callLater(self.reapProcess) and > not to reap all processes Fair enough. FWIW, looking at your original code, you may also want to call your Deferreds in a reactor.callLater. I learned the hard way that if you callback a Deferred from processEnded and the callback spawns another process, your process will still be Twisted's process table even though it's dying. > > ---------- > > I dig deeper and I found out that since I was using > reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not > installed. > > I have switched to using just reactor.run() and the process is now killed. Without my workaround, I continue to have the problem with the gtk2reactor. --Justin From exarkun at twistedmatrix.com Tue Sep 2 07:50:52 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 02 Sep 2014 13:50:52 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <5405C86F.2000609@editshare.com> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> Message-ID: <20140902135052.20413.975133368.divmod.xquotient.1791@top> On 01:38 pm, jmp at editshare.com wrote: > >FWIW, looking at your original code, you may also want to call your >Deferreds in a reactor.callLater. I learned the hard way that if you >callback a Deferred from processEnded and the callback spawns another >process, your process will still be Twisted's process table even though >it's dying. >> >>---------- >> >>I dig deeper and I found out that since I was using >>reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not >>installed. >> >>I have switched to using just reactor.run() and the process is now >>killed. > >Without my workaround, I continue to have the problem with the >gtk2reactor. Have you reported this bug? Jean-Paul From glyph at twistedmatrix.com Tue Sep 2 17:29:41 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 2 Sep 2014 16:29:41 -0700 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <1522503.xpp1Bo2WKl@i5> References: <1522503.xpp1Bo2WKl@i5> Message-ID: On Sep 1, 2014, at 3:24 PM, Wolfgang Rohdewald wrote: > What is a non-class class or rather what sort of types is meant to > be insecure? Jelly is dynamically typed, so any value might show up in any position. In this case, a value shows up in the slot in the serialization of a method object which indicates that method's class might be any object, but it has to be a class. > Suggestions for a better error message instead of "non-class class"? It's not clear that there needs to be a better error message here; you'll only get this message if you have corrupt data on the wire, since a correct implementation of PB will never put anything other than a class in that slot. (Except I think it might be broken in the face of new-style classes; ClassType is the old-style class type, 'type' is the new one, so, that should probably be fixed, as per your other thread...) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Wed Sep 3 00:45:58 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 03 Sep 2014 08:45:58 +0200 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: References: <1522503.xpp1Bo2WKl@i5> Message-ID: <1925033.An9b9qoBL0@i5> I still do not like "non-class class" because this is a contradiction in itself. Anyway a non-class class is a class whose objects may not get a method attached. So far so good, but that was obvious. Am Dienstag, 2. September 2014, 16:29:41 schrieb Glyph Lefkowitz: > Except I think it might be broken in the face of new-style classes; > ClassType is the old-style class type, 'type' is the new one, so, > that should probably be fixed That is my problem. How do I know if the object is of a non-class class? As you say, current code does not handle new style classes. I cannot check if the object is of type "type" because int is also of type "type". An int certainly should not get a method attached. Then we could just as well remove this check. That is why I proposed to instead exclude a list of basic types int, float, list, dict, set and so on. -- Wolfgang From glyph at twistedmatrix.com Wed Sep 3 01:29:59 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 3 Sep 2014 00:29:59 -0700 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <1925033.An9b9qoBL0@i5> References: <1522503.xpp1Bo2WKl@i5> <1925033.An9b9qoBL0@i5> Message-ID: <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> On Sep 2, 2014, at 11:45 PM, Wolfgang Rohdewald wrote: > I still do not like "non-class class" because this is a contradiction > in itself. How about "non-class found in class slot when deserializing method object"? > Anyway a non-class class is a class whose objects may not get a > method attached. So far so good, but that was obvious. Uh... no? A non-class in an object which is not a class. The error is when that object is provided as the class associated with a method. > Am Dienstag, 2. September 2014, 16:29:41 schrieb Glyph Lefkowitz: >> Except I think it might be broken in the face of new-style classes; >> ClassType is the old-style class type, 'type' is the new one, so, >> that should probably be fixed > > That is my problem. How do I know if the object is of a non-class class? isinstance(something, (types.ClassType, type)). > As you say, current code does not handle new style classes. Yes, but it could be easily modified to do so. > I cannot check if the object is of type "type" because > int is also of type "type". An int certainly should not get > a method attached. Then we could just as well remove this check. >>> isinstance(object, type) True >>> isinstance(object(), type) False > That is why I proposed to instead exclude a list of basic types > int, float, list, dict, set and so on. I'm not sure I understand the proposal. But in any case - it's not necessary. It's clearly possible to determine if a particular value is a type or not. -glyph From wolfgang.kde at rohdewald.de Wed Sep 3 01:55:26 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 03 Sep 2014 09:55:26 +0200 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> References: <1522503.xpp1Bo2WKl@i5> <1925033.An9b9qoBL0@i5> <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> Message-ID: <1709945.N3rAP3M3F9@i5> Am Mittwoch, 3. September 2014, 00:29:59 schrieb Glyph: > > That is my problem. How do I know if the object is of a non-class class? > > isinstance(something, (types.ClassType, type)). but that will make it possible to attach a method even to type "int" Python2: >>> isinstance(int, type) True > >>> isinstance(object, type) > True > >>> isinstance(object(), type) > False > Sure but the test only gets the class and it should certainly not instantiate it. Meanwhile I believe it is best to simply remove the test since python itself will reject most: >>> a=int >>> a.x=5 Traceback (most recent call last): File "", line 1, in TypeError: can't set attributes of built-in/extension type 'int' -- Wolfgang From vikas.c.kumar at oracle.com Wed Sep 3 05:32:04 2014 From: vikas.c.kumar at oracle.com (vikas kumar) Date: Wed, 03 Sep 2014 17:02:04 +0530 Subject: [Twisted-Python] Return only when value/handle is available Message-ID: <5406FC34.9080109@oracle.com> Hi, I am little new to twisted. I've a function(get_handle) which returns a handle. But get_handle() may throw an exception(SystemDelayException) because of some delay in system. My requirement is : When I get SystemDelayException re-attempt get_handle() and return only when handle is available. How do I use deferred/reactor in this scenario? try: handle = yield self.get_handle() except SystemDelayException: // code to re-attempt get_handle and return only when handle is available Regards Vikas From jonas.brunsgaard at gmail.com Wed Sep 3 06:40:24 2014 From: jonas.brunsgaard at gmail.com (Jonas Brunsgaard) Date: Wed, 3 Sep 2014 14:40:24 +0200 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: <5406FC34.9080109@oracle.com> References: <5406FC34.9080109@oracle.com> Message-ID: I would fiddle around with something like this. from twisted.internet.defer import inlineCallbacks, returnValue, Deferred from twisted.internet import reactor @inlineCallbacks def foo(self, retries=10, interval=10): while True: try: returnValue((yield self.get_handle())) except SystemDelayException as e: if retries > 0: retries -= 1 d = Deferred() reactor.callLater(interval, d.callback, None) yield d else: raise e On Wed, Sep 3, 2014 at 1:32 PM, vikas kumar wrote: > Hi, > > I am little new to twisted. > I've a function(get_handle) which returns a handle. > > But get_handle() may throw an exception(SystemDelayException) because of > some delay in system. > My requirement is : When I get SystemDelayException re-attempt > get_handle() and return only when handle is available. How do I use > deferred/reactor in this scenario? > > try: > handle = yield self.get_handle() > except SystemDelayException: > // code to re-attempt get_handle and return only when handle is > available > > Regards > Vikas > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Wed Sep 3 06:42:33 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 13:42:33 +0100 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: <5406FC34.9080109@oracle.com> References: <5406FC34.9080109@oracle.com> Message-ID: One option in which we have both reactor and deferred :) def do_low_level_stuff(deferred) try: deferred.callback(self.get_handle()) except SystemDelayException: reactor.callLater(1, do_low_level_stuff, deferred) def do_stuff(): deferred = Deferred() do_low_level_stuff(deferred) return deferred Cheers On 3 September 2014 12:32, vikas kumar wrote: > Hi, > > I am little new to twisted. > I've a function(get_handle) which returns a handle. > > But get_handle() may throw an exception(SystemDelayException) because of > some delay in system. > My requirement is : When I get SystemDelayException re-attempt get_handle() > and return only when handle is available. How do I use deferred/reactor in > this scenario? > > try: > handle = yield self.get_handle() > except SystemDelayException: > // code to re-attempt get_handle and return only when handle is > available > > Regards > Vikas > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Adi Roiban From jmp at editshare.com Wed Sep 3 06:53:05 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Wed, 03 Sep 2014 08:53:05 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140902135052.20413.975133368.divmod.xquotient.1791@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> Message-ID: <54070F31.1010601@editshare.com> On 09/02/2014 09:50 AM, exarkun at twistedmatrix.com wrote: >>> >>> I dig deeper and I found out that since I was using >>> reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not >>> installed. >>> >>> I have switched to using just reactor.run() and the process is now >>> killed. >> >> Without my workaround, I continue to have the problem with the >> gtk2reactor. > > Have you reported this bug? No, not yet since I don't have a 100%-reliable test case since it's mostly related to races between the child process, the parent process, and signal handling. However, if that's not required, I'd love to get it in the system. --Justin From adi at roiban.ro Wed Sep 3 07:05:06 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 14:05:06 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140902135052.20413.975133368.divmod.xquotient.1791@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> Message-ID: On 2 September 2014 14:50, wrote: > On 01:38 pm, jmp at editshare.com wrote: >> [snip] >> >> Without my workaround, I continue to have the problem with the >> gtk2reactor. > > Have you reported this bug? > > Jean-Paul In my initial use case, signal handlers were not installed since I wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can use my custom Unix Daemon and Windows Service adapters. I understand why reactor.run has the installSignalHandlers argument. Do you think it would make sense to have some kind of public API which would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do reactor.stop()) but would install the SIGCHLD handler? Thanks! -- Adi Roiban From exarkun at twistedmatrix.com Wed Sep 3 07:39:46 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 13:39:46 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> Message-ID: <20140903133946.20413.1728050305.divmod.xquotient.1804@top> On 01:05 pm, adi at roiban.ro wrote: >On 2 September 2014 14:50, wrote: >>On 01:38 pm, jmp at editshare.com wrote: >>> >[snip] >>> >>>Without my workaround, I continue to have the problem with the >>>gtk2reactor. >> >>Have you reported this bug? >> >>Jean-Paul > >In my initial use case, signal handlers were not installed since I >wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can >use my custom Unix Daemon and Windows Service adapters. > >I understand why reactor.run has the installSignalHandlers argument. > >Do you think it would make sense to have some kind of public API which >would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do >reactor.stop()) but would install the SIGCHLD handler? Yes. Providing more fine-grain control over signal handlers would be a fine improvement. Another fine improvement would be making child processes work even if a SIGCHLD handler cannot be installed (for example, by polling children periodically, by using wait() in a sidecar process, or by using a better system-specific child process monitoring API (eg kqueue's EVFILT_PROC)). Jean-Paul From adi at roiban.ro Wed Sep 3 09:27:05 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 16:27:05 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140903133946.20413.1728050305.divmod.xquotient.1804@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> Message-ID: On 3 September 2014 14:39, wrote: > On 01:05 pm, adi at roiban.ro wrote: >> >> On 2 September 2014 14:50, wrote: >>> >>> On 01:38 pm, jmp at editshare.com wrote: >>>> >>>> >> [snip] >>>> >>>> >>>> Without my workaround, I continue to have the problem with the >>>> gtk2reactor. >>> >>> >>> Have you reported this bug? >>> >>> Jean-Paul >> >> >> In my initial use case, signal handlers were not installed since I >> wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can >> use my custom Unix Daemon and Windows Service adapters. >> >> I understand why reactor.run has the installSignalHandlers argument. >> >> Do you think it would make sense to have some kind of public API which >> would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do >> reactor.stop()) but would install the SIGCHLD handler? > > > Yes. Providing more fine-grain control over signal handlers would be a fine > improvement. Do you have any suggestion for how the calls should be made? reactor.run(installSignalHandlers=True, installStopHandlers=False) or reactor.installStopHandlers = False reactor.run() > Another fine improvement would be making child processes work even if a > SIGCHLD handler cannot be installed (for example, by polling children > periodically, by using wait() in a sidecar process, or by using a better > system-specific child process monitoring API (eg kqueue's EVFILT_PROC)). > I see that GlibReactorBase inherits from PosixReactorBase so it should install the SIGCHLD handler... this should not be the reason why gtk2 reactor don't work. As a poor man's fix and Unix independent fix maybe we can call reapAllProcess in a task.LoopingCall... What are the reasons why SIGCHLD handler cannot be installed? I am asking since I hope it could help me understant where and how to enable the poor man's fix... and how to fill the bug report. kqueue's EVFILT_PROC would be great, but I guess that we still need a general fix --------- For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles but keep SIGCHLD I do: # Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK handlers _SignalReactorMixin._handleSignals = lambda self: None reactor.run() -- Adi Roiban From adi at roiban.ro Wed Sep 3 09:49:36 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 16:49:36 +0100 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows Message-ID: Hi, If I call spawnProcess with a bad executable, on Unix the process will exit with exit code 1 while on Windows an OSError is raised. I am working on a multi-OS software and to make my life easier I ended up with this hack inspired by _BackRelay https://gist.github.com/adiroiban/bac493f00ce5e94738ce ---- Is there something already in Twisted doing this? Am I reinventing the wheel? If not, do you think that it would help to update _BackRelay to support timeout and cross-os error handling? -- Adi Roiban From exarkun at twistedmatrix.com Wed Sep 3 10:51:53 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 16:51:53 -0000 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows In-Reply-To: References: Message-ID: <20140903165153.20413.1282979568.divmod.xquotient.1807@top> On 03:49 pm, adi at roiban.ro wrote: >Hi, > >If I call spawnProcess with a bad executable, on Unix the process will >exit with exit code 1 while on Windows an OSError is raised. Ideally, these two implementations of the same interface would have the same error behavior in this case. In other words, this seems like a bug to me. Jean-Paul From adi at roiban.ro Wed Sep 3 11:17:01 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 18:17:01 +0100 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows In-Reply-To: References: Message-ID: On 3 September 2014 16:49, Adi Roiban wrote: > Hi, > > If I call spawnProcess with a bad executable, on Unix the process will > exit with exit code 1 while on Windows an OSError is raised. > > I am working on a multi-OS software and to make my life easier I ended > up with this hack inspired by _BackRelay > > https://gist.github.com/adiroiban/bac493f00ce5e94738ce > > ---- > > Is there something already in Twisted doing this? Am I reinventing the wheel? > > If not, do you think that it would help to update _BackRelay to > support timeout and cross-os error handling? > Searching the Twisted trac I found this ticket http://twistedmatrix.com/trac/ticket/4184 Maybe we can leave spawnProcess as the low-level API and then create another wrapper which will handle/raise the errors in an consistent way. For the case where command is not found, I prefer the behaviour from Windows where an OSError is raise, rather than looking at errReceived and trying to guess is execvpe call failed or not. -- Adi Roiban From adi at roiban.ro Wed Sep 3 11:25:50 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 18:25:50 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> Message-ID: Just for reference: For the gkt2 reactor problem I found this bug report https://twistedmatrix.com/trac/ticket/5289 ... I guess that there is a problem with gtk2 and spawnProcess On 3 September 2014 16:27, Adi Roiban wrote: > On 3 September 2014 14:39, wrote: >> On 01:05 pm, adi at roiban.ro wrote: >>> >>> On 2 September 2014 14:50, wrote: >>>> >>>> On 01:38 pm, jmp at editshare.com wrote: >>>>> >>>>> >>> [snip] >>>>> >>>>> >>>>> Without my workaround, I continue to have the problem with the >>>>> gtk2reactor. >>>> >>>> >>>> Have you reported this bug? >>>> >>>> Jean-Paul >>> >>> >>> In my initial use case, signal handlers were not installed since I >>> wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can >>> use my custom Unix Daemon and Windows Service adapters. >>> >>> I understand why reactor.run has the installSignalHandlers argument. >>> >>> Do you think it would make sense to have some kind of public API which >>> would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do >>> reactor.stop()) but would install the SIGCHLD handler? >> >> >> Yes. Providing more fine-grain control over signal handlers would be a fine >> improvement. > > Do you have any suggestion for how the calls should be made? > > reactor.run(installSignalHandlers=True, installStopHandlers=False) > > or > > reactor.installStopHandlers = False > reactor.run() > > >> Another fine improvement would be making child processes work even if a >> SIGCHLD handler cannot be installed (for example, by polling children >> periodically, by using wait() in a sidecar process, or by using a better >> system-specific child process monitoring API (eg kqueue's EVFILT_PROC)). >> > > I see that GlibReactorBase inherits from PosixReactorBase so it should > install the SIGCHLD handler... this should not be the reason why gtk2 > reactor don't work. > > As a poor man's fix and Unix independent fix maybe we can call > reapAllProcess in a task.LoopingCall... > What are the reasons why SIGCHLD handler cannot be installed? > > I am asking since I hope it could help me understant where and how to > enable the poor man's fix... and how to fill the bug report. > > kqueue's EVFILT_PROC would be great, but I guess that we still need a > general fix > > --------- > > For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles > but keep SIGCHLD I do: > > # Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK handlers > _SignalReactorMixin._handleSignals = lambda self: None > reactor.run() > > > -- > Adi Roiban -- Adi Roiban From vikas.c.kumar at oracle.com Wed Sep 3 11:52:13 2014 From: vikas.c.kumar at oracle.com (vikas kumar) Date: Wed, 03 Sep 2014 23:22:13 +0530 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: References: <5406FC34.9080109@oracle.com> Message-ID: <5407554D.2040509@oracle.com> Hi Jonas, This is nice. But still we are retrying it repeatedly at regular interval(10). Can't we get the notified asynchronously as and when handle is available. Regards Vikas On 9/3/2014 6:10 PM, Jonas Brunsgaard wrote: > I would fiddle around with something like this. > > from twisted.internet.defer import inlineCallbacks, returnValue, Deferred > from twisted.internet import reactor > > > @inlineCallbacks > def foo(self, retries=10, interval=10): > while True: > try: > returnValue((yield self.get_handle())) > except SystemDelayException as e: > if retries > 0: > retries -= 1 > d = Deferred() > reactor.callLater(interval, d.callback, None) > yield d > else: > raise e > > On Wed, Sep 3, 2014 at 1:32 PM, vikas kumar > wrote: > > Hi, > > I am little new to twisted. > I've a function(get_handle) which returns a handle. > > But get_handle() may throw an exception(SystemDelayException) > because of some delay in system. > My requirement is : When I get SystemDelayException re-attempt > get_handle() and return only when handle is available. How do I > use deferred/reactor in this scenario? > > try: > handle = yield self.get_handle() > except SystemDelayException: > // code to re-attempt get_handle and return only when handle > is available > > Regards > Vikas > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Wed Sep 3 11:55:17 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 17:55:17 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> Message-ID: <20140903175517.20413.1988429554.divmod.xquotient.1845@top> On 03:27 pm, adi at roiban.ro wrote: >On 3 September 2014 14:39, wrote: >> >>Yes. Providing more fine-grain control over signal handlers would be >>a fine >>improvement. > >Do you have any suggestion for how the calls should be made? > >reactor.run(installSignalHandlers=True, installStopHandlers=False) Perhaps. >or > >reactor.installStopHandlers = False >reactor.run() Probably not this one. Setting attributes on random things is kind of sad. :) It might be nice to try to be somewhat flexible - in case there's some reason to change what signals the reactor wants to handle in the future. Perhaps: reactor.run(installSignalHandlers={SIGCHLD}) An entirely different direction could be to make this bit of configuration into initialization for the reactor. from twisted.internet.epollreactor import install install(installSignalHandlers={SIGCHLD}) from twisted.internet import reactor ... reactor.run() By keeping these details away from `IReactorCore.run`, that method remains maximally useful. For example, if you could set up the reactor this way, a normal `twistd` plugin would still be able to benefit from your choice, even with twistd's naive call of `reactor.run()` with no extra arguments. Application code calling these `install` functions is already supported (it's how you select a specific reactor, after all). Some of the install functions even accept arguments already. This would actually eliminate another existing issue - `IReactorCore.run` is actually defined to take no arguments. The implementations ignore this because someone thought it was important to be able to disable installation of signal handlers. > >>Another fine improvement would be making child processes work even if >>a >>SIGCHLD handler cannot be installed (for example, by polling children >>periodically, by using wait() in a sidecar process, or by using a >>better >>system-specific child process monitoring API (eg kqueue's >>EVFILT_PROC)). > >I see that GlibReactorBase inherits from PosixReactorBase so it should >install the SIGCHLD handler... this should not be the reason why gtk2 >reactor don't work. Gtk messes with signals too. There are confusing order-of-execution dependencies and Gtk changes subtly from release to release, re-breaking things after we fix them or changing them to be broken in a different way. So that's *why* it probably doesn't work with Gtk2 reactor - if not *how*. ;) I think I missed the explanation of what in particular wasn't working with Gtk2 reactor though. >As a poor man's fix and Unix independent fix maybe we can call >reapAllProcess in a task.LoopingCall... >What are the reasons why SIGCHLD handler cannot be installed? Either because you want to run the reactor in a non-main thread (where Python won't let you install signal handlers for the good reason that mixing signals and threads has crazy behavior) or because you want to use a different library that depends on having its own SIGCHLD handler and you're not interested in Twisted's process support. >I am asking since I hope it could help me understant where and how to >enable the poor man's fix... and how to fill the bug report. > >kqueue's EVFILT_PROC would be great, but I guess that we still need a >general fix Perhaps, perhaps not. A general fix might be less code but having half a dozen specialized fixes, one for each reactor, would also still fix the problem. The different reactor implementations are essentially the big piles of specialized fixes needed to do non-blocking I/O on different platforms. This would just be a little more of the same. The sidecar process is an example of a general fix, though. The idea there is that Twisted itself runs a private child process (perhaps only when the first call to spawnProcess is made). It talks to that process using a file descriptor. That process can install a SIGCHLD handler (because Twisted owns it, application developers don't get to say they don't want one installed) or use another more invasive strategy for child process management. When you want to spawn a process, the main process tells the sidecar to do it. The sidecar relays traffic between the child and the original parent (or does something involving passing file descriptors across processes). This removes the need to ever install a SIGCHLD handler in the main process. It also probably enables some optimizations (reapProcesses is O(N!) on the number of child processes right now) that are very tricky or impossible otherwise. Jean-Paul >--------- > >For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles >but keep SIGCHLD I do: > ># Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK >handlers >_SignalReactorMixin._handleSignals = lambda self: None >reactor.run() > > >-- >Adi Roiban From exarkun at twistedmatrix.com Wed Sep 3 12:00:48 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 18:00:48 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140903175517.20413.1988429554.divmod.xquotient.1845@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> Message-ID: <20140903180048.20413.1938651844.divmod.xquotient.1848@top> On 05:55 pm, exarkun at twistedmatrix.com wrote: > >Gtk messes with signals too. There are confusing order-of-execution >dependencies and Gtk changes subtly from release to release, re- >breaking things after we fix them or changing them to be broken in a >different way. > >So that's *why* it probably doesn't work with Gtk2 reactor - if not >*how*. ;) > >I think I missed the explanation of what in particular wasn't working >with Gtk2 reactor though. Oh right, that was where Justin's bug was. Jean-Paul From exarkun at twistedmatrix.com Wed Sep 3 12:07:03 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 18:07:03 -0000 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows In-Reply-To: References: Message-ID: <20140903180703.20413.1947044842.divmod.xquotient.1855@top> On 05:17 pm, adi at roiban.ro wrote: >On 3 September 2014 16:49, Adi Roiban wrote: >>Hi, >> >>If I call spawnProcess with a bad executable, on Unix the process will >>exit with exit code 1 while on Windows an OSError is raised. >> >>I am working on a multi-OS software and to make my life easier I ended >>up with this hack inspired by _BackRelay >> >>https://gist.github.com/adiroiban/bac493f00ce5e94738ce >> >>---- >> >>Is there something already in Twisted doing this? Am I reinventing the >>wheel? >> >>If not, do you think that it would help to update _BackRelay to >>support timeout and cross-os error handling? > >Searching the Twisted trac I found this ticket >http://twistedmatrix.com/trac/ticket/4184 > >Maybe we can leave spawnProcess as the low-level API and then create >another wrapper which will handle/raise the errors in an consistent >way. > >For the case where command is not found, I prefer the behaviour from >Windows where an OSError is raise, rather than >looking at errReceived and trying to guess is execvpe call failed or >not. It is more convenient, certainly. Unfortunately on POSIX the underlying API isn't synchronous - which is why the POSIX implementation doesn't deliver the exception synchronously. So to be uniform we probably need to make the Windows behavior asynchronous. I suggest we can improve the situation by reporting the error more in an easier-to-recognize way when we report it asynchronously, though. For example, instead of forcing the application to parse stderr we could deliver a different exception type to processEnded. A better solution would be to make `spawnProcess` properly asynchronous - have it return a `Deferred` that fires with an `IProcessTransport` and let that `Deferred` fire with a `Failure` if there is a problem creating the new process. But that's probably more like "introduce an API to replace `spawnProcess`" (for which I believe there is also a ticket). Jean-Paul From exarkun at twistedmatrix.com Wed Sep 3 12:07:42 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 18:07:42 -0000 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: <5407554D.2040509@oracle.com> References: <5406FC34.9080109@oracle.com> <5407554D.2040509@oracle.com> Message-ID: <20140903180742.20413.944089773.divmod.xquotient.1856@top> On 05:52 pm, vikas.c.kumar at oracle.com wrote: >Hi Jonas, > >This is nice. >But still we are retrying it repeatedly at regular interval(10). Can't >we get the notified asynchronously as and when handle is available. You tell us. What's a handle? Where do you get it from? Does that system produce an event when it is ready? Jean-Paul From glyph at twistedmatrix.com Wed Sep 3 15:28:40 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 3 Sep 2014 14:28:40 -0700 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <1709945.N3rAP3M3F9@i5> References: <1522503.xpp1Bo2WKl@i5> <1925033.An9b9qoBL0@i5> <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> <1709945.N3rAP3M3F9@i5> Message-ID: On Sep 3, 2014, at 12:55 AM, Wolfgang Rohdewald wrote: > Am Mittwoch, 3. September 2014, 00:29:59 schrieb Glyph: >>> That is my problem. How do I know if the object is of a non-class class? >> >> isinstance(something, (types.ClassType, type)). > > but that will make it possible to attach a method even to type "int" Yes, that's fine. 'int' has methods, there's no reason that those methods couldn't be serialized by jelly. For example: >>> (3).conjugate The bound method object there is the sort of thing that we're talking about. Particularly since we might be talking about a subclass of 'int' with its own overridden conjugate method, and the im_class attribute says which class the method's function actually came from. > Python2: > >>>> isinstance(int, type) > True > >>>>> isinstance(object, type) >> True >>>>> isinstance(object(), type) >> False > Sure but the test only gets the class and it should certainly not instantiate it. Right; the point is that you get a thing, and that thing may be a class object _or_ it might be an instance object, and if it's an instance that's invalid. > Meanwhile I believe it is best to simply remove the test since python itself > will reject most: > >>>> a=int >>>> a.x=5 > Traceback (most recent call last): > File "", line 1, in > TypeError: can't set attributes of built-in/extension type 'int' This is about serializing and deserializing existing methods, not assigning attributes to instances. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Thu Sep 4 01:26:39 2014 From: adi at roiban.ro (Adi Roiban) Date: Thu, 4 Sep 2014 08:26:39 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140903175517.20413.1988429554.divmod.xquotient.1845@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> Message-ID: On 3 September 2014 18:55, wrote: > On 03:27 pm, adi at roiban.ro wrote: >> >> On 3 September 2014 14:39, wrote: [snip] >> Do you have any suggestion for how the calls should be made? >> >> reactor.run(installSignalHandlers=True, installStopHandlers=False) > > > Perhaps. [snip] > It might be nice to try to be somewhat flexible - in case there's some > reason to change what signals the reactor wants to handle in the future. > Perhaps: > > reactor.run(installSignalHandlers={SIGCHLD}) > > An entirely different direction could be to make this bit of configuration > into initialization for the reactor. > > from twisted.internet.epollreactor import install > install(installSignalHandlers={SIGCHLD}) > > from twisted.internet import reactor > ... > reactor.run() > > By keeping these details away from `IReactorCore.run`, that method remains > maximally useful. For example, if you could set up the reactor this way, a > normal `twistd` plugin would still be able to benefit from your choice, even > with twistd's naive call of `reactor.run()` with no extra arguments. > > Application code calling these `install` functions is already supported > (it's how you select a specific reactor, after all). Some of the install > functions even accept arguments already. > > This would actually eliminate another existing issue - `IReactorCore.run` is > actually defined to take no arguments. The implementations ignore this > because someone thought it was important to be able to disable installation > of signal handlers. I am happy to have a simple reactor.run() and move installSignalHandlers somewhere else. working with install(installSignalHandlers={SIGCHLD}) seems a bit complicated, as I assume that many developers rely on the automatic reactor installation. In the same time, I assume that 'installSignalHandlers' argument would be supported by all reactors this is why maybe we can have something like: from twisted.internet import reactor def customHandler(signum, frame): pass reactor.installSignalHandlers( SIGCHLD=True, # Install default handler SIGTERM=None, # Don't install handler SIGINT=customHandler, # Install custom handler # SIGBREAK is not request so that default handler is installed. ) # reactor.installSignalHandlers() installs all default handlers. reactor.run() ---- reactor.run(InstallSignalHandlers=True|False) would be deprecated. In case reactor.installSignalHandlers is not called before run(), all default handlers will be installed. [snip] > The sidecar process is an example of a general fix, though. The idea there > is that Twisted itself runs a private child process (perhaps only when the > first call to spawnProcess is made). It talks to that process using a file > descriptor. That process can install a SIGCHLD handler (because Twisted > owns it, application developers don't get to say they don't want one > installed) or use another more invasive strategy for child process > management. When you want to spawn a process, the main process tells the > sidecar to do it. The sidecar relays traffic between the child and the > original parent (or does something involving passing file descriptors across > processes). > > This removes the need to ever install a SIGCHLD handler in the main process. > It also probably enables some optimizations (reapProcesses is O(N!) on the > number of child processes right now) that are very tricky or impossible > otherwise. > > Jean-Paul Thanks for the details regarding the side-process dedicated to child process management. Not sure if we need a separate ticket for that, or add it as a comment to https://twistedmatrix.com/trac/ticket/5710 Thanks! -- Adi Roiban From sangiovanni at nweb.it Thu Sep 4 04:52:07 2014 From: sangiovanni at nweb.it (Fabio Sangiovanni) Date: Thu, 4 Sep 2014 12:52:07 +0200 Subject: [Twisted-Python] Graceful shutdown of twistd application Message-ID: Hello list, I need to implement a graceful shutdown procedure for a twistd application. The application is made up of two services: an internet.TCPClient and an internet.TCPServer. They're glued together with a MultiService instance, which is in turn set to have 'application' as parent. The server and the client work together, making a proxy (SMTP server and AMQP client). My goal is the following: - intercept a SIGTERM signal - 'block' on the server side: since it's SMTP I get this by setting a variable that makes the server return tempfails (4xx) for new messages, while keeping current sessions active - wait until current requests are satisfied (I keep a dictionary of current pending messages) - shut the whole thing down What is the best solution for this use case? It's not really clear to me how to catch SIGTERM and handle pending requests *before* the underlying services start to shutdown (i.e. even addSystemEventTrigger('before', 'shutdown', callable) is called too late for my needs). Thank you very much for your help! Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Thu Sep 4 05:59:15 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 11:59:15 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> Message-ID: <20140904115915.20413.342593090.divmod.xquotient.1871@top> On 07:26 am, adi at roiban.ro wrote: >On 3 September 2014 18:55, wrote: >>On 03:27 pm, adi at roiban.ro wrote: >>> >>>On 3 September 2014 14:39, wrote: >[snip] >>>Do you have any suggestion for how the calls should be made? >>> >>>reactor.run(installSignalHandlers=True, installStopHandlers=False) >> Also note there is an old, widely scoped ticket: https://twistedmatrix.com/trac/ticket/2415 with some more stuff (not necessarily directly related to your comments on signal handling) on it. What would be really nice is if someone collected *all* of the complaints about `spawnProcess` into one place and integrated solutions to them into a design for a replacement. :) Jean-Paul From exarkun at twistedmatrix.com Thu Sep 4 06:02:48 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 12:02:48 -0000 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: References: Message-ID: <20140904120248.20413.662616188.divmod.xquotient.1874@top> On 10:52 am, sangiovanni at nweb.it wrote: >Hello list, > >I need to implement a graceful shutdown procedure for a twistd >application. >The application is made up of two services: an internet.TCPClient and >an internet.TCPServer. >They're glued together with a MultiService instance, which is in turn >set >to have 'application' as parent. >The server and the client work together, making a proxy (SMTP server >and >AMQP client). > >My goal is the following: >- intercept a SIGTERM signal >- 'block' on the server side: since it's SMTP I get this by setting a >variable that makes the server return tempfails (4xx) for new messages, >while keeping current sessions active >- wait until current requests are satisfied (I keep a dictionary of >current >pending messages) >- shut the whole thing down This is exactly what before shutdown triggers are for. Alternatively, use the higher-level API and implement `stopService` on one of your services. Either way, return a `Deferred` that only fires when you're satisfied it is time for shutdown to proceed. You said before shutdown triggers are too late but you didn't say why. I think that's based on a misunderstanding - but if not, then explain why it doesn't work for your scenario. Jean-Paul >What is the best solution for this use case? It's not really clear to >me >how to catch SIGTERM and handle pending requests *before* the >underlying >services start to shutdown (i.e. even addSystemEventTrigger('before', >'shutdown', callable) is called too late for my needs). > >Thank you very much for your help! > >Fabio From sangiovanni at nweb.it Thu Sep 4 06:36:00 2014 From: sangiovanni at nweb.it (Fabio Sangiovanni) Date: Thu, 4 Sep 2014 14:36:00 +0200 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: <20140904120248.20413.662616188.divmod.xquotient.1874@top> References: <20140904120248.20413.662616188.divmod.xquotient.1874@top> Message-ID: On Thu, Sep 4, 2014 at 2:02 PM, wrote: > > > You said before shutdown triggers are too late but you didn't say why. I > think that's based on a misunderstanding - but if not, then explain why it > doesn't work for your scenario. > Hi, thanks for your reply. I've tried the following: def sleep(secs): log.msg('from within trigger') d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10) This is what I can see in the logs: Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] Received SIGTERM, shutting down. Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] from within trigger Sep 4 14:25:06 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] will retry in 2 seconds Sep 4 14:25:06 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] Stopping factory <__builtin__.RabbitMQClientFactory instance at 0x00000000057172c0> Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] (TCP Port 10025 Closed) Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] Stopping factory <__builtin__.TempfailingESMTPFactory instance at 0x00000000057172a0> Sep 4 14:25:09 prepyproxy01 proxy [4924]: [-] Starting factory <__builtin__.RabbitMQClientFactory instance at 0x00000000057172c0> Sep 4 14:25:09 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] [rmq01] RabbitMQ connection established Sep 4 14:25:16 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] will retry in 2 seconds Sep 4 14:25:16 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] Stopping factory <__builtin__.RabbitMQClientFactory instance at 0x00000000057172c0> Sep 4 14:25:16 prepyproxy01 proxy [4924]: [-] Main loop terminated. Sep 4 14:25:16 prepyproxy01 proxy [4924]: [-] Server Shut Down. It seems to me that the shutdown phase doesn't wait for the deferred to fire before stopping my client and server. To be clear: my expected result is: - SIGTERM - pause 10s - client/server shutdown I am surely missing something, but I really can't figure out what. Oh, for the records: I'm using Twisted 13.2.0 on Pypy. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi.libotean at proatria.com Thu Sep 4 06:51:58 2014 From: adi.libotean at proatria.com (Adi Libotean) Date: Thu, 04 Sep 2014 15:51:58 +0300 Subject: [Twisted-Python] High throughput database logger Message-ID: <5408606E.2030401@proatria.com> Hi, I'm looking at various options for implementing a high throughput database logger that will work with Twisted. My requirements, listed by importance: 1) small memory footprint 2) high speed 3) low garbage generation The application I'm working on runs continuously (24/7). I've experimented a bit with pysqlite and Twisted to see which approach is better suited (see attached example). ---- Question 1: I noticed that all of the Twisted based versions are very slow compared to the plain sqlite3 test. This seems to be caused by atomic transaction management, namely a commit after each insert. Would be interested to know if there is a simple way to avoid this and do my own transaction management (aka batch commit). One other thing is the greatly varying amounts of garbage generated (peak memory) and memory usage between the Twisted variants. ---- Question 2: I would have expected B (Twisted ADBAPI) to behave very similar to C/E since I'm using a connection pool of size 1 and all requests are queued and handled sequentially. Could any of you please give me some pointers as to why this is happening? ---- Question 3: Even though objgraph lists the exact same reference count once the code is ran, the amount of used memory greatly differs. Any ideas what might be causing this? Any suggestions and/or pointers on how to improve/do this are more than welcome. Thank you for your time, Adrian -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: db-insert-test.py URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: results.txt URL: From exarkun at twistedmatrix.com Thu Sep 4 06:55:49 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 12:55:49 -0000 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: References: <20140904120248.20413.662616188.divmod.xquotient.1874@top> Message-ID: <20140904125549.20413.481091133.divmod.xquotient.1886@top> On 12:36 pm, sangiovanni at nweb.it wrote: >On Thu, Sep 4, 2014 at 2:02 PM, wrote: >> >> >>You said before shutdown triggers are too late but you didn't say why. >>I >>think that's based on a misunderstanding - but if not, then explain >>why it >>doesn't work for your scenario. > >Hi, thanks for your reply. > >I've tried the following: > >def sleep(secs): > log.msg('from within trigger') > d = defer.Deferred() > reactor.callLater(secs, d.callback, None) > return d > >reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10) All 'before' trigger are run concurrently. If you're using `Application` then your `sleep` trigger runs concurrently with the application's `stopService` trigger (because `Application` has its stopService added as another 'before' shutdown' trigger alongside yours). If you want to delay your application shutdown, you need to cooperate a little more closely with it. Either attach your application shutdown code as a callback to the sleep Deferred or move the sleep into the stopService implementation of one of the services on your application and trigger the remaining stopService calls (eg the stopService call on the MultiService you mentioned) when the sleep Deferred fires there. Jean-Paul From exarkun at twistedmatrix.com Thu Sep 4 07:20:15 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 13:20:15 -0000 Subject: [Twisted-Python] High throughput database logger In-Reply-To: <5408606E.2030401@proatria.com> References: <5408606E.2030401@proatria.com> Message-ID: <20140904132015.20413.679407771.divmod.xquotient.1912@top> On 12:51 pm, adi.libotean at proatria.com wrote: >Hi, > >I'm looking at various options for implementing a high throughput >database logger that will work with Twisted. > >My requirements, listed by importance: > >1) small memory footprint >2) high speed >3) low garbage generation > >The application I'm working on runs continuously (24/7). I've >experimented a bit with pysqlite and Twisted to see which approach is >better suited (see attached example). > >---- > >Question 1: I noticed that all of the Twisted based versions are very >slow compared to the plain sqlite3 test. This seems to be caused by >atomic transaction management, namely a commit after each insert. Not only this but in some of the Twisted versions you've introduced a round-trip communication from the reactor thread to a worker thread between each operation. This will greatly impact throughput by adding lots of latency to each insert. >Would be interested to know if there is a simple way to avoid this and >do my own transaction management (aka batch commit). Using twisted.enterprise.adbapi? You could probably hack something horrible together but it would definitely be a hack. I suggest you take a look at adbapi2 instead - . >One other thing is the greatly varying amounts of garbage generated >(peak memory) and memory usage between the Twisted variants. "Garbage" and "peak memory" are different things. The Twisted-using version does a lot more - and some of your Twisted-using versions put the *entire* data set into memory (in a vastly expanded form, where each insert is represented by multiple large objects including Deferreds). So it's not too surprising the memory usage is greater. >---- > >Question 2: I would have expected B (Twisted ADBAPI) to behave very >similar to C/E since I'm using a connection pool of size 1 and all >requests are queued and handled sequentially. > >Could any of you please give me some pointers as to why this is >happening? You didn't actually label the code with these letters. :) I'm guessing B is the `adbapi` function, C is `inline_callbacks`, and E is `cooperator`. Also you didn't say in what respect you expected them to behavior similarly. You expected their memory usage to be the same? You expected their runtime to be the same? You expected them to put the same data into the database? As far as memory usage goes, B uses lots of memory for the same reason `semaphore` (D?) uses lots of memory. You queue up the entire dataset in memory as piles of tuples, lists, Deferreds, etc. adbapi might be executing the operations one at a time, but the *loop* inside `adbapi` runs all the way to the end all in one go. It starts every one of those `runOperation`s before any of them (probably) has a chance to execute. >---- > >Question 3: Even though objgraph lists the exact same reference count >once the code is ran, the amount of used memory greatly differs. Any >ideas what might be causing this? Hopefully the above helps explain this. Something else you might consider is batching up your inserts (not just only committing after a batch of inserts). Since SQLite3 can only write from a single thread at a time, you're effectively limited to serialized inserts - so it doesn't make sense to try to start a second insert before the first has finished. When the first finishes, if 50 more data points have arrived, you should do one insert for all 50 of those - not 50 inserts each for one piece of data. This cuts off a bunch of your overhead - Python objects, round- trip latency for inter-thread communication, function calls, etc. Jean-Paul >Any suggestions and/or pointers on how to improve/do this are more than >welcome. > >Thank you for your time, >Adrian From adi at roiban.ro Fri Sep 5 00:30:35 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 5 Sep 2014 07:30:35 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140904115915.20413.342593090.divmod.xquotient.1871@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> Message-ID: On 4 September 2014 12:59, wrote: [snip] > What would be really nice is if someone collected *all* of the complaints > about `spawnProcess` into one place and integrated solutions to them into a > design for a replacement. :) > > Jean-Paul Since I can not create wiki pages here are my notes: To do: * Always return a deferred. #2415 * Provide the option to timeout the execution... and on timeout errback with a dedicated failure #2415 * Add childs to the same process group #2415 * allways errback when failing to spawn (rather than raising OSError) and errback with a different failure when failing to spawn the executable #4184 * on Unix provide a method to install SIGCHLD handler, independent of the general reactor.run() signal handlers. ... maybe related to #5710 * Allow passing Unicode environment (at least on Windows) maybe needed for Python3 #5968 * Allow passing Unicode command and arguments on Windows (current not supported by pywin32 CreateProcess) #6470... maybe needed for Python3 #5968 * Improve fork/exec for speed and memory usage #5710 .... maybe by creating a auxiliary process which handles forks and child management. * Execute as a different account in Windows * Spawn only after reactor starts ? I am ok with forking before. * Support client endpoint? ------- For a new design: * I like the ProcessProtocol API * spawnProcess will always return a deferred... for backward compatibility we need to decide a name for this new method which returns a deferred. Maybe createProcess or connectProcess * create spawnUnixProcess which has childFDs uid/gid * create spawnWindowsProcess which has Windows specific options * createProcess / connectProcess will call spawnUnixProcess or spawnWindowsProcess depending on current OS * ProcessProtocol.processEnded is similar to connectionLost.. so maybe it can be rename it and add other methods so that IProcessProtocol can inherit from IProtocol. I guess that this will help with connectors. ------ I think that child management problem, does not affect the public API so I am not discussing it here. Timeouts are handled at protocol level and I see that we have 2 timeout types: total execution time and timeout after last child data was received. ----- Does it make sense to put it on wiki or what should be the Twisted Enhancement Proposal process? Can I get 'create page' rights on Trac? Thanks! -- Adi Roiban From sangiovanni at nweb.it Fri Sep 5 02:42:17 2014 From: sangiovanni at nweb.it (Fabio Sangiovanni) Date: Fri, 5 Sep 2014 10:42:17 +0200 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: <20140904125549.20413.481091133.divmod.xquotient.1886@top> References: <20140904120248.20413.662616188.divmod.xquotient.1874@top> <20140904125549.20413.481091133.divmod.xquotient.1886@top> Message-ID: Hi, I just wanted to give an update about the way I got it, in case this could be useful for somebody. I essentially inherited from service.MultiService like this: class GracefulMultiService(service.MultiService): def stopService(self): d = defer.succeed(None) d.addCallback(self._wait_for_your_cleanup_code_to_complete) d.addCallback(lambda _: self) d.addCallback(service.MultiService.stopService) return d @defer.inlineCallbacks def _wait_for_your_cleanup_code_to_complete(self): [...] yield sleep(10) # cleanup code example def sleep(secs): d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d Thanks again for your help! On Thu, Sep 4, 2014 at 2:55 PM, wrote: > On 12:36 pm, sangiovanni at nweb.it wrote: > >> On Thu, Sep 4, 2014 at 2:02 PM, wrote: >> >>> >>> >>> You said before shutdown triggers are too late but you didn't say why. I >>> think that's based on a misunderstanding - but if not, then explain why >>> it >>> doesn't work for your scenario. >>> >> >> Hi, thanks for your reply. >> >> I've tried the following: >> >> def sleep(secs): >> log.msg('from within trigger') >> d = defer.Deferred() >> reactor.callLater(secs, d.callback, None) >> return d >> >> reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10) >> > > All 'before' trigger are run concurrently. If you're using `Application` > then your `sleep` trigger runs concurrently with the application's > `stopService` trigger (because `Application` has its stopService added as > another 'before' shutdown' trigger alongside yours). > > If you want to delay your application shutdown, you need to cooperate a > little more closely with it. Either attach your application shutdown code > as a callback to the sleep Deferred or move the sleep into the stopService > implementation of one of the services on your application and trigger the > remaining stopService calls (eg the stopService call on the MultiService > you mentioned) when the sleep Deferred fires there. > > Jean-Paul > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -- *Fabio Sangiovanni | System Integrator* *T* +39 0372 24525* F* +39 0372 800725 Via Dei Comizi Agrari 10, 26100 Cremona - Italia Milano - Cremona - San Francisco MailUp? La soluzione per l'invio di email e SMS mailup.it -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Fri Sep 5 06:13:55 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 05 Sep 2014 12:13:55 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> Message-ID: <20140905121355.20413.575214074.divmod.xquotient.2052@top> On 06:30 am, adi at roiban.ro wrote: >On 4 September 2014 12:59, wrote: >[snip] >>What would be really nice is if someone collected *all* of the >>complaints >>about `spawnProcess` into one place and integrated solutions to them >>into a >>design for a replacement. :) >> >>Jean-Paul > >Since I can not create wiki pages here are my notes: > >To do: > >* Always return a deferred. #2415 > >* Provide the option to timeout the execution... and on timeout errback >with > a dedicated failure #2415 > >* Add childs to the same process group #2415 > >* allways errback when failing to spawn (rather than raising OSError) >and > errback with a different failure when failing to spawn the executable >#4184 > >* on Unix provide a method to install SIGCHLD handler, independent of >the > general reactor.run() signal handlers. ... maybe related to #5710 > >* Allow passing Unicode environment (at least on Windows) maybe needed >for Python3 #5968 > >* Allow passing Unicode command and arguments on Windows (current not >supported by pywin32 CreateProcess) #6470... maybe needed for Python3 >#5968 > >* Improve fork/exec for speed and memory usage #5710 .... maybe by >creating > a auxiliary process which handles forks and child management. > >* Execute as a different account in Windows > >* Spawn only after reactor starts ? I am ok with forking before. > >* Support client endpoint? > >------- > >For a new design: > >* I like the ProcessProtocol API I don't. :) ProcessProtocol means you can't re-use any of your existing IProtocol implementations. Oops. It seems like a better API would let you say "Here is a protocol, hook its output up to fd 0 in the process, hook fd 1 in the process up to its input. Here is another protocol, hook fd 2 in the process up to its input" (this would probably be a common configuration - "speak some regular protocol over stdin/stdout, have a little special logic (probably logging) for stderr). On the other hand, it's possible to build this on top of ProcessProtocol and no one ever has... >* spawnProcess will always return a deferred... for backward >compatibility > we need to decide a name for this new method which returns a deferred. > Maybe createProcess or connectProcess > >* create spawnUnixProcess which has childFDs uid/gid > >* create spawnWindowsProcess which has Windows specific options > >* createProcess / connectProcess will call spawnUnixProcess or >spawnWindowsProcess > depending on current OS > >* ProcessProtocol.processEnded is similar to connectionLost.. > so maybe it can be rename it and add other methods so that >IProcessProtocol > can inherit from IProtocol. I guess that this will help with >connectors. Let's not introduce new cases where inheritance is encouraged. >------ > >I think that child management problem, does not affect the public API >so I am >not discussing it here. > >Timeouts are handled at protocol level and I see that we have 2 timeout >types: >total execution time and timeout after last child data was received. > >----- > >Does it make sense to put it on wiki or what should be the Twisted >Enhancement Proposal process? I think it's easier to maintain a document on the wiki than in threads on the mailing list. >Can I get 'create page' rights on Trac? Hm. I thought everyone had wiki edit rights (except for a small number of privileged pages) by default - so I'm not sure what's going on here or how to give you permissions. Perhaps something changed in the trac upgrade or as part of the recent ticket permission reconfiguration. Jean-Paul From adi at roiban.ro Fri Sep 5 07:14:32 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 5 Sep 2014 14:14:32 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140905121355.20413.575214074.divmod.xquotient.2052@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> <20140905121355.20413.575214074.divmod.xquotient.2052@top> Message-ID: On 5 September 2014 13:13, wrote: > On 06:30 am, adi at roiban.ro wrote: [snip] >> For a new design: >> >> * I like the ProcessProtocol API > > > I don't. :) ProcessProtocol means you can't re-use any of your existing > IProtocol implementations. Oops. > > It seems like a better API would let you say "Here is a protocol, hook its > output up to fd 0 in the process, hook fd 1 in the process up to its input. > Here is another protocol, hook fd 2 in the process up to its input" (this > would probably be a common configuration - "speak some regular protocol over > stdin/stdout, have a little special logic (probably logging) for stderr). > > On the other hand, it's possible to build this on top of ProcessProtocol and > no one ever has... I lied about ProcessProtocol API... I don't like the current API as it is ... but I like the idea of a single protocol. Maybe is not that hard to convert IProcessProtocol into IProtocol. create makeConnection, rename outRecevied to dataReceived, rename processEnded to connectionLost... rename errReceived to errorReceived I assume that most executables will implement a single protocol and adding another IProtocol to handle stderror seems complicated. I assume that the stderr protocol will want to share a lot of state/data with the stdout/stdin protocol. [snip] >> ----- >> >> Does it make sense to put it on wiki or what should be the Twisted >> Enhancement Proposal process? > > > I think it's easier to maintain a document on the wiki than in threads on > the mailing list. >> >> Can I get 'create page' rights on Trac? > > > Hm. I thought everyone had wiki edit rights (except for a small number of > privileged pages) by default - so I'm not sure what's going on here or how > to give you permissions. Perhaps something changed in the trac upgrade or > as part of the recent ticket permission reconfiguration. My bad... I tried trac/spawnProcess instead of trac/wiki/spawnProcess and I did not read the error message and assume that somehow I don't have WIKI_CREATE permissions. Here it is: https://twistedmatrix.com/trac/wiki/spawnProcess-refactoring -- Adi Roiban From p.mayers at imperial.ac.uk Fri Sep 5 08:09:37 2014 From: p.mayers at imperial.ac.uk (Phil Mayers) Date: Fri, 05 Sep 2014 15:09:37 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140905121355.20413.575214074.divmod.xquotient.2052@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> <20140905121355.20413.575214074.divmod.xquotient.2052@top> Message-ID: <5409C421.30306@imperial.ac.uk> On 05/09/14 13:13, exarkun at twistedmatrix.com wrote: > I don't. :) ProcessProtocol means you can't re-use any of your existing > IProtocol implementations. Oops. > > On the other hand, it's possible to build this on top of ProcessProtocol > and no one ever has... I do this about 3 times a year. Yes - ProcessProtocol must die! From glyph at twistedmatrix.com Fri Sep 5 12:47:09 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 5 Sep 2014 11:47:09 -0700 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140905121355.20413.575214074.divmod.xquotient.2052@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> <20140905121355.20413.575214074.divmod.xquotient.2052@top> Message-ID: <40FEB580-7E6A-4F19-9D40-DA4CA21C9A03@twistedmatrix.com> On Sep 5, 2014, at 5:13 AM, exarkun at twistedmatrix.com wrote: > It seems like a better API would let you say "Here is a protocol, hook its output up to fd 0 in the process, hook fd 1 in the process up to its input. Here is another protocol, hook fd 2 in the process up to its input" (this would probably be a common configuration - "speak some regular protocol over stdin/stdout, have a little special logic (probably logging) for stderr). +1 > On the other hand, it's possible to build this on top of ProcessProtocol and no one ever has... Of course we have _sort of_ done it a gajillion times: -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Sep 5 12:50:15 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 5 Sep 2014 11:50:15 -0700 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> Message-ID: <41C3CC5A-4F38-440E-9CDC-423A40CAAA5B@twistedmatrix.com> On Sep 4, 2014, at 11:30 PM, Adi Roiban wrote: > Since I can not create wiki pages here are my notes: I went to add you to wiki_editors, and I found that someone had already done so ;). Please go ahead and create a wiki page - perhaps under - so we can update it as consensus develops in this thread? Thanks for collecting all this information Adi, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac at twistedmatrix.com Sun Sep 7 00:05:15 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 07 Sep 2014 06:05:15 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1410069915.49.1225728082364432683@twistedmatrix.com An HTML attachment was scrubbed... URL: From kebin70 at gmail.com Sun Sep 7 01:51:36 2014 From: kebin70 at gmail.com (Kevin Mcintyre) Date: Sun, 7 Sep 2014 00:51:36 -0700 Subject: [Twisted-Python] pb.Copyable knowledge barrier In-Reply-To: References: <23AED729-A718-4887-A755-A9A044B1E898@twistedmatrix.com> Message-ID: Jelly performance factors below expectations. Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. At this point I'm unsure of what jelly actually does well. who's the JellyCon2014 keynote speaker? On Sat, Aug 9, 2014 at 6:58 PM, Glyph wrote: > On Aug 8, 2014, at 9:17 PM, Kevin Mcintyre wrote: > > correction. I was stupidly reprocessing list on remote copy side. Seeing > much better results now ~10K per second. > > > Whew. I don't know exactly how well I would expect this perform, but > those other results looked off by a couple orders of magnitude :-). > > -glyph > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Sep 7 10:43:18 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 7 Sep 2014 09:43:18 -0700 Subject: [Twisted-Python] pb.Copyable knowledge barrier In-Reply-To: References: <23AED729-A718-4887-A755-A9A044B1E898@twistedmatrix.com> Message-ID: <3CDFCECA-735B-4F59-A361-53C9760C529B@twistedmatrix.com> On Sep 7, 2014, at 12:51 AM, Kevin Mcintyre wrote: > Jelly performance factors below expectations. I am not clear on the significance of this statement. > Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. Asking whether your copy is good any more is a PB-level task. Jelly itself is a separate layer which is about getting the right data to the right place, not keeping it updated. > My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. "problematic" is passive voice :-). What are the problems? > At this point I'm unsure of what jelly actually does well. Malheureusement, Jelly's main claim to fame is "it's better than pickle". But with all this renewed interest in PB perhaps we'll get pre-deserialization schema enforcement and type checking, and then it will have some real advantages :). > who's the JellyCon2014 keynote speaker? One day we will have a conference - Tx/Rx, of course - and there will be a talk on this. But first we need several people in the community who wants their contribution to Twisted to be organization and community oriented rather than code ;-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Sun Sep 7 19:26:53 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 03:26:53 +0200 Subject: [Twisted-Python] Python3: should paths be bytes or str? Message-ID: <1469549.Iq1ZMIjpzY@i5> The porting guide says No byte paths in sys.path. I am not sure what this means, I would assume that file paths should always be native strings. Or does it mean that sys.path must only contain ascii bytes? doc for FilePath says On both Python 2 and Python 3, paths can only be bytes. and svn commit 35410 by itamarst changed the doc for some path functions in python/filepath.py from str to bytes but not all of them: fgrep 'type path:' filepath.py @type path: L{str} @type path: L{str} @type path: L{bytes} @type path: L{bytes} @type path: L{bytes} I stumbled upon this while trying to find out how much work it might be to make bin/trial run with python3 admin/run-python3-tests already passes for all twisted.spread related tests but I still need to clean up a lot. after adding an assert to FilePath.__init__, python3 bin/trial ... gives File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run config.parseOptions() File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in parseOptions self.postOptions() File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in postOptions _BasicOptions.postOptions(self) File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in postOptions self['reporter'] = self._loadReporterByName(self['reporter']) File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in _loadReporterByName for p in plugin.getPlugins(itrial.IReporter): File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins allDropins = getCache(package) File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache mod = getModule(module.__name__) File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in getModule return theSystemPath[moduleName] File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in __getitem__ self._findEntryPathString(moduleObject)), File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in _findEntryPathString if _isPackagePath(FilePath(topPackageObj.__file__)): File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in __init__ assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) AssertionError: path must be bytes: '/home/wr/ssdsrc/Twisted/twisted/__init__.py' -- Wolfgang From exarkun at twistedmatrix.com Sun Sep 7 20:14:10 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 02:14:10 -0000 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <1469549.Iq1ZMIjpzY@i5> References: <1469549.Iq1ZMIjpzY@i5> Message-ID: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> On 01:26 am, wolfgang.kde at rohdewald.de wrote: >The porting guide says > >No byte paths in sys.path. What porting guide is that? > >doc for FilePath says > On both Python 2 and Python 3, paths can only be bytes. > > >I stumbled upon this while trying to find out how much work it might be >to make bin/trial run with python3 > >admin/run-python3-tests already passes for all twisted.spread related >tests but I still need to clean up a lot. > >after adding an assert to FilePath.__init__, python3 bin/trial ... >gives > > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in >run > config.parseOptions() > File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in >parseOptions > self.postOptions() > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in >postOptions > _BasicOptions.postOptions(self) > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in >postOptions > self['reporter'] = self._loadReporterByName(self['reporter']) > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in >_loadReporterByName > for p in plugin.getPlugins(itrial.IReporter): > File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in >getPlugins > allDropins = getCache(package) > File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in >getCache > mod = getModule(module.__name__) > File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in >getModule > return theSystemPath[moduleName] > File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in >__getitem__ > self._findEntryPathString(moduleObject)), > File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in >_findEntryPathString > if _isPackagePath(FilePath(topPackageObj.__file__)): > File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, >in __init__ > assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) >AssertionError: path must be bytes: >'/home/wr/ssdsrc/Twisted/twisted/__init__.py' If paths are being represented using unicode somewhere and you want to use them with FilePath then you have to encode them (or you have to add unicode path support to FilePath and let FilePath encode them). Unfortunately it's not entirely obvious how to make FilePath support unicode paths since not all platforms Twisted supports represent filesystem paths using unicode. The choice python-dev made to bridge this gap was the creation of the "surrogateescape" error handler for the UTC-8 codec. This lets you pretend that any time you need to convert between bytes and unicode the correct codec is UTF-8 (with this special error handler). It's not clear this was a good choice (since the result is unicode strings that may contain garbage which will confuse other software) but it's also not clear it's possible for Twisted to try to make any other choice (at some point Twisted has to interoperate with the path-related APIs in Python itself - `sys.path`, for example). Not sure if that helps you at all. Maybe it outlines the problem a little more clearly, at least. Jean-Paul From wolfgang.kde at rohdewald.de Mon Sep 8 03:19:37 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 11:19:37 +0200 Subject: [Twisted-Python] PEP3131: non-ascii identifiers Message-ID: <2095383.3s724Eqe0K@i5> This does not seem to be supported by Python yet. Should that be enabled at all? If one process with PY3 sends such identifiers to a separate process with PY2, that will fail. I am not sure if that would be a problem, whoever uses this must make sure PY3 is used everywhere. If this should be forbidden, I will add a test to test_pb for this. And of course somebody should document that somewhere. There more PEP3131 is used, the more users will fall into this trap. If this should be enabled (which I think is not difficult, at least for pb): At least the patch below will be needed (only for PY3), maybe it is already sufficient. Given that nativeString and networkString are always used (done that for pb). networkString may then return bytes with the high bit set But since networkString is called in many places I want to ask and make sure that it may really be changed this way. https://twistedmatrix.com/documents/14.0.0/core/specifications/banana.html does not speak against it, so I wonder why networkString has that limitation to 7bit. concrete banana-encoded example, from modified test_pb: (the method name is getSimple?) test_pb still passes with patched nativeString/networkString (but I only have one test for this so far, test_refcount). b'\x07\x80\x07\x82message\x01\x81\x03\x82foo\x0b\x82getSimple\xc3\xa4\x01\x81\x01\x80\x05\x82tuple\x01\x80\n\x82dictionary' diff --git twisted/python/compat.py twisted/python/compat.py index 6f76c39..6919cf6 100644 --- twisted/python/compat.py +++ twisted/python/compat.py @@ -348,10 +348,9 @@ def nativeString(s): raise TypeError("%r is neither bytes nor unicode" % s) if _PY3: if isinstance(s, bytes): - return s.decode("ascii") + return s.decode("utf-8") else: - # Ensure we're limited to ASCII subset: - s.encode("ascii") + return s else: if isinstance(s, unicode): return s.encode("ascii") @@ -428,7 +427,7 @@ if _PY3: def networkString(s): if not isinstance(s, unicode): raise TypeError("Can only convert text to bytes on Python 3, I got %r" % (s,)) - return s.encode('ascii') + return s.encode('utf-8') def networkChar(integer): """ -- Wolfgang From wolfgang.kde at rohdewald.de Mon Sep 8 03:20:05 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 11:20:05 +0200 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: <1726686.JYd9teDfKp@i5> Am Montag, 8. September 2014, 02:14:10 schrieb exarkun at twistedmatrix.com: > On 01:26 am, wolfgang.kde at rohdewald.de wrote: > >The porting guide says > > > >No byte paths in sys.path. > > What porting guide is that? https://twistedmatrix.com/trac/wiki/Plan/Python3 see the reviewer check list > it's also not clear it's possible for Twisted to try to make any other > choice (at some point Twisted has to interoperate with the path-related > APIs in Python itself - `sys.path`, for example). Am Montag, 8. September 2014, 02:14:10 schrieb exarkun at twistedmatrix.com: > If paths are being represented using unicode somewhere and you want to > use them with FilePath then you have to encode them (or you have to add > unicode path support to FilePath and let FilePath encode them). I always thought module names must be ascii-only but now found PEP3131 So we should do the same for twisted.python.modules as in those other places grepped below. And add that assert isinstance(path, bytes) for PY3 in FilePath. And maybe this should go into the above check list? I have no edit rights in the Wiki. BUT ? I will stop trying to port python/modules.py, the usage of the same strings for both module names and file paths is too much interwoven, I do not want to touch that. My feeling is that file names should all be unicode, converting them only where needed. But then I am not an expert about this. Next problem - PEP3131. See separate mail. grep -r __file__ | grep encode web/test/test_webclient.py:serverPEM = FilePath(test.__file__.encode("utf-8")).sibling(b'server.pem') test/ssl_helpers.py:certPath = nativeString(FilePath(__file__.encode("utf-8") test/test_setup.py:if not FilePath(twisted.__file__.encode('utf-8')).sibling(b'topfiles').child(b'setup.py').exists(): python/test/test_deprecate.py: self.assertEqual(FilePath(module.__file__.encode("utf-8")), internet/test/test_gireactor.py: path = FilePath(__file__.encode("utf-8")).sibling( -- Wolfgang From techtonik at gmail.com Mon Sep 8 04:26:58 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 8 Sep 2014 13:26:58 +0300 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: On Mon, Sep 8, 2014 at 5:14 AM, wrote: > On 01:26 am, wolfgang.kde at rohdewald.de wrote: >> >> The porting guide says >> >> No byte paths in sys.path. > > > What porting guide is that? >> >> >> doc for FilePath says >> On both Python 2 and Python 3, paths can only be bytes. >> >> >> I stumbled upon this while trying to find out how much work it might be >> to make bin/trial run with python3 >> >> admin/run-python3-tests already passes for all twisted.spread related >> tests but I still need to clean up a lot. >> >> after adding an assert to FilePath.__init__, python3 bin/trial ... gives >> >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run >> config.parseOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in >> parseOptions >> self.postOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in >> postOptions >> _BasicOptions.postOptions(self) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in >> postOptions >> self['reporter'] = self._loadReporterByName(self['reporter']) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in >> _loadReporterByName >> for p in plugin.getPlugins(itrial.IReporter): >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins >> allDropins = getCache(package) >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache >> mod = getModule(module.__name__) >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in >> getModule >> return theSystemPath[moduleName] >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in >> __getitem__ >> self._findEntryPathString(moduleObject)), >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in >> _findEntryPathString >> if _isPackagePath(FilePath(topPackageObj.__file__)): >> File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in >> __init__ >> assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) >> AssertionError: path must be bytes: >> '/home/wr/ssdsrc/Twisted/twisted/__init__.py' > > > If paths are being represented using unicode somewhere and you want to use > them with FilePath then you have to encode them (or you have to add unicode > path support to FilePath and let FilePath encode them). > > Unfortunately it's not entirely obvious how to make FilePath support unicode > paths since not all platforms Twisted supports represent filesystem paths > using unicode. It really depends on filesystem, not on a platform. Platform just makes sure that you won't shoot it in the foot. So to behave good you need to translate you path knowledge to platform knowledge when you have to make change. In data transformation theory that may mean: [ ] get data about path in native format [ ] detect the source encoding of filesystem [ ] figure out if you can work with native format [ ] python 2 way - just work with bytes [ ] python 3 way - look if native filesystem format is convertible to unicode [ ] if conversion is symmetrical - operate in unicode [ ] if not convertible, alternatives (options, switches) [ ] fail and explain why to user in user actionable manner (don't use ?) [ ] use some symmetrical mapping to unicode and mark path objects as `mapped` so that there is a trace of hacks on filepaths [ ] provide API on objects without ability to use names directly [ ] transform the name to a "safe" valid value loosing the original name and explain the user why and what happened to the old name > The choice python-dev made to bridge this gap was the creation of the > "surrogateescape" error handler for the UTC-8 codec. This lets you pretend > that any time you need to convert between bytes and unicode the correct > codec is UTF-8 (with this special error handler). > > It's not clear this was a good choice (since the result is unicode strings > that may contain garbage which will confuse other software) but it's also > not clear it's possible for Twisted to try to make any other choice (at some > point Twisted has to interoperate with the path-related APIs in Python > itself - `sys.path`, for example). > > Not sure if that helps you at all. Maybe it outlines the problem a little > more clearly, at least. I think that it should be a choice of application maintainers. If they want to create files with dots at the end, Windows allows this, but doesn't support it through standard WinAPI calls, because of FAT. But you can use special path transformation prefix \\?\ to do this on NTFS or on a remote machine. In networking OS plays less role, but the new choice for every platform filesystem is not clear for users. They don't realize where the problem comes from. In the end it all depends on FS first, then on OS API (which can be skipped thanks to direct disk access), then on FS library, then on application. Application should be able to opt-in for handling all possible cases, or just "safe" subset or something middle, so a task of a framework is to just describe the problem well and ensure that people can make their choice. -- anatoly t. From exarkun at twistedmatrix.com Mon Sep 8 04:54:44 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 10:54:44 -0000 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <2095383.3s724Eqe0K@i5> References: <2095383.3s724Eqe0K@i5> Message-ID: <20140908105444.20413.527660985.divmod.xquotient.2139@top> On 09:19 am, wolfgang.kde at rohdewald.de wrote: >This does not seem to be supported by Python yet. > >Should that be enabled at all? >If one process with PY3 sends such identifiers to >a separate process with PY2, that will fail. I am not >sure if that would be a problem, whoever uses this must >make sure PY3 is used everywhere. This is why we will *not* change the PB wire protocol as part of the porting work. The wire protocol will remain the same whether you are using Python 2 or Python 3 to run your program or. This is the point of a protocol, after all. It is to let two programs communicate with each other. >If this should be forbidden, I will add a test to >test_pb for this. And of course somebody should document that >somewhere. There more PEP3131 is used, the more users will >fall into this trap. I'm not exactly sure what you mean here. Using unicode where only bytes are allowed is probably already forbidden throughout PB. > >If this should be enabled (which I think is not difficult, >at least for pb): > >At least the patch below will be needed (only for PY3), >maybe it is already sufficient. Given that nativeString >and networkString are always used (done that for pb). > >networkString may then return bytes with the high bit set Definitely not. >But since networkString is called in many places I want to ask and >make sure that it may really be changed this way. > > >https://twistedmatrix.com/documents/14.0.0/core/specifications/banana.html >does not speak against it, so I wonder why networkString has that >limitation >to 7bit. That is the sole purpose of `networkString`. It is a work-around for the inconvenient fact that Python changed the meaning of the string literal syntax from bytes to unicode. > >concrete banana-encoded example, from modified test_pb: (the method >name is getSimple?) >test_pb still passes with patched nativeString/networkString (but I >only have one test for this so far, test_refcount). > >b'\x07\x80\x07\x82message\x01\x81\x03\x82foo\x0b\x82getSimple\xc3\xa4\x01\x81\x01\x80\x05\x82tuple\x01\x80\n\x82dictionary' > > >diff --git twisted/python/compat.py twisted/python/compat.py >index 6f76c39..6919cf6 100644 >--- twisted/python/compat.py >+++ twisted/python/compat.py >@@ -348,10 +348,9 @@ def nativeString(s): > raise TypeError("%r is neither bytes nor unicode" % s) > if _PY3: > if isinstance(s, bytes): >- return s.decode("ascii") >+ return s.decode("utf-8") > else: >- # Ensure we're limited to ASCII subset: >- s.encode("ascii") >+ return s > else: > if isinstance(s, unicode): > return s.encode("ascii") >@@ -428,7 +427,7 @@ if _PY3: > def networkString(s): > if not isinstance(s, unicode): > raise TypeError("Can only convert text to bytes on Python >3, I got %r" % (s,)) >- return s.encode('ascii') >+ return s.encode('utf-8') > > def networkChar(integer): > """ This change definitely won't be acceptable. It completely removes the feature `networkString` exists to provide: verifying that strings that might be either unicode or bytes can still be implicitly combined into bytes. Can you point out the specific places where you think PB needs to start using UTF-8 instead of ASCII? Those are the places that need to be fixed, not the underlying porting helpers they happen to use. Jean-Paul From wolfgang.kde at rohdewald.de Mon Sep 8 05:44:30 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 13:44:30 +0200 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <20140908105444.20413.527660985.divmod.xquotient.2139@top> References: <2095383.3s724Eqe0K@i5> <20140908105444.20413.527660985.divmod.xquotient.2139@top> Message-ID: <1572241.mjvUuZoz8Z@i5> Am Montag, 8. September 2014, 10:54:44 schrieb exarkun at twistedmatrix.com: > Can you point out the specific places where you think PB needs to start > using UTF-8 instead of ASCII? Those are the places that need to be > fixed, not the underlying porting helpers they happen to use. My question is whether PEP3131 should be supported or not. After all, it is an integral part of Python3. I have no opinion about this, I just want to point it out as a potential problem. So you say it should not be supported, and I have no problem with that. Just to be clear about it: In Python3 you can legally write def ?nic?deMethod?Name(self): or see https://mail.python.org/pipermail/python-3000/2007-June/008172.html for a nice Russian example. but PB cannot transfer those identifiers. Which is not what the user would expect. This should be clearly communicated somewhere in the docs, IMHO. -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 8 06:04:46 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 12:04:46 -0000 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <1572241.mjvUuZoz8Z@i5> References: <2095383.3s724Eqe0K@i5> <20140908105444.20413.527660985.divmod.xquotient.2139@top> <1572241.mjvUuZoz8Z@i5> Message-ID: <20140908120446.20413.1993708093.divmod.xquotient.2146@top> On 11:44 am, wolfgang.kde at rohdewald.de wrote: >Am Montag, 8. September 2014, 10:54:44 schrieb >exarkun at twistedmatrix.com: >>Can you point out the specific places where you think PB needs to >>start >>using UTF-8 instead of ASCII? Those are the places that need to be >>fixed, not the underlying porting helpers they happen to use. > >My question is whether PEP3131 should be supported or not. After all, >it is an integral part of Python3. I have no opinion about this, >I just want to point it out as a potential problem. > >So you say it should not be supported, and I have no problem with that. > >Just to be clear about it: > >In Python3 you can legally write > >def ?nic?deMethod?Name(self): > >or see >https://mail.python.org/pipermail/python-3000/2007-June/008172.html >for a nice Russian example. > >but PB cannot transfer those identifiers. Which is not what the >user would expect. PB supports unicode perfectly well and has for many years. This is why I asked which specific part of PB has a problem. I don't see why you think unicode is a problem here. This doesn't mean there isn't a problem - after all, you've been looking at the code and trying to make it work on Python 3 and I haven't - but you haven't made it clear what that problem is. A failing unit test or a minimal example () would communicate this most clearly, but perhaps you can just mention a specific API and give an incomplete example of how it will fail when it runs up against the changes defined by PEP 3131. Jean-Paul > >This should be clearly communicated somewhere in the docs, IMHO. > >-- >Wolfgang From wolfgang.kde at rohdewald.de Mon Sep 8 07:07:19 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 15:07:19 +0200 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <20140908120446.20413.1993708093.divmod.xquotient.2146@top> References: <2095383.3s724Eqe0K@i5> <1572241.mjvUuZoz8Z@i5> <20140908120446.20413.1993708093.divmod.xquotient.2146@top> Message-ID: <1573940.73e26bT3Za@i5> Am Montag, 8. September 2014, 12:04:46 schrieb exarkun at twistedmatrix.com: > PB supports unicode perfectly well and has for many years. This is why > I asked which specific part of PB has a problem. PB transfers names of methods and classes as bytes, not as unicode. Which is logical since PY2 does not support unicode identifiers, and bytes is already a native PY2 string. Unicode is only used for content. It not yet always clear to me what is content and what is a formal string like method names or the *_atom strings which must be bytes, this needs more testing. I guess I should patch banana.py such that it dumps all it encodes or decodes into one file, so I can compare output from PY2/PY3 tests. I was assuming that suddenly transferring method names as unicode would really be a break of wire protocol stability, or do you think otherwise? If you think this is acceptable, I will check if the existing twisted code can handle getting those as unicode without source code changes. Not sure. Just tested this with Python2.6, and I am surprised that it works: >>> getattr(A,u'x') Supporting PEP3131 would only introduce a backward-incompatibility. Of course you are right that this is not part of porting. Right now I have a long list of small unsorted git commits, I will have to do a lot of reshuffling and cleaning before I will ask you how to get it into the official codebase. Not all of those commits are strictly porting, some just clean the code, making the porting commits simpler. > A failing unit test or a minimal example () would > communicate this most clearly, but perhaps you can just mention a > specific API and give an incomplete example of how it will fail when it > runs up against the changes defined by PEP 3131. see my first mail in this thread: take test_pb.py, rename getSimple to getSimple?, run the test. -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 8 08:26:19 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 14:26:19 -0000 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <1573940.73e26bT3Za@i5> References: <2095383.3s724Eqe0K@i5> <1572241.mjvUuZoz8Z@i5> <20140908120446.20413.1993708093.divmod.xquotient.2146@top> <1573940.73e26bT3Za@i5> Message-ID: <20140908142619.20413.1258694452.divmod.xquotient.2237@top> On 01:07 pm, wolfgang.kde at rohdewald.de wrote: >Am Montag, 8. September 2014, 12:04:46 schrieb >exarkun at twistedmatrix.com: >>PB supports unicode perfectly well and has for many years. This is >>why >>I asked which specific part of PB has a problem. > >PB transfers names of methods and classes as bytes, not as unicode. PB actually transfers *everything* as bytes. ;) Bytes are the only thing you can send over a socket. If you have anything else - integers, unicode, whatever - you have to encode them as bytes first. As far as I know, the PB protocol is not specified apart from the implementation in Twisted (and this being the only implementation (I am intentionally disregarding TwistedJava) it must serve as the specification, I think). What you meant above, I think, is that PB represents method names as bytes at the banana layer. That is, when you want to call a remote method, you indicate its name by supplying bytes to the banana encoding layer - not unicode (which is good because banana doesn't actually support unicode at all, that's a jelly feature). This does indeed mean we don't simply want to start sending unicode to refer to methods by name - because we can't! At least, not unless we extend banana to support a new type which we probably don't want to do - that would be another incompatible protocol change and so not allowed (since it could break interoperability between different implementations of PB). So, it is necessary to continue to represent method names using bytes. This is fairly easily done. On Python 3, encode any unicode strings which represent method names (using a well-known encoding, probably UTF-8) when making the call and decode them in the same way when dispatching those calls. This can almost be done at the application level: # Some Python 3 code def remote_?(self): pass ref.callRemote(u"?".encode("utf-8")) except that Python 3 has actually changed to enforce the type of the second argument of getattr - if it is not a unicode string then a TypeError is raised - so it's not possible to make the decoding step happen (which one might otherwise have done using `__getattr__` or by adding the encoded name of the method to the class dictionary). So if it is going to be supported in Twisted's PB API then that support probably needs to be in Twisted's PB implementation. The same general idea applies, though. Just move the encoding into the implementation of `callRemote`: def callRemote(self, _name, ...): ... _name.encode("utf-8") ... And add a corresponding decode to the other side (probably in `_recvMessage`). This would make the Python 3 PB API be "method names are unicode strings" which makes sense considering the decisions that were made for Python 3. Note that it does not change the wire protocol - method names are still bytes at the banana level. Or does it? These bytes were previously always an ASCII subset. Is expanding out of the ASCII range an incompatible change? What could this break? Let's say you have Python 2 talking to Python 2. It's already possible to construct a method call like this: class Foo(Referenceable): def remote_foo(self): pass setattr(Foo, u"remote_foo?".encode("utf-8"), Foo.remote_foo) ... ref.callRemote(u"remote_foo?".encode("utf-8")) This actually works. Python 2 doesn't much care about how you name your class attributes. PB doesn't care that the high bit is set in one or more of the bytes in the method name. It all just works. So let's say you have Python 2 talking to Python 3 instead. In Python 3, you can't do that setattr() call (the language and runtime disallow it). But you can have `def remote_foo?(self)` instead. If PB on Python 3 decodes the method name before dispatching it (using UTF-8) then again things work. And if you reverse the situation and PB on Python 3 encodes the method name before sending it, then Python 2 is still happen because it can operate on that UTF-8 encoded byte string. Finally, if Python 3 talks to Python 3 then it also works because the sending side encodes and the receiving side decodes. So we get to make a judgement call here, I think. Without a specification there's no objectively correct answer. So, because the current implementation is actually perfectly compatible with non-ASCII bytes - even though the intent is clearly that you would never have those - combined with the point that I made above, that there are no other PB implementations, I suspect it's fine to expand beyond ASCII here because it won't actually break any real world programs. The only case that I can think of that actually would be a problem is the case where someone is already sending non-ASCII, non-UTF-8 method names around. These might decode wrong or might fail to decode at all. I don't think this is likely enough to worry about - but maybe someone who is doing this will speak up and prove me wrong. ;) And to repeat myself a bit, none of this should change the Python 2 PB API. It should continue accepting bytes - because that's what it always accepted. Separately, we could introduce a new feature to support unicode on Python 2. This would be done in the usual way for introducing new features into any Twisted APIs (and there aren't reasonable backwards compatibility considerations here as far as I can tell). The point would be to make it a little more convenient for Python 2 applications to interact with other PB applications that have decided to use unicode method names. Lastly, on another topic, I am subscribed to the mailing list - you don't have to cc me on your replies. Jean-Paul > >Which is logical since PY2 does not support unicode identifiers, >and bytes is already a native PY2 string. Unicode is only used >for content. It not yet always clear to me what is content and >what is a formal string like method names or the *_atom strings >which must be bytes, this needs more testing. > >I guess I should patch banana.py such that it dumps all it encodes >or decodes into one file, so I can compare output from PY2/PY3 tests. > >I was assuming that suddenly transferring method names as unicode >would really be a break of wire protocol stability, or do you >think otherwise? If you think this is acceptable, I will check >if the existing twisted code can handle getting those as unicode >without source code changes. Not sure. Just tested this with >Python2.6, and I am surprised that it works: >>>>getattr(A,u'x') > > > >Supporting PEP3131 would only introduce a backward-incompatibility. > >Of course you are right that this is not part of porting. > >Right now I have a long list of small unsorted git commits, I will >have to do a lot of reshuffling and cleaning before I will ask >you how to get it into the official codebase. >Not all of those commits are strictly porting, some just clean >the code, making the porting commits simpler. >>A failing unit test or a minimal example () would >>communicate this most clearly, but perhaps you can just mention a >>specific API and give an incomplete example of how it will fail when >>it >>runs up against the changes defined by PEP 3131. > >see my first mail in this thread: take test_pb.py, rename getSimple >to getSimple?, run the test. > >-- >Wolfgang > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python at twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From kebin70 at gmail.com Mon Sep 8 15:36:53 2014 From: kebin70 at gmail.com (Kevin Mcintyre) Date: Mon, 8 Sep 2014 14:36:53 -0700 Subject: [Twisted-Python] pb.Copyable knowledge barrier In-Reply-To: <3CDFCECA-735B-4F59-A361-53C9760C529B@twistedmatrix.com> References: <23AED729-A718-4887-A755-A9A044B1E898@twistedmatrix.com> <3CDFCECA-735B-4F59-A361-53C9760C529B@twistedmatrix.com> Message-ID: Jelly performance factors below expectations. I am not clear on the significance of this statement. Not significant, just an observation. I've made this statement before, backed-off after finding errors on my code, but after fixing I see performance across the bus that still seem slow. "Seem slow" lacks any scientific backing. I would be curious to hear use cases for PB/Jelly that went beyond the docs. Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. Asking whether your copy is good any more is a PB-level task. Jelly itself is a separate layer which is about getting the right data to the right place, not keeping it updated. Yes PB level, I guess I'm looking for a convention where 1 does not exist. In the renewed interest the comments have tended to overlook copyable, or see copyable as being flushed out. My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. "problematic" is passive voice :-). What are the problems? Only parroting what I've read on cacheable, haven't delved into yet. At this point I'm unsure of what jelly actually does well. Malheureusement, Jelly's main claim to fame is "it's better than pickle". But with all this renewed interest in PB perhaps we'll get pre-deserialization schema enforcement and type checking, and then it will have some real advantages :). Agreed. This is what I looked to Jelly/PB for, having found so much in twisted that simplifies. who's the JellyCon2014 keynote speaker? *One* day we will have a conference - Tx/Rx, of course - and there will be a talk on this. But first we need several people in the community who wants their contribution to Twisted to be organization and community oriented rather than code ;-). On Sun, Sep 7, 2014 at 9:43 AM, Glyph wrote: > > On Sep 7, 2014, at 12:51 AM, Kevin Mcintyre wrote: > > Jelly performance factors below expectations. > > > I am not clear on the significance of this statement. > > Can we say Copyable is the lowest order jelly? The notion that a copy > holder can't ask "is my copy good anymore?" makes it so. Essentially root > says, I'd prefer not to repeat unit of work nor keep track of the resulting > copies, here have the original or resulting copy. > > > Asking whether your copy is good any more is a PB-level task. Jelly > itself is a separate layer which is about getting the right data to the > right place, not keeping it updated. > > My main issue is a copy-holder calling for a copy to determine is the copy > is good anymore. I know, see cacheable but it's problematic. > > > "problematic" is passive voice :-). What are the problems? > > At this point I'm unsure of what jelly actually does well. > > > Malheureusement, Jelly's main claim to fame is "it's better than pickle". > But with all this renewed interest in PB perhaps we'll get > pre-deserialization schema enforcement and type checking, and then it will > have some real advantages :). > > who's the JellyCon2014 keynote speaker? > > > *One* day we will have a conference - Tx/Rx, of course - and there will > be a talk on this. But first we need several people in the community who > wants their contribution to Twisted to be organization and community > oriented rather than code ;-). > > -glyph > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Sep 9 00:01:50 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 8 Sep 2014 23:01:50 -0700 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: On Sep 7, 2014, at 7:14 PM, exarkun at twistedmatrix.com wrote: > On 01:26 am, wolfgang.kde at rohdewald.de wrote: >> The porting guide says >> >> No byte paths in sys.path. > > What porting guide is that? >> >> doc for FilePath says >> On both Python 2 and Python 3, paths can only be bytes. >> >> >> I stumbled upon this while trying to find out how much work it might be >> to make bin/trial run with python3 >> >> admin/run-python3-tests already passes for all twisted.spread related >> tests but I still need to clean up a lot. >> >> after adding an assert to FilePath.__init__, python3 bin/trial ... gives >> >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run >> config.parseOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in parseOptions >> self.postOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in postOptions >> _BasicOptions.postOptions(self) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in postOptions >> self['reporter'] = self._loadReporterByName(self['reporter']) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in _loadReporterByName >> for p in plugin.getPlugins(itrial.IReporter): >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins >> allDropins = getCache(package) >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache >> mod = getModule(module.__name__) >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in getModule >> return theSystemPath[moduleName] >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in __getitem__ >> self._findEntryPathString(moduleObject)), >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in _findEntryPathString >> if _isPackagePath(FilePath(topPackageObj.__file__)): >> File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in __init__ >> assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) >> AssertionError: path must be bytes: '/home/wr/ssdsrc/Twisted/twisted/__init__.py' > > If paths are being represented using unicode somewhere and you want to use them with FilePath then you have to encode them (or you have to add unicode path support to FilePath and let FilePath encode them). > > Unfortunately it's not entirely obvious how to make FilePath support unicode paths since not all platforms Twisted supports represent filesystem paths using unicode. > > The choice python-dev made to bridge this gap was the creation of the "surrogateescape" error handler for the UTC-8 codec. This lets you pretend that any time you need to convert between bytes and unicode the correct codec is UTF-8 (with this special error handler). > > It's not clear this was a good choice (since the result is unicode strings that may contain garbage which will confuse other software) but it's also not clear it's possible for Twisted to try to make any other choice (at some point Twisted has to interoperate with the path-related APIs in Python itself - `sys.path`, for example). > > Not sure if that helps you at all. Maybe it outlines the problem a little more clearly, at least. The problem with making FilePath support unicode is that we want to provide an interface that applications can rely upon, specified in terms of specific types (bytes or text) so that when you get an IFilePath you know what you can do with it. As it is currently implemented, FilePath exposes its internal representation fairly directly, most notably as the ?.path? attribute, but also in the return-type of methods like "basename" and "segmentsFrom". FilePath doesn't exactly "support" unicode, in that it's specifically documented not to, but it's sort of hard to tell, since you can instantiate one with a unicode string in both python 2 and python 3, and get (apparently) correct results out of it for some methods. However, methods that need a string constant as part of their implementation, like siblingExtensionSearch and globChildren, will break unceremoniously when presented with unicode. Another decision that python-dev made to bridge the gap was to randomly allow different string types be passed to platform APIs, like this: >>> import os >>> os.listdir(u".") ['a', 'b', 'c'] >>> os.listdir(b".") [b'a', b'b', b'c'] >>> os.path.basename(b".") b'.' >>> os.path.basename(".") '.' This implies a parallel structure might be possible for FilePath: if you pass its constructor bytes, you get a BytesFilePath; if you pass it text, you get a TextFilePath. You can't mix the two, and once you've chosen a path you can't choose a different one. IFilePath could then document that all of its existing methods have the return type of "whatever got passed to __init__" (which is what the current implementation does about 2/3 of the time anyway on py3, and about 9/10 of the time on py2; we would just be making it work intentionally, all the way). But, it would then be possible to give BytesFilePath a "asText()" method and vice versa "asBytes()" - since it's the filesystem, metadata about encodings exists outside your program and you would not need to guess at encodings, you'd simply indicate what return value you'd like from methods like .basename() et. al. The more I think about this, the more I like it - it's a bit of annoying and subtle implementation work, but I think it would supply the behavior that most people want, remain compatible with most of the existing unspecified behavior, and it would address clean text/bytes separation without having a giant deprecation cycle and inventing a new interface. It's also the sort of implementation work which, after some discussion and consideration, we could be reasonably sure is *correct* rather than guessing at things. Thoughts? -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Sep 9 15:27:35 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 9 Sep 2014 14:27:35 -0700 Subject: [Twisted-Python] a brief note about buildbots Message-ID: Hi Twistosophists, Some of you doing reviews (you ARE all doing code reviews, right!?!?) might have noticed that some of the boxes on the left side of of the supported builds page were dark grey, meaning that they were offline. I spent some time over the last couple of days doing some administrative housekeeping and rebooting computers, and all of our supported builders are now back online! This makes me feel much better about the possibility of doing a release again someday :-). Speaking of colors of boxes on that page, though, we should probably try to turn some of them from red to green... -glyph P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're running many these tests on, and (B) my time to do this sort of maintenance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Tue Sep 9 16:56:18 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 10 Sep 2014 01:56:18 +0300 Subject: [Twisted-Python] a brief note about buildbots In-Reply-To: References: Message-ID: On Wed, Sep 10, 2014 at 12:27 AM, Glyph Lefkowitz wrote: > Hi Twistosophists, > > Some of you doing reviews (you ARE all doing code reviews, right!?!?) might > have noticed that some of the boxes on the left side of of the supported > builds page were dark grey, meaning that they were offline. > > I spent some time over the last couple of days doing some administrative > housekeeping and rebooting computers, and all of our supported builders are > now back online! This makes me feel much better about the possibility of > doing a release again someday :-). > > Speaking of colors of boxes on that page, though, we should probably try to > turn some of them from red to green... > > -glyph > > P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're > running many these tests on, and (B) my time to do this sort of maintenance. Long Live Glyph! (Rackspace probably receives their credits in the market share) I presume you're using Firefox for all browsing needs. Am I right? =) In Chrome I've noticed an empty box with missing image at the highscore page leading to https://sm7.sitemeter.com/meter.asp?site=sm7twistedmatrix And when I open linked buildbot page at: https://buildbot.twistedmatrix.com/boxes-supported?branch=trunk&num_builds=4 Chrome says that "Attackers might be trying to steal your information from buildbot.twistedmatrix.com (for example, passwords, messages, or credit cards)." Not your, of course, but mine, so I am double worried about that invalid security certificate issued to www.twistedmatrix.com and deployed on buildbot.t.c BTW, how do you count tests? Is this config available somewhere? -- anatoly t. From wolfgang.kde at rohdewald.de Tue Sep 9 20:40:00 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 10 Sep 2014 04:40 +0200 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: <5928596.KCb0BbjCGC@i5> Am Montag, 8. September 2014, 23:01:50 schrieb Glyph: > This implies a parallel structure might be possible for FilePath: > if you pass its constructor bytes, you get a BytesFilePath; > if you pass it text, you get a TextFilePath. You can't mix the two, > and once you've chosen a path you can't choose a different one. this sounds good. After the port of pb to PY3 is done, I might have a look at it, it probably should be done before trying to port modules.py and trial to PY3. -- Wolfgang From wolfgang.kde at rohdewald.de Tue Sep 9 20:50:35 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 10 Sep 2014 04:50:35 +0200 Subject: [Twisted-Python] jelly and instances Message-ID: <4250196.vmN52vaFvk@i5> doc says: Instance: s is an instance of UserString.UserString, with a __dict__ {'data': 'hello'}:: [\"UserString.UserString\", ['dictionary', ['data', 'hello']]] There is no test for instances. To be precise, no test ever enters jelly._unjelly_instance() I also cannot find any place in jelly.py where an Instance would be jellied. There is instance_atom = 'instance' but instance_atom is not used anywhere. Is this outdated code? If not, could you please give me a code example using this? -- Wolfgang From glyph at twistedmatrix.com Wed Sep 10 01:18:09 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 10 Sep 2014 00:18:09 -0700 Subject: [Twisted-Python] jelly and instances In-Reply-To: <4250196.vmN52vaFvk@i5> References: <4250196.vmN52vaFvk@i5> Message-ID: On Sep 9, 2014, at 7:50 PM, Wolfgang Rohdewald wrote: > doc says: > > Instance: s is an instance of UserString.UserString, with a __dict__ > {'data': 'hello'}:: > [\"UserString.UserString\", ['dictionary', ['data', 'hello']]] > > There is no test for instances. To be precise, no test ever enters > jelly._unjelly_instance() > > I also cannot find any place in jelly.py where an Instance would > be jellied. There is > > instance_atom = 'instance' > > but instance_atom is not used anywhere. > > Is this outdated code? If not, could you please give me a code > example using this? PB was developed during a period of far less rigorous development practices for Twisted. Particularly, we didn't do test-driven development. It's quite possible that this code was never used. Most likely I looked at the Python type system when originally developing PB and thought that I'd want to jelly types.InstanceType, then thought the better of it because types should be types and InstanceType was a wart (which python-dev apparently agreed with since it is gone from the new-style object model). I imagine the thing to do with this is simply deprecate->remove, if it's already unused. It's certainly not something we'd want to resurrect. Thanks, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Sep 10 01:24:54 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 10 Sep 2014 00:24:54 -0700 Subject: [Twisted-Python] a brief note about buildbots In-Reply-To: References: Message-ID: <96E5EA0F-40AE-4F7C-B7FE-3F25F25FA07A@twistedmatrix.com> On Sep 9, 2014, at 3:56 PM, anatoly techtonik wrote: > On Wed, Sep 10, 2014 at 12:27 AM, Glyph Lefkowitz > wrote: >> Hi Twistosophists, >> >> Some of you doing reviews (you ARE all doing code reviews, right!?!?) might >> have noticed that some of the boxes on the left side of of the supported >> builds page were dark grey, meaning that they were offline. >> >> I spent some time over the last couple of days doing some administrative >> housekeeping and rebooting computers, and all of our supported builders are >> now back online! This makes me feel much better about the possibility of >> doing a release again someday :-). >> >> Speaking of colors of boxes on that page, though, we should probably try to >> turn some of them from red to green... >> >> -glyph >> >> P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're >> running many these tests on, and (B) my time to do this sort of maintenance. > > Long Live Glyph! > (Rackspace probably receives their credits in the market share) Thanks for saying so :-). > I presume you're using Firefox for all browsing needs. Am I right? =) In Chrome > I've noticed an empty box with missing image at the highscore page leading to > https://sm7.sitemeter.com/meter.asp?site=sm7twistedmatrix Nope, I have just grown blind to that box. We should really just give up on sitemeter and delete it everywhere. They were injecting some nasty stuff into our pages on some other areas of the site and it has been removed. > And when I open linked buildbot page at: > https://buildbot.twistedmatrix.com/boxes-supported?branch=trunk&num_builds=4 > Chrome says that "Attackers might be trying to steal your information from > buildbot.twistedmatrix.com (for example, passwords, messages, or credit cards)." > Not your, of course, but mine, so I am double worried about that > invalid security > certificate issued to www.twistedmatrix.com and deployed on buildbot.t.c buildbot.t.c and www.t.c are the same host and, upon closer inspection, you will see they're using the same certificate. The problem is that it's not valid for buildbot.t.c because of dumb rules. I rolled a cert for buildbot some time ago, but it was never installed. In the meanwhile I have written (also thanks to Rackspace!), which makes setting up SNI sites _much_ easier. If someone wanted to submit a pull request to and and the other attendant files in t-web, I'm sure I could deploy it. Or I will (eventually) get around to it. > BTW, how do you count tests? Is this config available somewhere? Not sure. Hopefully someone more familiar with the buildbot configuration will opine :-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Sep 10 01:26:13 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 10 Sep 2014 00:26:13 -0700 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <5928596.KCb0BbjCGC@i5> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> <5928596.KCb0BbjCGC@i5> Message-ID: <937D5E16-5F36-4BFE-A6D9-7EC5AF8CBD97@twistedmatrix.com> On Sep 9, 2014, at 7:40 PM, Wolfgang Rohdewald wrote: > Am Montag, 8. September 2014, 23:01:50 schrieb Glyph: >> This implies a parallel structure might be possible for FilePath: >> if you pass its constructor bytes, you get a BytesFilePath; >> if you pass it text, you get a TextFilePath. You can't mix the two, >> and once you've chosen a path you can't choose a different one. > > this sounds good. After the port of pb to PY3 is done, I might have > a look at it, it probably should be done before trying to port > modules.py and trial to PY3. Thank you very much for putting all this work in. I encourage you to also get involved with doing code reviews, so that the >50 tickets currently on won't distract other reviewers from ever getting to your contributions! :) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From reubenhwk at gmail.com Thu Sep 11 01:37:07 2014 From: reubenhwk at gmail.com (Reuben Hawkins) Date: Thu, 11 Sep 2014 00:37:07 -0700 Subject: [Twisted-Python] epoll 2.4 kernel Message-ID: Not sure if this is the right mailing list for this.... epoll should *not* be used on 2.4 kernels... I have 2.4 kernel running on a 32-bit x86 machine and another 2.4 kernel running on 64-bit x64 and I'm having an issue with twisted (via buildbot). I can import epoll from select on the 64-bit machine (it *should* fail but doesn't) and this later causes twisted to fail... >>> from select import epoll >>> epoll(1) Traceback (most recent call last): File "", line 1, in IOError: [Errno 38] Function not implemented On the 32-bit machine, however, importing epoll is properly failing... >>> from select import epoll Traceback (most recent call last): File "", line 1, in ImportError: cannot import name epoll >>> twisted/internet/default.py is hitting this same problem. It's incorrectly picking epoll when it shouldn?t because import epoll is working when it shouldn't. I 'fixed' the problem on my machine by adding a check method in epollreactor.py which does this... def check(): try: epoll(1) except IOError: return False return True and I import and call the check method in default.py. An alternate fix can be... in twisted/internet/default.py try: if platform.isLinux(): try: from select import epoll epoll(1) from twisted.internet.epollreactor import install except (IOError, ImportError): from twisted.internet.pollreactor import install Any insight would be good. I had to compile my own Python 2.7.8 on these old computers...maybe python is incorrectly detecting epoll in the build process? Thanks in advance, Reuben From glyph at twistedmatrix.com Thu Sep 11 03:42:26 2014 From: glyph at twistedmatrix.com (Glyph) Date: Thu, 11 Sep 2014 02:42:26 -0700 Subject: [Twisted-Python] epoll 2.4 kernel In-Reply-To: References: Message-ID: <98068ED4-68BB-44E2-A6FE-34466DF6ED03@twistedmatrix.com> On Sep 11, 2014, at 12:37 AM, Reuben Hawkins wrote: > Not sure if this is the right mailing list for this.... > > epoll should *not* be used on 2.4 kernels... I'm sorry to say that Twisted does not currently support 2.4 kernels. Our set of supported platforms is defined by the set of buildbots that we maintain: As far as I know none of those platforms use a 2.4 kernel. If you wanted to contribute or fund some resources for a 2.4 buildbot, we could consider it, but... it seems like this would be a waste of time. Even my ancient WiFi routers which I can never upgrade because of proprietary firmware drivers still have a 2.6 kernel at least. As I understand it, 2.4 was end-of-lifed 8 years ago, and even at that point it was pretty well into its decline. Why are you using it? Is "just use a really old version of Twisted" an acceptable answer for you? I ask because I am wondering if supporting it is something it would be reasonable to entertain. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at itamarst.org Thu Sep 11 04:14:10 2014 From: itamar at itamarst.org (Itamar Turner-Trauring) Date: Thu, 11 Sep 2014 06:14:10 -0400 Subject: [Twisted-Python] epoll 2.4 kernel In-Reply-To: References: Message-ID: <541175F2.4060902@itamarst.org> Easiest thing for you to do is install a poll reactor as the custom reactor: https://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html#core-howto-choosing-reactor-poll From sank.daniel at gmail.com Fri Sep 12 00:20:17 2014 From: sank.daniel at gmail.com (Daniel Sank) Date: Thu, 11 Sep 2014 23:20:17 -0700 Subject: [Twisted-Python] workflow Message-ID: I would greatly benefit from some direction regarding workflow. In particular, how do you guys deal with having an installed version of twisted, and a version (versions?) with which to tinker? Currently, I have twisted installed in the normal end user way, and I have a clone of my fork of the git repo. I mess around in the git repo clone, but I don't know how to e.g. run the tests or run applications using the git clone. glyph explained this to me on irc a while back. Are the instructions from that recorded somehwere? Why am I asking? Because I want to do reviews and I'm not doing it because I don't know how to set up a sane environment. -- Daniel Sank -------------- next part -------------- An HTML attachment was scrubbed... URL: From hs at ox.cx Fri Sep 12 03:29:04 2014 From: hs at ox.cx (Hynek Schlawack) Date: Fri, 12 Sep 2014 11:29:04 +0200 Subject: [Twisted-Python] workflow In-Reply-To: References: Message-ID: On 12 Sep 2014, at 8:20, Daniel Sank wrote: > I would greatly benefit from some direction regarding workflow. In > particular, how do you guys deal with having an installed version of > twisted, and a version (versions?) with which to tinker? > > Currently, I have twisted installed in the normal end user way, and I > have > a clone of my fork of the git repo. I mess around in the git repo > clone, > but I don't know how to e.g. run the tests or run applications using > the > git clone. > > glyph explained this to me on irc a while back. Are the instructions > from > that recorded somehwere? > > Why am I asking? Because I want to do reviews and I'm not doing it > because > I don't know how to set up a sane environment. Like any other project: I have a virtualenv where I install Twisted?s git checkout with ?pip -e .? plus all the dependencies I need. I also have a tox.ini for common permutations of deps and Python 3, but unfortunately the tests tend to fail with weird heisenerrors at least on OS X so I can?t rely on that. From netwotkstudent at yahoo.com Fri Sep 12 15:26:37 2014 From: netwotkstudent at yahoo.com (User User) Date: Fri, 12 Sep 2014 14:26:37 -0700 Subject: [Twisted-Python] How to send data from another reactor instance Message-ID: <1410557197.78081.YahooMailNeo@web160406.mail.bf1.yahoo.com> Hi, I have the codes below and I would like to know how is it possible to send out data which came to "a" instance from "b" i mean data to port 2000 go out from port "2001" !? class fact(DatagramProtocol): def datagramReceived(self, datagram, addr): self.transport.write( datagram, some_where ) a = reactor.listenUDP(2001, fact()) b = reactor.listenUDP(2002, fact()) -------------- next part -------------- An HTML attachment was scrubbed... URL: From werner at thieprojects.ch Fri Sep 12 18:21:53 2014 From: werner at thieprojects.ch (Werner Thie) Date: Fri, 12 Sep 2014 14:21:53 -1000 Subject: [Twisted-Python] How to send data from another reactor instance In-Reply-To: <1410557197.78081.YahooMailNeo@web160406.mail.bf1.yahoo.com> References: <1410557197.78081.YahooMailNeo@web160406.mail.bf1.yahoo.com> Message-ID: <54138E21.70709@thieprojects.ch> Hi consider this for TCP, adapt to UDP http://stackoverflow.com/questions/15640640/python-twisted-man-in-the-middle-implementation/15645169#15645169 Werner On 9/12/14 11:26 AM, User User wrote: > Hi, > I have the codes below and I would like to know how is it possible to > send out data which came to "a" instance from "b" i mean data to port > 2000 go out from port "2001" !? Is there anybody listening on port 2000? > > class fact(DatagramProtocol): > def datagramReceived(self, datagram, addr): > self.transport.write( datagram, some_where ) > > a = reactor.listenUDP(2001, fact()) > b = reactor.listenUDP(2002, fact()) > > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > From trac at twistedmatrix.com Sun Sep 14 00:05:15 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 14 Sep 2014 06:05:15 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1410674715.71.5935457979269114748@twistedmatrix.com An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Sun Sep 14 18:09:17 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 15 Sep 2014 00:09:17 -0000 Subject: [Twisted-Python] Python 3.3 buildslaves Message-ID: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Hello, I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot configuration/availability issues. Who can look at why one of the Python 3.3 slaves is offline and why the two slaves are configured differently? Thanks, Jean-Paul From adi at roiban.ro Mon Sep 15 00:07:12 2014 From: adi at roiban.ro (Adi Roiban) Date: Mon, 15 Sep 2014 07:07:12 +0100 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: <20140915000917.31272.1552860934.divmod.xquotient.92@top> References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 September 2014 01:09, wrote: > Hello, > > I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker > for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot > configuration/availability issues. > > Who can look at why one of the Python 3.3 slaves is offline and why > the two slaves are configured differently? > > Thanks, > Jean-Paul I would llike to help, but I don't know if I have the permission to do it. In the past, I took a look at how buildbots are configured, and I discovered that there are a lot of customizations and that there is a separate repo/process for updating the buildbots and that special privileges are required to touch the buildslaves. -------- >From my point of view, #7355 is a simple ticket and the fact that it took 4 months and it is still not merged should raise some serious concerns regarding the sustainability of the current Twisted development process. -- Adi Roiban From glyph at twistedmatrix.com Mon Sep 15 00:16:27 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 14 Sep 2014 23:16:27 -0700 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: <20140915000917.31272.1552860934.divmod.xquotient.92@top> References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On Sep 14, 2014, at 5:09 PM, exarkun at twistedmatrix.com wrote: > Hello, > > I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker > for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot > configuration/availability issues. Thank you for (specifically) bringing this up and (generally) getting the 14.1 blockers cleared out. I'm beginning to notice a concerted effort :-). > Who can look at why one of the Python 3.3 slaves is offline and why > the two slaves are configured differently? I've been gradually accumulating the various credentials that I need for maintaining all the buildbots. On my to-do list is centralizing and documenting how all of these are maintained to make it a bit less obscure. It may surprise you that you personally actually have access to those credentials! For the time being I've sent you a separate personal message explaining how you might get access to it, although I apologize that that's as good as we can do right now. This isn't exactly blocked by the following ticket, but filing this is the last thing I did while trying to figure out the Python 3 buildbot situation: I will probably have another look at this tomorrow whether it's fixed or not, but I certainly wouldn't mind if some other folks bumped that ticket and politely reminded the Fedora infrastructure team that invalid certs are a pretty serious operational issue. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Sep 15 00:38:53 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 14 Sep 2014 23:38:53 -0700 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On Sep 14, 2014, at 11:07 PM, Adi Roiban wrote: > On 15 September 2014 01:09, wrote: >> Hello, >> >> I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker >> for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot >> configuration/availability issues. >> >> Who can look at why one of the Python 3.3 slaves is offline and why >> the two slaves are configured differently? >> >> Thanks, >> Jean-Paul > > I would llike to help, but I don't know if I have the permission to do it. > > In the past, I took a look at how buildbots are configured, and I > discovered that there are a lot of customizations and that there is a > separate repo/process for updating the buildbots and that special > privileges are required to touch the buildslaves. Special privileges are required because one of the aspects of buildslave maintenance is cloud control panel access to the cloud providers that maintain this hardware, which, if abused, could end up billing the non-profit which manages Twisted's funds an unlimited amount of money. The buildbots are also managed via shell accounts, which could of course be used to deploy malware. If you are interested in helping to perform maintenance on the buildslaves, access to those permissions could be arranged for you. It had not occurred to me to ask you before, because if I recall correctly, you'd refused commit privileges when they were offered in the past. -glyph From hawkowl at atleastfornow.net Mon Sep 15 03:51:24 2014 From: hawkowl at atleastfornow.net (HawkOwl) Date: Mon, 15 Sep 2014 17:51:24 +0800 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 Sep 2014, at 14:16, Glyph wrote: > On Sep 14, 2014, at 5:09 PM, exarkun at twistedmatrix.com wrote: > >> Hello, >> >> I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker >> for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot >> configuration/availability issues. > > Thank you for (specifically) bringing this up and (generally) getting the 14.1 blockers cleared out. I'm beginning to notice a concerted effort :-). Thanks, exarkun, Glyph, as well as the others which have helped fix tickets and help things along for the 14.1 release ? it?s not going to be long before I can get 14.1pre1 gracing all your hard drives! <3 -hawkowl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hs at ox.cx Mon Sep 15 05:31:08 2014 From: hs at ox.cx (Hynek Schlawack) Date: Mon, 15 Sep 2014 13:31:08 +0200 Subject: [Twisted-Python] 14.1 & regressions (was: Python 3.3 buildslaves) In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 Sep 2014, at 8:16, Glyph wrote: >> I noticed that https://twistedmatrix.com/trac/ticket/7355 is a >> blocker >> for the release of Twisted 14.1 and #7355 is blocked Python 3.3 >> buildbot >> configuration/availability issues. > > Thank you for (specifically) bringing this up and (generally) getting > the 14.1 blockers cleared out. I'm beginning to notice a concerted > effort :-). Speaking of which: I would greatly appreciate if someone with intimate knowledge of the test plumbing code could help me to figure out https://twistedmatrix.com/trac/ticket/7370 and https://twistedmatrix.com/trac/ticket/7429 *** In any case, as soon as the buildbot?s Python installations are updated to more recent versions of cryptography/pyOpenSSL our buildbots will turn red immediately. Thanks, Hynek From exarkun at twistedmatrix.com Tue Sep 16 07:26:35 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 16 Sep 2014 13:26:35 -0000 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders Message-ID: <20140916132635.31272.1641679653.divmod.xquotient.97@top> Hello, These two builders appear to be having problems. See, for example: https://buildbot.twistedmatrix.com/builders/natty64-py2.7maint/builds/4193 https://buildbot.twistedmatrix.com/builders/twistedchecker/builds/2229 Jean-Paul From glyph at twistedmatrix.com Tue Sep 16 18:14:47 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 16 Sep 2014 17:14:47 -0700 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: <20140916132635.31272.1641679653.divmod.xquotient.97@top> References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> Message-ID: <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> On Sep 16, 2014, at 6:26 AM, exarkun at twistedmatrix.com wrote: > Hello, > > These two builders appear to be having problems. See, for example: > > https://buildbot.twistedmatrix.com/builders/natty64-py2.7maint/builds/4193 > https://buildbot.twistedmatrix.com/builders/twistedchecker/builds/2229 For what it's worth, the EOL on Natty was almost 2 years ago now: I'm inclined to do-release-upgrade on that buildbot and simply remove the natty builder, and work on fixing the twistedchecker builder once everything is on supported versions of stuff. If nobody objects by tomorrow I'll probably just proceed with that, since I don't think it was ever explicitly our intent to support OSes that have been EOL'd by their vendors (except perhaps the unkillable, eternal XP...). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From hawkowl at atleastfornow.net Wed Sep 17 05:57:48 2014 From: hawkowl at atleastfornow.net (HawkOwl) Date: Wed, 17 Sep 2014 19:57:48 +0800 Subject: [Twisted-Python] Twisted 14.0.1 Release Announcement Message-ID: <2FECA0C4-21A1-4F16-A667-E397095C3B0E@atleastfornow.net> On behalf of Twisted Matrix Laboratories, I?m releasing Twisted 14.0.1, a security release for Twisted 14.0. It is strongly suggested that users of 14.0.0 upgrade to this release. This patches a bug in Twisted Web?s Agent, where BrowserLikePolicyForHTTPS would not honour the trust root given, and would use the system trust root instead. This would have broken, for example, attempting to pin the issuer for your HTTPS application because you only trust one issuer. Note: on OS X, with the system OpenSSL, you still can't fully rely on this API for issuer pinning, due to modifications by Apple ? please see for more details. You can find the downloads at (or alternatively ). The NEWS file is also available at . Thanks for Alex Gaynor for discovering the bug, Glyph & Alex for developing a patch, and David Reid for reviewing it. Twisted Regards, HawkOwl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mithrandi at mithrandi.net Wed Sep 17 09:41:36 2014 From: mithrandi at mithrandi.net (Tristan Seligmann) Date: Wed, 17 Sep 2014 17:41:36 +0200 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> Message-ID: On 17 September 2014 02:14, Glyph Lefkowitz wrote: > For what it's worth, the EOL on Natty was almost 2 years ago now: > > Speaking of EOL, squeeze is effectively EOL now too (unless you count the squeeze-lts effort, which is very limited in scope); as such, I would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) which are still running squeeze. I think newer version of Debian are already covered by other buildslaves; so are these still needed at all? -- mithrandi, i Ainil en-Balandor, a faer Ambar From twisted-python at 2xlp.com Wed Sep 17 19:31:16 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Wed, 17 Sep 2014 21:31:16 -0400 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? Message-ID: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> Background: I'm using a 3rd party API client, which uses httplib2 in the internals. ( the google-plus api ) I'm placing this blocking code within a DeferredList, that runs within a twisted.enterprise.adbapi.ConnectionPool.runInteraction In my initial tests, everything worked fine -- but I was only making a single GET/POST, as I kept the size of the DeferredList to 1 while I worked on the code. As soon as I increased the limit and had 2 GET/POST events firing off at the same time, I started getting a ssl/socket error ( httplib2 > httplib > socket > ssl ). It works fine in twisted if I make simultaneous requests otherwise (using twisted or the `requests` package). this seems to be tied to how httplib2/httplib is running within twisted. Has anyone experienced this before? if so, do you know if there is a workaround? The easiest fix, is probably for me to write a new ApiClient tomorrow for the methods I need. I'd just like to avoid that if possible From andrew at bemusement.org Wed Sep 17 19:44:11 2014 From: andrew at bemusement.org (Andrew Bennetts) Date: Thu, 18 Sep 2014 11:44:11 +1000 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? In-Reply-To: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> References: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> Message-ID: <20140918014411.GA29843@flay.puzzling.org> Jonathan Vanasco wrote: > > Background: > I'm using a 3rd party API client, which uses httplib2 in the internals. > ( the google-plus api ) > I'm placing this blocking code within a DeferredList, that runs within a > twisted.enterprise.adbapi.ConnectionPool.runInteraction > > In my initial tests, everything worked fine -- but I was only making a single > GET/POST, as I kept the size of the DeferredList to 1 while I worked on the > code. > > As soon as I increased the limit and had 2 GET/POST events firing off at the > same time, I started getting a ssl/socket error ( httplib2 > httplib > > socket > ssl ). What's the error? > It works fine in twisted if I make simultaneous requests otherwise (using > twisted or the `requests` package). this seems to be tied to how > httplib2/httplib is running within twisted. There shouldn't be anything special about httplib2/httplib ?within twisted?. Unless httplib2 is mucking with global socket defaults or something (and I doubt it does) I wouldn't expect any interaction between the two at all. It's no different to using httplib2 in a thread without Twisted. -Andrew. > > Has anyone experienced this before? if so, do you know if there is a > workaround? > > The easiest fix, is probably for me to write a new ApiClient tomorrow for the > methods I need. I'd just like to avoid that if possible > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > From glyph at twistedmatrix.com Wed Sep 17 23:16:49 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 17 Sep 2014 22:16:49 -0700 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> Message-ID: <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> On Sep 17, 2014, at 8:41 AM, Tristan Seligmann wrote: > On 17 September 2014 02:14, Glyph Lefkowitz wrote: >> For what it's worth, the EOL on Natty was almost 2 years ago now: >> >> > > Speaking of EOL, squeeze is effectively EOL now too (unless you count > the squeeze-lts effort, which is very limited in scope); as such, I > would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) > which are still running squeeze. I think newer version of Debian are > already covered by other buildslaves; so are these still needed at > all? Not as far as I know. I think you can feel free to deprovision them (although I think you need to submit a PR for the buildmaster's config to remove them from the buildmaster and supported list) -g From dustin at v.igoro.us Thu Sep 18 06:35:45 2014 From: dustin at v.igoro.us (Dustin J. Mitchell) Date: Thu, 18 Sep 2014 08:35:45 -0400 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? In-Reply-To: <20140918014411.GA29843@flay.puzzling.org> References: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> <20140918014411.GA29843@flay.puzzling.org> Message-ID: Are you by any chance using the same HTTPConnection instance in both threads? Dustin On Wed, Sep 17, 2014 at 9:44 PM, Andrew Bennetts wrote: > Jonathan Vanasco wrote: >> >> Background: >> I'm using a 3rd party API client, which uses httplib2 in the internals. >> ( the google-plus api ) >> I'm placing this blocking code within a DeferredList, that runs within a >> twisted.enterprise.adbapi.ConnectionPool.runInteraction >> >> In my initial tests, everything worked fine -- but I was only making a single >> GET/POST, as I kept the size of the DeferredList to 1 while I worked on the >> code. >> >> As soon as I increased the limit and had 2 GET/POST events firing off at the >> same time, I started getting a ssl/socket error ( httplib2 > httplib > >> socket > ssl ). > > What's the error? > >> It works fine in twisted if I make simultaneous requests otherwise (using >> twisted or the `requests` package). this seems to be tied to how >> httplib2/httplib is running within twisted. > > There shouldn't be anything special about httplib2/httplib ?within twisted?. > Unless httplib2 is mucking with global socket defaults or something (and I doubt > it does) I wouldn't expect any interaction between the two at all. It's no > different to using httplib2 in a thread without Twisted. > > -Andrew. > >> >> Has anyone experienced this before? if so, do you know if there is a >> workaround? >> >> The easiest fix, is probably for me to write a new ApiClient tomorrow for the >> methods I need. I'd just like to avoid that if possible >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python at twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From twisted-python at 2xlp.com Thu Sep 18 10:58:55 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Thu, 18 Sep 2014 12:58:55 -0400 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? In-Reply-To: References: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> <20140918014411.GA29843@flay.puzzling.org> Message-ID: <5BFD8277-22EF-4764-93E7-DD75268E9A4F@2xlp.com> On Sep 18, 2014, at 8:35 AM, Dustin J. Mitchell wrote: > Are you by any chance using the same HTTPConnection instance in both threads? > > Dustin Ack! It looks like the snippet of sample code that I used as a constructor did that. I did not see that. This api library and docs are... frustrating. i've been trying to keep everything tied to the sample code to avoid problems. Everything is working perfect now. Thanks. ( the error was below, sorry for not posting that last night ). including here for search engines in case someone else uses this. ---- File "build/bdist.macosx-10.6-intel/egg/apiclient/http.py", line 716, in execute File "/Users/jvanasco/webserver/environments/stddev-2.7.5/lib/python2.7/site-packages/httplib2/__init__.py", line 1570, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/Users/jvanasco/webserver/environments/stddev-2.7.5/lib/python2.7/site-packages/httplib2/__init__.py", line 1317, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/Users/jvanasco/webserver/environments/stddev-2.7.5/lib/python2.7/site-packages/httplib2/__init__.py", line 1300, in _conn_request content = response.read() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 548, in read s = self.fp.read() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1311, in read return s + self._file.read() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 351, in read data = self._sock.recv(rbufsize) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 241, in recv return self.read(buflen) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 160, in read return self._sslobj.read(len) ssl.SSLError: [Errno 1] _ssl.c:1363: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number From mithrandi at mithrandi.net Sat Sep 20 12:01:44 2014 From: mithrandi at mithrandi.net (Tristan Seligmann) Date: Sat, 20 Sep 2014 20:01:44 +0200 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> Message-ID: On 18 September 2014 07:16, Glyph wrote: > On Sep 17, 2014, at 8:41 AM, Tristan Seligmann wrote: >> Speaking of EOL, squeeze is effectively EOL now too (unless you count >> the squeeze-lts effort, which is very limited in scope); as such, I >> would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) >> which are still running squeeze. I think newer version of Debian are >> already covered by other buildslaves; so are these still needed at >> all? > > Not as far as I know. I think you can feel free to deprovision them (although I think you need to submit a PR for the buildmaster's config to remove them from the buildmaster and supported list) Is the buildmaster's config in a publically accessible repo? If so, where? (If not, how do I get access?) -- mithrandi, i Ainil en-Balandor, a faer Ambar From weykent at weasyl.com Sat Sep 20 12:31:22 2014 From: weykent at weasyl.com (weykent) Date: Sat, 20 Sep 2014 11:31:22 -0700 Subject: [Twisted-Python] On testing things wrapping other things Message-ID: Hi, Recently I worked on refactoring a project so that it was, in my opinion, more testable. For reference, the diff is available at , and the important parts in the changed version are at . My question: is this a good factoring for testing a wrapper? It seems to make sense to split apart crafting and issuing requests to clients, as it lets you test the contents of a request to see what will happen instead of having to issue the request and test that its effects happened. On a previous iteration of this project, I wrote tests that did exactly that: issued requests and checked the bytes that were sent to a StringTransport. Those tests ended up being harder to read and write, and seemed to rely on the exact implementation of the thing I was wrapping. Thanks in advance for any feedback on the matter, ~weykent From trac at twistedmatrix.com Sun Sep 21 00:05:17 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 21 Sep 2014 06:05:17 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1411279517.16.9147535600233956262@twistedmatrix.com An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Sep 21 11:29:42 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 21 Sep 2014 10:29:42 -0700 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> Message-ID: <8E15331B-F496-4EC8-B6E3-DDD15837B57F@twistedmatrix.com> On Sep 20, 2014, at 11:01 AM, Tristan Seligmann wrote: > On 18 September 2014 07:16, Glyph wrote: >> On Sep 17, 2014, at 8:41 AM, Tristan Seligmann wrote: >>> Speaking of EOL, squeeze is effectively EOL now too (unless you count >>> the squeeze-lts effort, which is very limited in scope); as such, I >>> would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) >>> which are still running squeeze. I think newer version of Debian are >>> already covered by other buildslaves; so are these still needed at >>> all? >> >> Not as far as I know. I think you can feel free to deprovision them (although I think you need to submit a PR for the buildmaster's config to remove them from the buildmaster and supported list) > > Is the buildmaster's config in a publically accessible repo? If so, > where? (If not, how do I get access?) It's here: -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Sep 21 11:36:37 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 21 Sep 2014 10:36:37 -0700 Subject: [Twisted-Python] On testing things wrapping other things In-Reply-To: References: Message-ID: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> On Sep 20, 2014, at 11:31 AM, weykent wrote: > Hi, > > Recently I worked on refactoring a project so that it was, in my opinion, more testable. For reference, the diff is available at , and the important parts in the changed version are at . > > My question: is this a good factoring for testing a wrapper? It seems to make sense to split apart crafting and issuing requests to clients, as it lets you test the contents of a request to see what will happen instead of having to issue the request and test that its effects happened. If anything I'd say that the problem with this code is that it's not a public, documented interface. The clients of this library might want to test it as well :). > On a previous iteration of this project, I wrote tests that did exactly that: issued requests and checked the bytes that were sent to a StringTransport. Those tests ended up being harder to read and write, and seemed to rely on the exact implementation of the thing I was wrapping. If you're going to check the bytes in a StringTransport, that means you need an implementation of the other end of the protocol as well. If you're just asserting about the exact bytes, then that's really brittle and will break in response to lots of incidental changes. You can use this as a part of a higher-level testing strategy, like so: but in this case the unit test code doesn't see the response to its requests until it's been parsed into a sensible data structure, and assertions are about that. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From weykent at weasyl.com Sun Sep 21 11:52:12 2014 From: weykent at weasyl.com (weykent) Date: Sun, 21 Sep 2014 10:52:12 -0700 Subject: [Twisted-Python] On testing things wrapping other things In-Reply-To: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> References: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> Message-ID: On Sep 21, 2014, at 10:36 AM, Glyph wrote: > If anything I'd say that the problem with this code is that it's not a public, documented interface. The clients of this library might want to test it as well :). Ah! You are referring to both _wrap and _issueRequest here, right? That makes sense, and it?s easy to make those documented public interfaces. > If you're going to check the bytes in a StringTransport, that means you need an implementation of the other end of the protocol as well. If you're just asserting about the exact bytes, then that's really brittle and will break in response to lots of incidental changes. > > You can use this as a part of a higher-level testing strategy, like so: > > > > but in this case the unit test code doesn't see the response to its requests until it's been parsed into a sensible data structure, and assertions are about that. Okay, this makes sense. Of course, Twisted doesn?t come with an implementation of a memcache server, and I?d rather not implement one myself for the purpose of testing. I think I?ll stick with my latest approach, then. Would you employ the same approach for testing non-wrappers as well? That is, if you were testing the client implementation of some wire protocol, would you still write parsers for the server implementation of the protocol to make assertions about the sent bytes? As far as I can tell, this is not what, say, the existing memcache protocol tests do. From glyph at twistedmatrix.com Sun Sep 21 12:07:53 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 21 Sep 2014 11:07:53 -0700 Subject: [Twisted-Python] On testing things wrapping other things In-Reply-To: References: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> Message-ID: <04A598A2-0EB5-44A1-AEF6-A937F3244DE8@twistedmatrix.com> On Sep 21, 2014, at 10:52 AM, weykent wrote: > On Sep 21, 2014, at 10:36 AM, Glyph wrote: > >> If anything I'd say that the problem with this code is that it's not a public, documented interface. The clients of this library might want to test it as well :). > > Ah! You are referring to both _wrap and _issueRequest here, right? That makes sense, and it?s easy to make those documented public interfaces. > >> If you're going to check the bytes in a StringTransport, that means you need an implementation of the other end of the protocol as well. If you're just asserting about the exact bytes, then that's really brittle and will break in response to lots of incidental changes. >> >> You can use this as a part of a higher-level testing strategy, like so: >> >> >> >> but in this case the unit test code doesn't see the response to its requests until it's been parsed into a sensible data structure, and assertions are about that. > > Okay, this makes sense. Of course, Twisted doesn?t come with an implementation of a memcache server, and I?d rather not implement one myself for the purpose of testing. I think I?ll stick with my latest approach, then. > > Would you employ the same approach for testing non-wrappers as well? That is, if you were testing the client implementation of some wire protocol, would you still write parsers for the server implementation of the protocol to make assertions about the sent bytes? As far as I can tell, this is not what, say, the existing memcache protocol tests do. If I were testing the wire protocol itself I'd probably assert about the exact bytes directly. At that level, changes to the exact bytes being emitted are not an incidental change, that's exactly what the system under test is supposed to be doing; producing bytes. So changing those tests would definitely be worthwhile. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandrieu at greenflex.com Mon Sep 22 08:01:06 2014 From: sandrieu at greenflex.com (Simon Andrieu) Date: Mon, 22 Sep 2014 14:01:06 +0000 Subject: [Twisted-Python] GPRS/Modem protocol through USB support Message-ID: Hi, I work for a company that develops telemetry boxes, and that uses the Twisted framework running on the main board. It is planned to integrate a cellular module connected via USB or serial link to the main board. Could you help me to know if GPRS/3G/Modem protocols through USB are supported with Twisted please? And how it is possible to use this interface? Thanks for your help. Best Regards, Simon Andrieu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bram at diomedia.be Mon Sep 22 08:57:55 2014 From: bram at diomedia.be (Bram Van Steenlandt) Date: Mon, 22 Sep 2014 16:57:55 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: References: Message-ID: <542038F3.5000400@diomedia.be> Hi, I'm using a gsm modem for sending sms messages, the modem is rs232 so I have to use and ftdi adapter. You can use these with pyserial in a thread in twisted. It uses AT commands, I have one from sierra wireless, some phones can do the same. 3G/internet is something else, here you just need something the operating system will support, twisted doesn't care how you are connected. Bram op 22-09-14 16:01, Simon Andrieu schreef: > > Hi, > > I work for a company that develops telemetry boxes, and that uses the > Twisted framework running on the main board. > > It is planned to integrate a cellular module connected via USB or > serial link to the main board. > > Could you help me to know if GPRS/3G/Modem protocols through USB are > supported with Twisted please? And how it is possible to use this > interface? > > Thanks for your help. > > Best Regards, > > Simon Andrieu > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Mon Sep 22 09:05:29 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 22 Sep 2014 17:05:29 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <542038F3.5000400@diomedia.be> References: <542038F3.5000400@diomedia.be> Message-ID: <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> Hi Bram & Simon, I?ll answer both e-mails in one go :-) On 22 Sep 2014, at 16:57, Bram Van Steenlandt wrote: > I'm using a gsm modem for sending sms messages, the modem is rs232 so I have to use and ftdi adapter. > You can use these with pyserial in a thread in twisted. Huh! Interesting; what drove that choice, given twisted?s serial support? > op 22-09-14 16:01, Simon Andrieu schreef: >> >> Could you help me to know if GPRS/3G/Modem protocols through USB are supported with Twisted please? And how it is possible to use this interface? Depends what that thing exposes. Usually it?s a character device, so you end up whistling some bytes. However, if you are *just* interested in a very standard Internet connection setup, it is probably much easier to use the existing OS tools for connecting to the internet using that model, than to write something using Twisted. Twisted would be interesting if you want to do interesting things beyond that; e.g. talk to the device and ask it for all sorts of e.g. telemetry data, data that is not already exposed in some standard, easy to consume way. hth lvh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bram at diomedia.be Mon Sep 22 09:31:33 2014 From: bram at diomedia.be (Bram Van Steenlandt) Date: Mon, 22 Sep 2014 17:31:33 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> Message-ID: <542040D5.9010404@diomedia.be> No specific reason actually, I didn't even know twisted had support for a serial port. This being said: ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. How long does twisted has this serial port support actually ? Bram op 22-09-14 17:05, Laurens Van Houtven schreef: > Hi Bram & Simon, > > I'll answer both e-mails in one go :-) > > On 22 Sep 2014, at 16:57, Bram Van Steenlandt wrote: > >> I'm using a gsm modem for sending sms messages, the modem is rs232 so I have to use and ftdi adapter. >> You can use these with pyserial in a thread in twisted. > Huh! Interesting; what drove that choice, given twisted's serial support? > >> op 22-09-14 16:01, Simon Andrieu schreef: >>> Could you help me to know if GPRS/3G/Modem protocols through USB are supported with Twisted please? And how it is possible to use this interface? > Depends what that thing exposes. Usually it's a character device, so you end up whistling some bytes. However, if you are *just* interested in a very standard Internet connection setup, it is probably much easier to use the existing OS tools for connecting to the internet using that model, than to write something using Twisted. Twisted would be interesting if you want to do interesting things beyond that; e.g. talk to the device and ask it for all sorts of e.g. telemetry data, data that is not already exposed in some standard, easy to consume way. > > hth > lvh > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From twisted-python at 2xlp.com Mon Sep 22 10:00:35 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Mon, 22 Sep 2014 12:00:35 -0400 Subject: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool) Message-ID: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> I've got some database transactional code running within a thread via RunInteraction (it needs to rollback on error). One of the features on the logic is that it interacts with a 3rd party API, which may or may not have a ratelimit in effect. Most 3rd party APIs i consume have a minute/hourly based limit -- so I just do a graceful fail on the transaction to requeue it, and set a local timeout. This particular API has a per-second meter in effect. It is considerably more efficient to just sleep for a second and "try again" a second later. I end up being able to process over twice as many API calls in a given period with that strategy. The problem I ran into, is that I need a second database connection to note that a ratelimit has been created in the database -- and i'd like to pull that db connection out of the twisted connection pool to simplify my connection management. - I can't do it within the existing connection, because I am in a transaction and the limit won't appear to other twisted jobs. - I can't use runInteraction() off the pool, because that will defer to a thread and I am in a thread, and the last time I did something like that... everything broke. - the stopgap I have right now is just creating/closing a raw pscycopg2 connection and using that. it works, but it requires a bit more management of postgres to ensure I can balance the right number of connections. has anyone had a need like this before? if so, what did you do? i basically just need to make two "INSERT" statements from a second connection. From _ at lvh.io Mon Sep 22 10:03:26 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 22 Sep 2014 18:03:26 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <542040D5.9010404@diomedia.be> References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> <542040D5.9010404@diomedia.be> Message-ID: On 22 Sep 2014, at 17:31, Bram Van Steenlandt wrote: > No specific reason actually, I didn't even know twisted had support for a serial port. > > This being said: > ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. > I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). > Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. Use cases vary :-) In a wait-dosomething-wait scenario, I find @inlineCallbacks to typically yield virtually the same code you ended up with. When I was doing this, there was some pretty complex chatter back and forth, and I was multiplexing several connections over a single RS232 device, so Twisted certainly came in handy. In your case: don?t fix what ain?t broken, of course :) > Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. > > How long does twisted has this serial port support actually ? Well over a decade: Mon Dec 16 04:04:25 2002 to be precise. See svn rev trunk at 4528. hth lvh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bram at diomedia.be Mon Sep 22 11:34:04 2014 From: bram at diomedia.be (Bram Van Steenlandt) Date: Mon, 22 Sep 2014 19:34:04 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> <542040D5.9010404@diomedia.be> Message-ID: <54205D8C.2070606@diomedia.be> op 22-09-14 18:03, Laurens Van Houtven schreef: > On 22 Sep 2014, at 17:31, Bram Van Steenlandt wrote: > >> No specific reason actually, I didn't even know twisted had support for a serial port. >> >> This being said: >> ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. >> I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). >> Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. > Use cases vary :-) In a wait-dosomething-wait scenario, I find @inlineCallbacks to typically yield virtually the same code you ended up with. > > When I was doing this, there was some pretty complex chatter back and forth, and I was multiplexing several connections over a single RS232 device, so Twisted certainly came in handy. In your case: don't fix what ain't broken, of course :) Wouldn't it be difficult to combine the twisted method with external libaries like MinimalModbus, libftdi , libartnet etc ? Or is there some magic that I'm not aware of ? I there a disadvantage to using a thread ? >> Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. >> >> How long does twisted has this serial port support actually ? > Well over a decade: Mon Dec 16 04:04:25 2002 to be precise. See svn rev trunk at 4528. > > hth > lvh > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From werner at thieprojects.ch Mon Sep 22 15:25:43 2014 From: werner at thieprojects.ch (Werner Thie) Date: Mon, 22 Sep 2014 11:25:43 -1000 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <54205D8C.2070606@diomedia.be> References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> <542040D5.9010404@diomedia.be> <54205D8C.2070606@diomedia.be> Message-ID: <542093D7.5020607@thieprojects.ch> Aloha maybe you can get some clues from here https://gist.github.com/claws/2464017 which is pretty close to what your looking for. HTH, Werner On 9/22/14 7:34 AM, Bram Van Steenlandt wrote: > > op 22-09-14 18:03, Laurens Van Houtven schreef: >> On 22 Sep 2014, at 17:31, Bram Van Steenlandt wrote: >> >>> No specific reason actually, I didn't even know twisted had support for a serial port. >>> >>> This being said: >>> ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. >>> I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). >>> Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. >> Use cases vary :-) In a wait-dosomething-wait scenario, I find @inlineCallbacks to typically yield virtually the same code you ended up with. >> >> When I was doing this, there was some pretty complex chatter back and forth, and I was multiplexing several connections over a single RS232 device, so Twisted certainly came in handy. In your case: don?t fix what ain?t broken, of course :) > Wouldn't it be difficult to combine the twisted method with external > libaries like MinimalModbus, libftdi , libartnet etc ? > Or is there some magic that I'm not aware of ? > I there a disadvantage to using a thread ? > > >>> Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. >>> >>> How long does twisted has this serial port support actually ? >> Well over a decade: Mon Dec 16 04:04:25 2002 to be precise. See svn rev trunk at 4528. >> >> hth >> lvh >> >> >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python at twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > From glyph at twistedmatrix.com Mon Sep 22 23:45:16 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 22 Sep 2014 22:45:16 -0700 Subject: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool) In-Reply-To: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> References: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> Message-ID: On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco wrote: > has anyone had a need like this before? if so, what did you do? Yes. I re-wrote twisted.enterprise entirely, with a better design. You can see the result here: And you can 'pip install' it from here: Hopefully it will see a non-alpha release at some point. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From twisted-python at 2xlp.com Tue Sep 23 11:23:49 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Tue, 23 Sep 2014 13:23:49 -0400 Subject: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool) In-Reply-To: References: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> Message-ID: oh great! I'll jump through the docs this weekend! thanks so much for the reply and for writing that! On Sep 23, 2014, at 1:45 AM, Glyph wrote: > > On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco wrote: > >> has anyone had a need like this before? if so, what did you do? > > Yes. I re-wrote twisted.enterprise entirely, with a better design. > > You can see the result here: > > > > And you can 'pip install' it from here: > > > > Hopefully it will see a non-alpha release at some point. > > -glyph > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python // Jonathan Vanasco c. 646.729.6436 | 415.501.9815 e. jonathan at 2xlp.com w. http://findmeon.com/user/jvanasco linkedin. http://linkedin.com/in/jonathanvanasco blog. http://destructuring.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Wed Sep 24 20:26:17 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 25 Sep 2014 02:26:17 +0000 (UTC) Subject: [Twisted-Python] Security Advisory: bash remote code execution Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Today a security release of Bash was issued, fixing a critical vulnerability. This vulnerability allows an attacker to inject and execute arbitrary code on many web servers and other applications. This issue does not affect Twisted directly, but will affect many users of Twisted. Any web server which is serving traffic over a CGI or CGI-like interface (including WSGI) should upgrade its version of Bash immediately. This issue has been assigned CVE-2014-6271. A complete description of the bug is also available: http://seclists.org/oss- sec/2014/q3/650 New packages have been issued for the following operating systems: * Debian: http://www.debian.org/security/2014/dsa-3032 * Ubuntu: http://www.ubuntu.com/usn/usn-2362-1/ * Red Hat Enterprise Linux: https://rhn.redhat.com/errata/RHSA-2014-1293.html Please be aware that there are reports that the current patches do not completely solve the issue, it is likely that users will need to perform updates a second time. Sincerely, Alex Gaynor -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJUI30jAAoJEBJfXGff6UCEQfkQAIi36kVAmGimqiKeqgpbTX5F Z9TRMIhwVtZBtohQXj34QB+y382ArzLXve6toaj0si7lbQN0KnqOqsCyhJ5magxp ZtX/6ZDYcrgRM07sP+JVS+OO5YZnrH/EjfcxKA6kyDk2Rbs7rq21EHuw13HaS1Xo cPiusCM8XIhWZ9Fv8H1fSF7VkXb791kQJGMgcivvww3Z9+WltNbIuDOc57iuXIQ8 AnOxGkGcJL0qQHYkBYqXQejFNBMy0JZNE15ORyep7rYjDvCVacCfOyfjyd9kqpTt QHomR1XdMhZBffpygKbHnJ7wIyFcG0eQMwJxsX8gelmevvVwMmGC+qbmN4R+h1F6 HoSe0y5JHRNFgk3ClKBFgJtD1PB4Jrxl2YqCr+EyzwaQERzSes0bCGy4D1/VVn/Z JWoedrFl7R/pf+DAD/krTA0fVbfDprhlvc4c72p94m0h2xzgcI3RCHT+ZopyIMW8 7Fw0hawx8366m1E0RZBJsi/zvq8Eq/WhVbeTl2C9Vqgg3yE+DUvg6a65y78yXUN4 wgOzI/34ZHQbTll4wRBabllR7yU/WFCLJE95TTS5QIXhikzkd+T7MjiqeSSfEIfZ 55XFvKy1Z2AM/USozvLfwM5siCn+r2o3BmQ97Wm5i0YTbrJqUW2KFSM0xZp3YBQc ryYFOFCmIfZpjPEIheqD =oBm7 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Wed Sep 24 23:44:06 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 24 Sep 2014 22:44:06 -0700 Subject: [Twisted-Python] Security Advisory: bash remote code execution In-Reply-To: References: Message-ID: On Sep 24, 2014, at 7:26 PM, Alex Gaynor wrote: > Please be aware that there are reports that the current patches do not > completely solve the issue, it is likely that users will need to perform > updates a second time. These reports are being tracked as CVE-2014-7169. More information here: Good night and good luck, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From haggardii at gmail.com Thu Sep 25 09:09:20 2014 From: haggardii at gmail.com (Matt Haggard) Date: Thu, 25 Sep 2014 09:09:20 -0600 Subject: [Twisted-Python] Security Advisory: bash remote code execution In-Reply-To: References: Message-ID: > > Any web server which is serving traffic over a CGI or CGI-like interface > (including WSGI) should upgrade its version of Bash immediately. > I feel ignorant, but I'm confused about how WSGI is affected (and have failed to exploit my WSGI app). AFAICT from reading the code, Twisted's WSGIResource doesn't invoke a shell. I see that it has an `environ` attribute that gets filled with user-provided information, but I don't see how that makes it into a shell's environment. We'll patch bash anyway. Thanks, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Sep 25 13:54:29 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 25 Sep 2014 12:54:29 -0700 Subject: [Twisted-Python] Security Advisory: bash remote code execution In-Reply-To: References: Message-ID: <3FF9F33B-AFC4-474C-9CC5-AAC62C8D1735@twistedmatrix.com> On Sep 25, 2014, at 8:09 AM, Matt Haggard wrote: > > > > Any web server which is serving traffic over a CGI or CGI-like interface > > (including WSGI) should upgrade its version of Bash immediately. > > > > I feel ignorant, but I'm confused about how WSGI is affected (and have failed to exploit my WSGI app). AFAICT from reading the code, Twisted's WSGIResource doesn't invoke a shell. I see that it has an `environ` attribute that gets filled with user-provided information, but I don't see how that makes it into a shell's environment. As Alex's post said, this vulnerability does not affect Twisted directly. The point is that most people deploying web services are doing so in a UNIX environment, and in so doing they are probably invoking scripts of various kinds, or executables which may have been replaced with wrapper shell-scripts. It's hard to audit for environment variables containing attacker-controlled data, and this is the sort of thing we've all been trained to expect is safe, if they're variables in our own "namespace", so it's possible that any number of 3rd-party tools you are using with Twisted are vulnerable in surprising ways. So everybody should just upgrade :). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Sep 25 16:31:27 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 25 Sep 2014 15:31:27 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool Message-ID: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Hi all, (For those of you wondering about the subject, note that this message is sent in compliance with This email sent in compliance with .) I've been trying to improve the reliability of the buildbots. (On that note, by the way, several builders are failing and more will be soon as security updates roll out, it would be great if someone could review before the whole fleet turns red.) In so doing I (re-)discovered this bug: . At first I just wanted to delete an intermittently failing test which appears to be nonsense. However, after some prodding from exarkun, I investigated and discovered that this very poorly-written test case does in fact illustrate a real problem with our current threadpool implementation which can result in deadlocks, hangs on exit, and other unpleasantness. In my use of Twisted I have encountered several "huh, this can't happen, maybe cosmic rays or something" bugs which might have been caused by this, so I would very much like to fix it. One reason that our threadpool implementation is problematic is that it has a somewhat complex internal design, lots of weird little features (context preservation, completion callbacks, workload estimation) all built at the same layer with no composition. I tried to figure out a way to salvage its current architecture and I could not reasonably do so. I have prototyped a completely new threadpool implementation ("twisted.threads") in a spike. It is definitely not ready for review, but I would definitely appreciate design-level feedback on the code right now. You can see the implementation here: (or here: if that's more your speed) It's less code than the existing implementation and provides more useful features at the same time. It might, in fact, provide some of the threading primitives that we would need for a reactor-per-thread implementation. However, there is a very ugly implementation detail that prevents us from marching onwards to a glorious multithreading future: twisted.internet.interfaces.IReactorThreads.getThreadPool. The nominally "public" interface provided by its documented return type, the concrete (old-style!) class twisted.python.threadpool.ThreadPool, is ridiculously over-broad. Here are the features (attributes and methods) in its "public" interface: callInThread callInThreadWithCallback adjustPoolsize start stop q min max joined started name waiters threads working workers start startAWorker stopAWorker dumpStats __getstate__ __setstate__ threadFactory currentThread Here's what I would like its public interface to be, more or less what it's intended to be, based on the ways it's documented and used in Twisted and in the various projects I can see using it: callInThread callInThreadWithCallback adjustPoolsize start stop It would not be too much effort to also preserve, for legacy purposes: min max joined started name dumpStats However, I would REALLY like to delete these implementation details: workers threads working startAWorker q __getstate__ __setstate__ threadFactory currentThread Deleting these implementation details straight up would also make it such that anyone who inherited from the public class ThreadPool would not be able to override any of these things (oh my goodness, why would you do that, I really hope nobody has done that). The three ways I could proceed, in order of my own preference, are: Put the new implementation with entirely new interfaces into twisted.threads, put a compatibility shim into twisted.python.threadpool.ThreadPool that wraps those objects, provides my more minimal interface proposed above, and deletes 90% of its tests. Change IReactorThreads.getThreadPool to return a documented interface considerably more restricted than Compatibility implications: twisted.internet.interfaces.IReactorThreads's contract would be relaxed. Implementors of IReactorThreads would not see an impact unless they were talking to deleted parts of ThreadPool internally. ThreadPool's clients would have several methods removed, and subclassing would effectively not be possible any more (a bunch of overridable hooks would have been removed, and no replacements would be provided). However, I prefer this option because there are an extremely restrictive set of use-cases (basically: only logging, any behavioral changes would have been broken) where clients could have made legitimate use of these attributes or overriding any of these functions. Delete all of twisted.python.threadpool's test coverage, deprecate the entire module, and delete it in the next release (14.1+1). This has the advantage of making the test suite reliable, and gets the benefits for any users of callInThread, but does not relay any of the benefits to users of ThreadPool. The only ones doing this directly in Twisted are deferToThreadPool (and by extension, deferToThread) and WSGIResource. I could update these in tandem, however; the code changes would be very small. I don't think it would be incompatible to make WSGIResource itself accept a different type of constructor argument (although if we don't, I wonder if subclassing anything with a constructor can be considered applicable to the compatibility policy in Twisted; hmm). Actually go through the trouble of emulating all the attributes on ThreadPool and emulating them as best I can. Delete the direct tests for ThreadPool itself and write a private subclass that will be returned from getThreadPool that returns a compatibility shim which isinstance(ThreadPool) for compatibility and still has all the gross attributes. This change would be technically compatible, but would be a huge amount of additional work and I am doubtful that it would provide any value. So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From weykent at weasyl.com Thu Sep 25 19:36:16 2014 From: weykent at weasyl.com (weykent) Date: Thu, 25 Sep 2014 18:36:16 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? Yes. Specifically, I am maintaining this AMP responder method: @FetchThreadPoolStats.responder def fetchThreadPoolStats(self): pool = self.factory.threadPool return dict( threadsWaiting=len(pool.waiters), threadsWorking=len(pool.working), workerQueueSize=pool.q.qsize(), ) These statistics are chunked into graphite and displayed as a nice little graph. They?ve been quite useful on some occasions: sometimes all of the threads in a thread pool for a WSGI application got blocked, and the symptoms were that all of the requests coming in were timing out at the proxy in front of twisted. We were able to quickly tell that the problem was the WSGI thread pool because the threadsWorking count was at its limit and the workerQueueSize was skyrocketing. > If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) I don?t really see this as an abuse of the public interface. If possible, I?d like to see this diagnostic information preserved in the new interface, as I consider it invaluable information as long as we?re using twisted as a WSGI runner. I will say that it is certainly possible for me to emulate these attributes, but that would require touching implementation details of WSGIResource. I?m not sure which is worse. ~weykent From glyph at twistedmatrix.com Fri Sep 26 01:10:01 2014 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 26 Sep 2014 00:10:01 -0700 Subject: [Twisted-Python] naming question before we end up committed... Message-ID: While working on the package I just brought up in another thread, I realize we have another big new package coming in 14.1, "twisted.python.logger", and I'm wondering what the ".python" part of that package name is doing in there, and what value it adds. It's where "twisted.python.log" used to be, of course, but now I'm just wondering if we're just making the same mistake twice. Does anyone have an opinion on renaming it to "twisted.logger"? Or have a better suggestion? -glyph From glyph at twistedmatrix.com Fri Sep 26 01:43:22 2014 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 26 Sep 2014 00:43:22 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> Message-ID: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> On Sep 25, 2014, at 6:36 PM, weykent wrote: > On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz wrote: > >> So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? > > Yes. Specifically, I am maintaining this AMP responder method: Wow, cool. So, okay, while I am a little unhappy that you're using this API in a slightly unfortunate way, that feeling is eclipsed by the joy that The Process Works :-). Thank you very much for responding so promptly, so specifically, and including the exact code that we need to discuss. > @FetchThreadPoolStats.responder > def fetchThreadPoolStats(self): > pool = self.factory.threadPool > return dict( > threadsWaiting=len(pool.waiters), > threadsWorking=len(pool.working), > workerQueueSize=pool.q.qsize(), > ) > > These statistics are chunked into graphite and displayed as a nice little graph. They?ve been quite useful on some occasions: sometimes all of the threads in a thread pool for a WSGI application got blocked, and the symptoms were that all of the requests coming in were timing out at the proxy in front of twisted. We were able to quickly tell that the problem was the WSGI thread pool because the threadsWorking count was at its limit and the workerQueueSize was skyrocketing. Analytics, monitoring, logging, and statistics have historically been a blind spot for Twisted and I am really glad you're bringing this up. I was thinking about bringing it up in my original message, but as my colleagues and friends will tell you, I tend to put too many words into emails, so unless I'm sure I need those words, I will delete them. The new interface should very definitely have metrics-gathering functionality, and possibly even a logging component. >> If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) > > I don?t really see this as an abuse of the public interface. If possible, I?d like to see this diagnostic information preserved in the new interface, as I consider it invaluable information as long as we?re using twisted as a WSGI runner. It's not exactly an "abuse", because the public interface is there, and you have a use-case, and you're satisfying it with what's available. For the purposes that you are using these attributes, I actually don't have a problem reproducing them at all. It should be pretty straightforward. My concern with exposing them overall is that what you've got with "ThreadPool.q" is not a queue that has a qsize() method that you can inspect, it's that it's a Queue that you can put things into, get things out of, and generally muck about with. Similarly pool.waiters is not simply an object with a __len__ that tells you how active the threads are, it's a list of Thread objects which you could potentially change the names of, turn into daemon threads, join(), and so on. More importantly it's a list that you could remove Thread objects from and that would affect their interactions with the pool. How about this: would you mind if ThreadPool were modified to still populate these attributes for monitoring and debugging purposes, but completely stopped honoring any destructive operations on them? In the case of "pool.q", there is no single Queue responsible for the whole pool, so I would like to make "put", "get", and "join" actually raise exceptions; in the case of "waiters" and "working" I guess I could fish out some actual Thread objects, that just involves a little bit of sad abstraction violation internal to the twisted.threads.Team implementation. My inclination would be to expand the thread pool to accommodate these usages, but still deprecate these attributes in the next version; but add a better "statistics" method that returned an object with "active", "pending", and "idle" attributes (all integers). The most important thing that I want to make sure nobody wants is to keep overriding (or calling) startAWorker, stopAWorker, __getstate__, __setstate__. Supporting callers of threadFactory and currentThread would be easy, but the attributes aren't really there for calling, they're there to be stubbed, and stubbing them won't keep working without tons of extra work. So would that work for you? > I will say that it is certainly possible for me to emulate these attributes, but that would require touching implementation details of WSGIResource. I?m not sure which is worse. Touching private (underscore-prefixed) implementation details is explicitly not supported - if you did it that way, you're on your own :-). And, as the end-user in this discussion, you win the argument by default. (You can see how I feel about supporting users here: ) Plus, if we're going to work on the thread-pool and have newer, better interfaces, WSGIResource's implementation is likely to change. Thanks a bunch for helping us improve Twisted, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Fri Sep 26 01:49:03 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 26 Sep 2014 08:49:03 +0100 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: On 26 September 2014 08:10, Glyph wrote: > While working on the package I just brought up in another thread, I realize we have another big new package coming in 14.1, "twisted.python.logger", and I'm wondering what the ".python" part of that package name is doing in there, and what value it adds. It's where "twisted.python.log" used to be, of course, but now I'm just wondering if we're just making the same mistake twice. > > Does anyone have an opinion on renaming it to "twisted.logger"? Or have a better suggestion? > > -glyph I think that twisted.logger is better. Description of twisted.python is: Twisted Python: Utilities and Enhancements for Python ... and for me the new logger is more than an utility or enhancement to python language or stdlib. -- Adi Roiban From _ at lvh.io Fri Sep 26 01:53:06 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Fri, 26 Sep 2014 09:53:06 +0200 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: <2C57F9FB-041F-462F-AE88-F07FAFC34BA1@lvh.io> +0 on ?logger? (top-level names have a history of whimsy) +1 on making it top-level hth lvh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From weykent at weasyl.com Fri Sep 26 02:12:53 2014 From: weykent at weasyl.com (weykent) Date: Fri, 26 Sep 2014 01:12:53 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> Message-ID: <02CD8913-6E7C-4D53-B3DE-91A8AED4B460@weasyl.com> On Sep 26, 2014, at 12:43 AM, Glyph wrote: > How about this: would you mind if ThreadPool were modified to still populate these attributes for monitoring and debugging purposes, but completely stopped honoring any destructive operations on them? In the case of "pool.q", there is no single Queue responsible for the whole pool, so I would like to make "put", "get", and "join" actually raise exceptions; in the case of "waiters" and "working" I guess I could fish out some actual Thread objects, that just involves a little bit of sad abstraction violation internal to the twisted.threads.Team implementation. Having objects with only nondestructive methods would be fine for me. I don?t even care about getting Thread objects out, and I certainly hope nobody else cares about getting Thread objects out either. > My inclination would be to expand the thread pool to accommodate these usages, but still deprecate these attributes in the next version; but add a better "statistics" method that returned an object with "active", "pending", and "idle" attributes (all integers). This sounds fantastic. I?ll graph whatever the statistics method emits, so feel free to include other useful or interesting metrics as well. (Though, I?m not sure what else there is that?s easy to keep track of and useful.) > So would that work for you? Yes, absolutely. > Touching private (underscore-prefixed) implementation details is explicitly not supported - if you did it that way, you're on your own :-). And, as the end-user in this discussion, you win the argument by default. (You can see how I feel about supporting users here: ) Plus, if we're going to work on the thread-pool and have newer, better interfaces, WSGIResource's implementation is likely to change. I haven?t touched private implementation details of WSGIResource, nor was I planning to. I?ve already had some code (in another project) break across versions of twisted because of that, and I don?t want it to happen again. > Thanks a bunch for helping us improve Twisted, Thank you for taking the time to write up your original post and this reply! ~weykent From adi at roiban.ro Fri Sep 26 02:20:30 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 26 Sep 2014 09:20:30 +0100 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> Message-ID: On 26 September 2014 08:43, Glyph wrote: [snip] > My inclination would be to expand the thread pool to accommodate these > usages, but still deprecate these attributes in the next version; but add a > better "statistics" method that returned an object with "active", "pending", > and "idle" attributes (all integers). +1 for statistics method I am using some of those member for my custom test case https://github.com/chevah/empirical/blob/master/chevah/empirical/testcase.py#L66 A method which return those integers would be enough for my use case. ---- +1 for reducing the public interface. With a public adjustPoolsize I prefer to also have read-only attributes for min / max pool size. With a public start/stop a public read-only started attribute might be useful.... for the case when you don't want to do a try/catch or just want to see the state of the pool. dumpStats could be replaced by a better method dedicated to statistics, which will return integer instead of using the logger. `joined` looks like the opposite of started/running... ie, it is redundant. Regarding `name` attribute, I guess that it might make sense in the case an application uses multiple thread pools... I have never used multiple pools so I can not comment on that. Thanks! -- Adi Roiban From adi at roiban.ro Fri Sep 26 02:26:46 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 26 Sep 2014 09:26:46 +0100 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 September 2014 07:38, Glyph wrote: [snip] > Special privileges are required because one of the aspects of buildslave maintenance is cloud control panel access to the cloud providers that maintain this hardware, which, if abused, could end up billing the non-profit which manages Twisted's funds an unlimited amount of money. The buildbots are also managed via shell accounts, which could of course be used to deploy malware. > > If you are interested in helping to perform maintenance on the buildslaves, access to those permissions could be arranged for you. It had not occurred to me to ask you before, because if I recall correctly, you'd refused commit privileges when they were offered in the past. > > -glyph Sorry for the delay. I would like to help the Twisted project in any way... including with buildslave maintenance. I think that at this level I only need ssh access. -------- I refused commit privileges as I wanted to contribute more patches and do more reviews as a read-only contributors, before starting to mess with Twisted repo. I think that it would be nice to have some process to initiate new contributors to the whole review process and once one or more senior-reviewers consider that the new contributors is good to do reviews by itself, to give him commit privileges. Cheers, -- Adi Roiban From exarkun at twistedmatrix.com Fri Sep 26 07:16:01 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 26 Sep 2014 13:16:01 -0000 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: <20140926131601.17800.326361613.divmod.xquotient.0@top> On 08:26 am, adi at roiban.ro wrote: > >I think that it would be nice to have some process to initiate new >contributors to the whole review process and once one or more >senior-reviewers consider that the new contributors is good to do >reviews by itself, to give him commit privileges. +1 Jean-Paul From radix at twistedmatrix.com Fri Sep 26 09:53:20 2014 From: radix at twistedmatrix.com (Christopher Armstrong) Date: Fri, 26 Sep 2014 10:53:20 -0500 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> Message-ID: On September 25, 2014 at 8:39:54 PM, weykent (weykent at weasyl.com) wrote: On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? Yes. Specifically, I am maintaining this AMP responder method: @FetchThreadPoolStats.responder def fetchThreadPoolStats(self): pool = self.factory.threadPool return dict( threadsWaiting=len(pool.waiters), threadsWorking=len(pool.working), workerQueueSize=pool.q.qsize(), ) These statistics are chunked into graphite and displayed as a nice little graph. They?ve been quite useful on some occasions: sometimes all of the threads in a thread pool for a WSGI application got blocked, and the symptoms were that all of the requests coming in were timing out at the proxy in front of twisted. We were able to quickly tell that the problem was the WSGI thread pool because the threadsWorking count was at its limit and the workerQueueSize was skyrocketing. We did very similar stuff (logging thread pool size to graphite) at id, so they're probably still using those bits too. > If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-)? I don?t really see this as an abuse of the public interface. If possible, I?d like to see this diagnostic information preserved in the new interface, as I consider it invaluable information as long as we?re using twisted as a WSGI runner.? I will say that it is certainly possible for me to emulate these attributes, but that would require touching implementation details of WSGIResource. I?m not sure which is worse.? ~weykent? --? Christopher Armstrong http://twitter.com/radix http://wordeology.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jml at mumak.net Sat Sep 27 00:47:52 2014 From: jml at mumak.net (Jonathan Lange) Date: Sat, 27 Sep 2014 07:47:52 +0100 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> Message-ID: On Fri, Sep 26, 2014 at 8:43 AM, Glyph wrote: > > On Sep 25, 2014, at 6:36 PM, weykent wrote: > > On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz > wrote: > > So, does anyone out there have any code which makes use of the > aforementioned bad attributes of ThreadPool, whose applications would break > if I removed them? > > > Yes. Specifically, I am maintaining this AMP responder method: > > > Wow, cool. > > So, okay, while I am a *little* unhappy that you're using this API in a > slightly unfortunate way, that feeling is eclipsed by the joy that The > Process Works :-). > Yes! I saw this thread and immediately grabbed a colleague, shoved my phone under his face and said *this* is what I love about Twisted. Carry on. jml -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac at twistedmatrix.com Sun Sep 28 00:05:18 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 28 Sep 2014 06:05:18 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1411884318.24.6180239602225432630@twistedmatrix.com An HTML attachment was scrubbed... URL: From hs at ox.cx Sun Sep 28 00:19:43 2014 From: hs at ox.cx (Hynek Schlawack) Date: Sun, 28 Sep 2014 08:19:43 +0200 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: > Does anyone have an opinion on renaming it to "twisted.logger"? Or have a better suggestion? +1 on dropping `python` here. That's a weird namespace anyway something as central as logging shouldn't be there. From wolfgang.kde at rohdewald.de Sun Sep 28 00:52:11 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Sun, 28 Sep 2014 08:52:11 +0200 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) Message-ID: <2329405.l76rWJfZKg@i5> I am getting this traceback on about every source file like: i5:~/ssdsrc/alt/Twisted/twisted/test (pb3) twistedchecker reflect_helper_ZDE.py ************* Module reflect_helper_ZDE W9001: 1,0: Missing copyright header W9002: 1,0: Missing a reference to test module in header C0103: 1,0: Invalid name "reflect_helper_ZDE" for type module (should match (([a-z_][a-z0-9_]*))$) W9208: 1,0: Missing docstring Traceback (most recent call last): File "/usr/local/bin/twistedchecker", line 5, in pkg_resources.run_script('TwistedChecker==0.2.0', 'twistedchecker') File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script execfile(script_filename, namespace, namespace) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/EGG-INFO/scripts/twistedchecker", line 10, in Runner().run(sys.argv[1:]) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/core/runner.py", line 325, in run self.linter.check(args) File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 542, in check self.check_astng_module(astng, walker, rawcheckers) File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 615, in check_astng_module walker.walk(astng) File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/utils.py", line 553, in walk cb(astng) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 179, in visit_module self._runPEP8Checker(node.file) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 188, in _runPEP8Checker recorder = PEP8WarningRecorder(file) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 49, in __init__ self.run() File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 78, in run pep8.Checker.check_all(self) File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1445, in check_all self.check_logical() File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1338, in check_logical for offset, text in self.run_check(check, argument_names) or (): File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1278, in run_check return check(*arguments) TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) -- Wolfgang From chriswwolfe at gmail.com Sun Sep 28 09:36:46 2014 From: chriswwolfe at gmail.com (Chris) Date: Sun, 28 Sep 2014 10:36:46 -0500 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: +1 for twisted.logger. From glyph at twistedmatrix.com Sun Sep 28 18:15:37 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Sep 2014 17:15:37 -0700 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <2329405.l76rWJfZKg@i5> References: <2329405.l76rWJfZKg@i5> Message-ID: How did you install twistedchecker? -glyph On Sep 27, 2014, at 11:52 PM, Wolfgang Rohdewald wrote: > I am getting this traceback on about every source file like: > > > i5:~/ssdsrc/alt/Twisted/twisted/test (pb3) twistedchecker reflect_helper_ZDE.py > ************* Module reflect_helper_ZDE > W9001: 1,0: Missing copyright header > W9002: 1,0: Missing a reference to test module in header > C0103: 1,0: Invalid name "reflect_helper_ZDE" for type module (should match (([a-z_][a-z0-9_]*))$) > W9208: 1,0: Missing docstring > Traceback (most recent call last): > File "/usr/local/bin/twistedchecker", line 5, in > pkg_resources.run_script('TwistedChecker==0.2.0', 'twistedchecker') > File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script > self.require(requires)[0].run_script(script_name, ns) > File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script > execfile(script_filename, namespace, namespace) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/EGG-INFO/scripts/twistedchecker", line 10, in > Runner().run(sys.argv[1:]) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/core/runner.py", line 325, in run > self.linter.check(args) > File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 542, in check > self.check_astng_module(astng, walker, rawcheckers) > File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 615, in check_astng_module > walker.walk(astng) > File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/utils.py", line 553, in walk > cb(astng) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 179, in visit_module > self._runPEP8Checker(node.file) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 188, in _runPEP8Checker > recorder = PEP8WarningRecorder(file) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 49, in __init__ > self.run() > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 78, in run > pep8.Checker.check_all(self) > File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1445, in check_all > self.check_logical() > File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1338, in check_logical > for offset, text in self.run_check(check, argument_names) or (): > File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1278, in run_check > return check(*arguments) > TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) > > > -- > Wolfgang > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From josh at bartletts.id.au Sun Sep 28 19:56:04 2014 From: josh at bartletts.id.au (Joshua Bartlett) Date: Mon, 29 Sep 2014 11:56:04 +1000 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: On 26 September 2014 08:31, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the > aforementioned bad attributes of ThreadPool, whose applications would break > if I removed them? If so, how can I make you feel as bad about yourselves > for using it as I feel bad about myself for writing it? ;-) > There are a few times in a codebase I help maintain where we want a thread pool of all daemon threads. The code we have for this overrides threadFactory() like so: class DaemonThreadPool(ThreadPool): def threadFactory(self, *args, **kwargs): t = threading.Thread(*args, **kwargs) t.setDaemon(True) return t How would I do this with your proposed new API? Josh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 29 00:16:00 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 29 Sep 2014 08:16 +0200 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: References: <2329405.l76rWJfZKg@i5> Message-ID: <4230856.dFXEK2FZA3@i5> Am Sonntag, 28. September 2014, 17:15:37 schrieb Glyph: > How did you install twistedchecker? installing is not even needed: git clone https://github.com/twisted/twistedchecker cd twistedchecker ~/src/Twisted/bin/trial twistedchecker trial being the latest from twisted svn trunk, I get the same error pylint --version No config file found, using default configuration pylint 0.26.0, astng 0.24.3, common 0.61.0 Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] -- Wolfgang From glyph at twistedmatrix.com Mon Sep 29 01:07:33 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Sep 2014 00:07:33 -0700 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <4230856.dFXEK2FZA3@i5> References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> Message-ID: <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> On Sep 28, 2014, at 11:16 PM, Wolfgang Rohdewald wrote: > Am Sonntag, 28. September 2014, 17:15:37 schrieb Glyph: >> How did you install twistedchecker? > > installing is not even needed: > > git clone https://github.com/twisted/twistedchecker > cd twistedchecker > ~/src/Twisted/bin/trial twistedchecker > > trial being the latest from twisted svn trunk, I get the same error > > pylint --version > No config file found, using default configuration > pylint 0.26.0, > astng 0.24.3, common 0.61.0 > Python 2.7.6 (default, Mar 22 2014, 22:59:56) > [GCC 4.8.2] Hmm. It seems as though twistedchecker does not itself have continuous integration set up, so I don't have any test results to look at. It seems to be working on our buildbots (for now, at least) but I'm not sure why. This week I will try to get merged, among other things. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 29 01:31:07 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 29 Sep 2014 09:31:07 +0200 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> Message-ID: <2623018.bnDGBy8Oq5@i5> Am Montag, 29. September 2014, 00:07:33 schrieb Glyph: > This week I will try to get merged, which requires exactly pep8 1.5.1 Ubuntu 14.04 installs 1.4.6 I have the current 1.5.7 After downgrading to 1.4.6, twistedchecker works correctly. -- Wolfgang From glyph at twistedmatrix.com Mon Sep 29 02:05:47 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Sep 2014 01:05:47 -0700 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: <20140915000917.31272.1552860934.divmod.xquotient.92@top> References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: <508ABC57-1F2C-4CFE-A49C-8594183E0AD6@twistedmatrix.com> On Sep 14, 2014, at 5:09 PM, exarkun at twistedmatrix.com wrote: > Hello, > > I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker > for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot > configuration/availability issues. > > Who can look at why one of the Python 3.3 slaves is offline and why > the two slaves are configured differently? The configuration of the two slaves has been made consistent, and I have reviewed and landed the relevant ticket. At long last, that particular square on the build matrix is green. -glyph From glyph at twistedmatrix.com Mon Sep 29 04:11:20 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Sep 2014 03:11:20 -0700 Subject: [Twisted-Python] glorious green Message-ID: <3B0EFF3D-F676-4CDE-BDE5-C02D34F26402@twistedmatrix.com> I will probably be going to sleep before py-select-gc gets around to finishing, but other than that, is completely green. If were reviewed, we might be able to make look a bit more like the solid green rectangle it really ought to be. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Sep 29 13:11:55 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 29 Sep 2014 12:11:55 -0700 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <2623018.bnDGBy8Oq5@i5> References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> <2623018.bnDGBy8Oq5@i5> Message-ID: On Sep 29, 2014, at 12:31 AM, Wolfgang Rohdewald wrote: > Am Montag, 29. September 2014, 00:07:33 schrieb Glyph: >> This week I will try to get merged, > > which requires exactly pep8 1.5.1 > > Ubuntu 14.04 installs 1.4.6 > I have the current 1.5.7 > > After downgrading to 1.4.6, twistedchecker works correctly. The issue is actually logilab-common, not pep8 itself; twistedchecker can deal with more recent pep8 versions. I fixed this on twistedchecker master: More exciting, though, is Adi Roiban's contribution of Travis-CI continuous integration: so we can pick up the pace on twistedchecker improvements a bit. (The buildbot already runs straight out of Git, so improvements go live almost immediately.) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Tue Sep 30 00:36:20 2014 From: adi at roiban.ro (Adi Roiban) Date: Tue, 30 Sep 2014 07:36:20 +0100 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> <2623018.bnDGBy8Oq5@i5> Message-ID: On 29 September 2014 20:11, Glyph Lefkowitz wrote: > On Sep 29, 2014, at 12:31 AM, Wolfgang Rohdewald > wrote: > > Am Montag, 29. September 2014, 00:07:33 schrieb Glyph: > > This week I will try to get > merged, > > > which requires exactly pep8 1.5.1 > > Ubuntu 14.04 installs 1.4.6 > I have the current 1.5.7 > > After downgrading to 1.4.6, twistedchecker works correctly. > > > The issue is actually logilab-common, not pep8 itself; twistedchecker can > deal with more recent pep8 versions. > I got this similar error when I was using pep8 1.5.1 ... after upgrade to 1.5.6 all was fine. Please check latest setup.py from master... and the install line from travis.yml Cheers! -- Adi Roiban From glyph at twistedmatrix.com Tue Sep 30 22:09:12 2014 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 30 Sep 2014 21:09:12 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: On Sep 28, 2014, at 6:56 PM, Joshua Bartlett wrote: > On 26 September 2014 08:31, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) > > > There are a few times in a codebase I help maintain where we want a thread pool of all daemon threads. The code we have for this overrides threadFactory() like so: > > > class DaemonThreadPool(ThreadPool): > def threadFactory(self, *args, **kwargs): > t = threading.Thread(*args, **kwargs) > t.setDaemon(True) > return t > > > How would I do this with your proposed new API? I suppose honoring the threadFactory attribute is possible. In the new thread pool prototype I've created so far, "Team", provides pluggability for creating different kinds of "Worker": and the thread "Worker" provides pluggability for creating different kinds of thread: so I'll make sure that this keeps working. That said: why did you need a threadpool of daemon threads? -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh at bartletts.id.au Tue Sep 30 23:12:36 2014 From: josh at bartletts.id.au (Joshua Bartlett) Date: Wed, 1 Oct 2014 15:12:36 +1000 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: On 1 October 2014 14:09, Glyph wrote: > > That said: why did you need a threadpool of daemon threads? > That's a very good question. One of my comaintainers thinks the answer is, "My codebase is old enough to vote and to answer your question I'd probably have to turn our SVN server back on. Please don't make me." :-) I think the real reason is that the third-party library that we're calling from the threads sometimes takes a really long time to return, which means that unless we use daemon threads the only timely way to shut down / restart the service is kill -9, which is not ideal. Josh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 1 05:39:25 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 01 Sep 2014 13:39:25 +0200 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved Message-ID: <1574968.iq743TZKTO@i5> Hi, while trying to port jelly to Python 3 (banana tests already pass with Python 3), I found a bug in jelly. It has already been reported 3 years ago with ticket 4935. And there is a fix attached to that ticket. This fix resolves the problem for me. Why has this fix never been included? It is necessary for porting to Python 3. -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 1 05:57:22 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 01 Sep 2014 11:57:22 -0000 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved In-Reply-To: <1574968.iq743TZKTO@i5> References: <1574968.iq743TZKTO@i5> Message-ID: <20140901115722.20413.1823536452.divmod.xquotient.1771@top> On 11:39 am, wolfgang.kde at rohdewald.de wrote: >Hi, > >while trying to port jelly to Python 3 (banana tests already pass with >Python 3), > >I found a bug in jelly. It has already been reported 3 years ago with >ticket 4935. > >And there is a fix attached to that ticket. This fix resolves the >problem for me. > >Why has this fix never been included? It is necessary for porting to >Python 3. Perhaps because the ticket was never given the "review" keyword - so no one ever noticed that there was a potential fix attached to it. See https://twistedmatrix.com/trac/wiki/ReviewProcess Jean-Paul >-- >Wolfgang > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python at twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From grigorescu_ciprian at yahoo.com Mon Sep 1 06:24:43 2014 From: grigorescu_ciprian at yahoo.com (Grigorescu Ciprian) Date: Mon, 1 Sep 2014 05:24:43 -0700 Subject: [Twisted-Python] Using spawnProcess Message-ID: <1409574283.53606.YahooMailNeo@web141705.mail.bf1.yahoo.com> Hello, I have wrote a custom protocol(inherit from SMTPClient) and factory(inherit from ClientFactory) in order to send emails from multiple ips(multiple connections, factories). Now I want to run this pice of code multiple times using spawnProcess to send emails simultaneous, each process having "n" connections from "n" ips. How can i use spawnProcess? Or is another way to send emails in a parallel fashion with twisted? Maybe another approach? Thank you, -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 1 06:31:16 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 01 Sep 2014 14:31:16 +0200 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved In-Reply-To: <20140901115722.20413.1823536452.divmod.xquotient.1771@top> References: <1574968.iq743TZKTO@i5> <20140901115722.20413.1823536452.divmod.xquotient.1771@top> Message-ID: <4619137.bWkyiyDUYJ@i5> Am Montag, 1. September 2014, 11:57:22 schrieb exarkun at twistedmatrix.com: > >Why has this fix never been included? It is necessary for porting to > >Python 3. > > Perhaps because the ticket was never given the "review" keyword - so no > one ever noticed that there was a potential fix attached to it. So I should formally be able to review this since I am not the author. What I can say for certain is that the fix looks OK, and porting to Python3 depends on it. Could somebody else please review this ticket 4935? -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 1 06:52:56 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 01 Sep 2014 12:52:56 -0000 Subject: [Twisted-Python] jelly with new-style classes: method references are not resolved In-Reply-To: <4619137.bWkyiyDUYJ@i5> References: <1574968.iq743TZKTO@i5> <20140901115722.20413.1823536452.divmod.xquotient.1771@top> <4619137.bWkyiyDUYJ@i5> Message-ID: <20140901125256.20413.1175149262.divmod.xquotient.1784@top> On 12:31 pm, wolfgang.kde at rohdewald.de wrote: >Am Montag, 1. September 2014, 11:57:22 schrieb >exarkun at twistedmatrix.com: >> >Why has this fix never been included? It is necessary for porting to >> >Python 3. >> >>Perhaps because the ticket was never given the "review" keyword - so >>no >>one ever noticed that there was a potential fix attached to it. > >So I should formally be able to review this since I am not the author. Hi Wolfgang, Your input on the issue would be much appreciated, yes. However, since neither you nor the contributor has commit access, you can't actually grant the ticket a passing review. >What I can say for certain is that the fix looks OK, and porting to >Python3 depends on it. > >Could somebody else please review this ticket 4935? I took a look at the patch. There are a couple issues: 1) The tests are not written as unit tests which integrate into the existing test suite. They need to be rewritten so that `trial twisted` will run them. 2) It wasn't immediately obvious to me how the issue is fixed by the patch. It would be nice to either comment on the ticket explaining how the change fixes the problem or add such information in a comment near the code that is changing. Thanks, Jean-Paul From exarkun at twistedmatrix.com Mon Sep 1 06:55:39 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 01 Sep 2014 12:55:39 -0000 Subject: [Twisted-Python] Using spawnProcess In-Reply-To: <1409574283.53606.YahooMailNeo@web141705.mail.bf1.yahoo.com> References: <1409574283.53606.YahooMailNeo@web141705.mail.bf1.yahoo.com> Message-ID: <20140901125539.20413.1865087905.divmod.xquotient.1789@top> On 12:24 pm, grigorescu_ciprian at yahoo.com wrote: >Hello, >I have wrote a custom protocol(inherit from SMTPClient) and >factory(inherit from ClientFactory) in order to send emails from >multiple ips(multiple connections, factories). > >Now I want to run this pice of code multiple times using spawnProcess >to send emails simultaneous, each process having "n" connections from >"n" ips. > >How can i use spawnProcess? Or is another way to send emails in a >parallel fashion with twisted? Maybe another approach? You don't need multiple processes to send email concurrently. If you were to use `twisted.mail.smtp.sendmail`, then you could do this by just making multiple calls to `sendmail`. You don't have to wait for one email to be completely sent before you start trying to send the next one. For example, to send two emails concurrently: from sys import stdout from twisted.mail.smtp import sendmail from twisted.internet.defer import gatherResults from twisted.internet.task import react from twisted.python.log import startLogging def main(): return gatherResults([ sendmail(...), sendmail(...), ]) startLogging(stdout) react(main, []) You should be able to do something similar with the API you've constructed (or perhaps you can just use `sendmail` instead). Jean-Paul From wolfgang.kde at rohdewald.de Mon Sep 1 16:24:33 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Tue, 02 Sep 2014 00:24:33 +0200 Subject: [Twisted-Python] what is a non-class class? Message-ID: <1522503.xpp1Bo2WKl@i5> >From spread/jelly.py in _unjelly_method: - if type(im_class) is not types.ClassType: raise InsecureJelly("Method found with non-class class.") Same problem in _unjelly_instance What is a non-class class or rather what sort of types is meant to be insecure? Suggestions for a better error message instead of "non-class class"? This should work with both old style and new style classes, right now it works only for old style and fails for new style. Working with Python3 gives bonus points. Current source code does not have test cases for both _unjelly_instance and _unjelly_method using new style classes. My patch for ticket 4935 has a such a test case for _unjelly_method which fails as expected. If only user defined classes are accepted as secure, see also https://stackoverflow.com/questions/14612865/how-to-check-if-object-is-instance-of-new-style-user-defined-class Would it be acceptable to instead exclude an explicit list of basic types as that stackoverflow answer suggests? if im_class in (int, long, float, bool, list, set, frozenset, dict, type(None), str, unicode, tuple, what else?): raise InsecureJelly https://twistedmatrix.com/documents/current/api/twisted.spread.jelly.InsecureJelly.html does not cover this. Python 2.7.6: >>> class A(object): ... pass ... >>> type(A) >>> type(int) >>> import types >>> types.ClassType >>> class B: ... pass ... >>> type(B) Python 3.3: >>> class A: ... pass ... >>> type(A) >>> type(int) -- Wolfgang From adi at roiban.ro Tue Sep 2 03:08:49 2014 From: adi at roiban.ro (Adi Roiban) Date: Tue, 2 Sep 2014 10:08:49 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures Message-ID: Hi, While using spawnProcess on Linux I found out that when an invalid executable is called there is a corner case in which a zombie process is left until main process exists and can not be closed. I wrote a test for this but I was not able to reproduce this error in isolation, event if I run the test for 10000 times. reapProcess will always succeed from the first call. For the production code I can always reproduce the problem. Inspecting the execution thread I found out that all pipes are closed but spawned process is not closed yet. Due to this Process.maybeCallProcessEnded() will call self.reapProcess(). In my case, os.waitpid(pid, os.WNOHANG) return 0, and self.reapProcess() will just ignore this case. Process handlers is still registered in reapProcessHandlers but reapAllProcesses is no longer called. If a add a reactor.callLater(self.reapProcess) the next call to os.waitpid will succeed and the forked process is closed. ------------ What am I doing wrong? Why reapAllProcesses is no longer called, even if reapProcessHandlers is not empty? My process protocol code is here https://gist.github.com/adiroiban/bac493f00ce5e94738ce but from what I see, this should happen for any protocol. Many thanks, -- Adi Roiban From jmp at editshare.com Tue Sep 2 05:05:33 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Tue, 02 Sep 2014 07:05:33 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: Message-ID: <5405A47D.3020106@editshare.com> On 09/02/2014 05:08 AM, Adi Roiban wrote: > Hi, > > While using spawnProcess on Linux I found out that when an invalid > executable is called there is a corner case in which a zombie process > is left until main process exists and can not be closed. > > I wrote a test for this but I was not able to reproduce this error in > isolation, event if I run the test for 10000 times. reapProcess will > always succeed from the first call. > > For the production code I can always reproduce the problem. > > Inspecting the execution thread I found out that all pipes are closed > but spawned process is not closed yet. Due to this > Process.maybeCallProcessEnded() will call self.reapProcess(). > > In my case, os.waitpid(pid, os.WNOHANG) return 0, and > self.reapProcess() will just ignore this case. We encountered this problem in our code too. We worked around it with the following code, which basically monkey-patches Twisted to "try again later" when waitpid returns 0. (Most of the code below is just copied from _BaseProcess; the important part is the "elif pid == 0" branch.) ----- """Workarounds for problems with Twisted.""" import errno import os from twisted.python import log from twisted.internet.process import ( _BaseProcess, reapAllProcesses, unregisterReapProcessHandler ) def workaround_reapProcess(reactor): """Install a workaround for unsticking reapProcess. Sometimes when a child process takes too long to die that reapProcess doesn't catch it in time. We add a hack where we add a timeout to the reactor to try again later. """ def reapProcess(self): """ Try to reap a process (without blocking) via waitpid. This is called when sigchild is caught or a Process object loses its "connection" (stdout is closed) This ought to result in reaping all zombie processes, since it will be called twice as often as it needs to be. (Unfortunately, this is a slightly experimental approach, since UNIX has no way to be really sure that your process is going to go away w/o blocking. I don't want to block.) """ try: try: pid, status = os.waitpid(self.pid, os.WNOHANG) except OSError, e: if e.errno == errno.ECHILD: # no child process pid = None else: raise except: log.msg('Failed to reap %d:' % self.pid) log.err() pid = None status = None if pid: self.processEnded(status) unregisterReapProcessHandler(pid, self) elif pid == 0: # Twisted seems to get stuck if pid is 0, which means that # the child process hasn't changed status, but if called # after SIGCHLD probably means that the child process is # in the process of dying, but hasn't quite died yet. # We'll try to kick the reactor to reap the processes # again in a bit. # # We're testing specifically against 0 because pid may # also be None in an error case. def unstick(): reapAllProcesses() reactor.callLater(1, unstick) _BaseProcess.reapProcess = reapProcess ---- To use this, import your reactor and then call workaround_reapProcess(reactor). Now that two of us have seen the same problem, we should probably file a ticket in the bug tracker. --Justin From 4kir4.1i at gmail.com Tue Sep 2 06:21:24 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Tue, 02 Sep 2014 16:21:24 +0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures References: <5405A47D.3020106@editshare.com> Message-ID: <87egvuqkl7.fsf@gmail.com> Justin Mazzola Paluska writes: Unrelated. > def unstick(): > reapAllProcesses() > reactor.callLater(1, unstick) > Why do you use `unstick` instead of `reapAllProcesses` directly. Do you expect that `reapAllProcesses` name will be rebound in the future? -- Akira From jmp at editshare.com Tue Sep 2 06:44:50 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Tue, 2 Sep 2014 08:44:50 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <87egvuqkl7.fsf@gmail.com> References: <5405A47D.3020106@editshare.com> <87egvuqkl7.fsf@gmail.com> Message-ID: On Sep 2, 2014 8:22 AM, "Akira Li" <4kir4.1i at gmail.com> wrote: > > Justin Mazzola Paluska writes: > > Unrelated. > > > def unstick(): > > reapAllProcesses() > > reactor.callLater(1, unstick) > > > > Why do you use `unstick` instead of `reapAllProcesses` directly. > Do you expect that `reapAllProcesses` name will be rebound in the future? I had a bunch of debug spew in unstick when I was developing this monkey patch. I left the indirection in in case I needed to add the debug spew back. --Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Tue Sep 2 07:31:00 2014 From: adi at roiban.ro (Adi Roiban) Date: Tue, 2 Sep 2014 14:31:00 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <5405A47D.3020106@editshare.com> References: <5405A47D.3020106@editshare.com> Message-ID: Problem solved... see below On 2 September 2014 12:05, Justin Mazzola Paluska wrote: > On 09/02/2014 05:08 AM, Adi Roiban wrote: [snip] > if pid: > self.processEnded(status) > unregisterReapProcessHandler(pid, self) > elif pid == 0: > # Twisted seems to get stuck if pid is 0, which means that > # the child process hasn't changed status, but if called > # after SIGCHLD probably means that the child process is > # in the process of dying, but hasn't quite died yet. > # We'll try to kick the reactor to reap the processes > # again in a bit. > # > # We're testing specifically against 0 because pid may > # also be None in an error case. > def unstick(): > reapAllProcesses() > reactor.callLater(1, unstick) > > _BaseProcess.reapProcess = reapProcess > > ---- > > To use this, import your reactor and then call > workaround_reapProcess(reactor). > > Now that two of us have seen the same problem, we should probably file a > ticket in the bug tracker. > --Justin My quick fix was to only call reactor.callLater(self.reapProcess) and not to reap all processes ---------- I dig deeper and I found out that since I was using reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not installed. I have switched to using just reactor.run() and the process is now killed. Thanks! -- Adi Roiban From jmp at editshare.com Tue Sep 2 07:38:55 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Tue, 02 Sep 2014 09:38:55 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> Message-ID: <5405C86F.2000609@editshare.com> On 09/02/2014 09:31 AM, Adi Roiban wrote: > Problem solved... see below > > On 2 September 2014 12:05, Justin Mazzola Paluska wrote: >> On 09/02/2014 05:08 AM, Adi Roiban wrote: > [snip] > >> if pid: >> self.processEnded(status) >> unregisterReapProcessHandler(pid, self) >> elif pid == 0: >> # Twisted seems to get stuck if pid is 0, which means that >> # the child process hasn't changed status, but if called >> # after SIGCHLD probably means that the child process is >> # in the process of dying, but hasn't quite died yet. >> # We'll try to kick the reactor to reap the processes >> # again in a bit. >> # >> # We're testing specifically against 0 because pid may >> # also be None in an error case. >> def unstick(): >> reapAllProcesses() >> reactor.callLater(1, unstick) >> >> _BaseProcess.reapProcess = reapProcess >> >> ---- >> >> To use this, import your reactor and then call >> workaround_reapProcess(reactor). >> >> Now that two of us have seen the same problem, we should probably file a >> ticket in the bug tracker. >> --Justin > My quick fix was to only call reactor.callLater(self.reapProcess) and > not to reap all processes Fair enough. FWIW, looking at your original code, you may also want to call your Deferreds in a reactor.callLater. I learned the hard way that if you callback a Deferred from processEnded and the callback spawns another process, your process will still be Twisted's process table even though it's dying. > > ---------- > > I dig deeper and I found out that since I was using > reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not > installed. > > I have switched to using just reactor.run() and the process is now killed. Without my workaround, I continue to have the problem with the gtk2reactor. --Justin From exarkun at twistedmatrix.com Tue Sep 2 07:50:52 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 02 Sep 2014 13:50:52 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <5405C86F.2000609@editshare.com> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> Message-ID: <20140902135052.20413.975133368.divmod.xquotient.1791@top> On 01:38 pm, jmp at editshare.com wrote: > >FWIW, looking at your original code, you may also want to call your >Deferreds in a reactor.callLater. I learned the hard way that if you >callback a Deferred from processEnded and the callback spawns another >process, your process will still be Twisted's process table even though >it's dying. >> >>---------- >> >>I dig deeper and I found out that since I was using >>reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not >>installed. >> >>I have switched to using just reactor.run() and the process is now >>killed. > >Without my workaround, I continue to have the problem with the >gtk2reactor. Have you reported this bug? Jean-Paul From glyph at twistedmatrix.com Tue Sep 2 17:29:41 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 2 Sep 2014 16:29:41 -0700 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <1522503.xpp1Bo2WKl@i5> References: <1522503.xpp1Bo2WKl@i5> Message-ID: On Sep 1, 2014, at 3:24 PM, Wolfgang Rohdewald wrote: > What is a non-class class or rather what sort of types is meant to > be insecure? Jelly is dynamically typed, so any value might show up in any position. In this case, a value shows up in the slot in the serialization of a method object which indicates that method's class might be any object, but it has to be a class. > Suggestions for a better error message instead of "non-class class"? It's not clear that there needs to be a better error message here; you'll only get this message if you have corrupt data on the wire, since a correct implementation of PB will never put anything other than a class in that slot. (Except I think it might be broken in the face of new-style classes; ClassType is the old-style class type, 'type' is the new one, so, that should probably be fixed, as per your other thread...) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Wed Sep 3 00:45:58 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 03 Sep 2014 08:45:58 +0200 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: References: <1522503.xpp1Bo2WKl@i5> Message-ID: <1925033.An9b9qoBL0@i5> I still do not like "non-class class" because this is a contradiction in itself. Anyway a non-class class is a class whose objects may not get a method attached. So far so good, but that was obvious. Am Dienstag, 2. September 2014, 16:29:41 schrieb Glyph Lefkowitz: > Except I think it might be broken in the face of new-style classes; > ClassType is the old-style class type, 'type' is the new one, so, > that should probably be fixed That is my problem. How do I know if the object is of a non-class class? As you say, current code does not handle new style classes. I cannot check if the object is of type "type" because int is also of type "type". An int certainly should not get a method attached. Then we could just as well remove this check. That is why I proposed to instead exclude a list of basic types int, float, list, dict, set and so on. -- Wolfgang From glyph at twistedmatrix.com Wed Sep 3 01:29:59 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 3 Sep 2014 00:29:59 -0700 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <1925033.An9b9qoBL0@i5> References: <1522503.xpp1Bo2WKl@i5> <1925033.An9b9qoBL0@i5> Message-ID: <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> On Sep 2, 2014, at 11:45 PM, Wolfgang Rohdewald wrote: > I still do not like "non-class class" because this is a contradiction > in itself. How about "non-class found in class slot when deserializing method object"? > Anyway a non-class class is a class whose objects may not get a > method attached. So far so good, but that was obvious. Uh... no? A non-class in an object which is not a class. The error is when that object is provided as the class associated with a method. > Am Dienstag, 2. September 2014, 16:29:41 schrieb Glyph Lefkowitz: >> Except I think it might be broken in the face of new-style classes; >> ClassType is the old-style class type, 'type' is the new one, so, >> that should probably be fixed > > That is my problem. How do I know if the object is of a non-class class? isinstance(something, (types.ClassType, type)). > As you say, current code does not handle new style classes. Yes, but it could be easily modified to do so. > I cannot check if the object is of type "type" because > int is also of type "type". An int certainly should not get > a method attached. Then we could just as well remove this check. >>> isinstance(object, type) True >>> isinstance(object(), type) False > That is why I proposed to instead exclude a list of basic types > int, float, list, dict, set and so on. I'm not sure I understand the proposal. But in any case - it's not necessary. It's clearly possible to determine if a particular value is a type or not. -glyph From wolfgang.kde at rohdewald.de Wed Sep 3 01:55:26 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 03 Sep 2014 09:55:26 +0200 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> References: <1522503.xpp1Bo2WKl@i5> <1925033.An9b9qoBL0@i5> <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> Message-ID: <1709945.N3rAP3M3F9@i5> Am Mittwoch, 3. September 2014, 00:29:59 schrieb Glyph: > > That is my problem. How do I know if the object is of a non-class class? > > isinstance(something, (types.ClassType, type)). but that will make it possible to attach a method even to type "int" Python2: >>> isinstance(int, type) True > >>> isinstance(object, type) > True > >>> isinstance(object(), type) > False > Sure but the test only gets the class and it should certainly not instantiate it. Meanwhile I believe it is best to simply remove the test since python itself will reject most: >>> a=int >>> a.x=5 Traceback (most recent call last): File "", line 1, in TypeError: can't set attributes of built-in/extension type 'int' -- Wolfgang From vikas.c.kumar at oracle.com Wed Sep 3 05:32:04 2014 From: vikas.c.kumar at oracle.com (vikas kumar) Date: Wed, 03 Sep 2014 17:02:04 +0530 Subject: [Twisted-Python] Return only when value/handle is available Message-ID: <5406FC34.9080109@oracle.com> Hi, I am little new to twisted. I've a function(get_handle) which returns a handle. But get_handle() may throw an exception(SystemDelayException) because of some delay in system. My requirement is : When I get SystemDelayException re-attempt get_handle() and return only when handle is available. How do I use deferred/reactor in this scenario? try: handle = yield self.get_handle() except SystemDelayException: // code to re-attempt get_handle and return only when handle is available Regards Vikas From jonas.brunsgaard at gmail.com Wed Sep 3 06:40:24 2014 From: jonas.brunsgaard at gmail.com (Jonas Brunsgaard) Date: Wed, 3 Sep 2014 14:40:24 +0200 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: <5406FC34.9080109@oracle.com> References: <5406FC34.9080109@oracle.com> Message-ID: I would fiddle around with something like this. from twisted.internet.defer import inlineCallbacks, returnValue, Deferred from twisted.internet import reactor @inlineCallbacks def foo(self, retries=10, interval=10): while True: try: returnValue((yield self.get_handle())) except SystemDelayException as e: if retries > 0: retries -= 1 d = Deferred() reactor.callLater(interval, d.callback, None) yield d else: raise e On Wed, Sep 3, 2014 at 1:32 PM, vikas kumar wrote: > Hi, > > I am little new to twisted. > I've a function(get_handle) which returns a handle. > > But get_handle() may throw an exception(SystemDelayException) because of > some delay in system. > My requirement is : When I get SystemDelayException re-attempt > get_handle() and return only when handle is available. How do I use > deferred/reactor in this scenario? > > try: > handle = yield self.get_handle() > except SystemDelayException: > // code to re-attempt get_handle and return only when handle is > available > > Regards > Vikas > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Wed Sep 3 06:42:33 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 13:42:33 +0100 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: <5406FC34.9080109@oracle.com> References: <5406FC34.9080109@oracle.com> Message-ID: One option in which we have both reactor and deferred :) def do_low_level_stuff(deferred) try: deferred.callback(self.get_handle()) except SystemDelayException: reactor.callLater(1, do_low_level_stuff, deferred) def do_stuff(): deferred = Deferred() do_low_level_stuff(deferred) return deferred Cheers On 3 September 2014 12:32, vikas kumar wrote: > Hi, > > I am little new to twisted. > I've a function(get_handle) which returns a handle. > > But get_handle() may throw an exception(SystemDelayException) because of > some delay in system. > My requirement is : When I get SystemDelayException re-attempt get_handle() > and return only when handle is available. How do I use deferred/reactor in > this scenario? > > try: > handle = yield self.get_handle() > except SystemDelayException: > // code to re-attempt get_handle and return only when handle is > available > > Regards > Vikas > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Adi Roiban From jmp at editshare.com Wed Sep 3 06:53:05 2014 From: jmp at editshare.com (Justin Mazzola Paluska) Date: Wed, 03 Sep 2014 08:53:05 -0400 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140902135052.20413.975133368.divmod.xquotient.1791@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> Message-ID: <54070F31.1010601@editshare.com> On 09/02/2014 09:50 AM, exarkun at twistedmatrix.com wrote: >>> >>> I dig deeper and I found out that since I was using >>> reactor.run(installSignalHandlers=False) _SIGCHLDWaker was not >>> installed. >>> >>> I have switched to using just reactor.run() and the process is now >>> killed. >> >> Without my workaround, I continue to have the problem with the >> gtk2reactor. > > Have you reported this bug? No, not yet since I don't have a 100%-reliable test case since it's mostly related to races between the child process, the parent process, and signal handling. However, if that's not required, I'd love to get it in the system. --Justin From adi at roiban.ro Wed Sep 3 07:05:06 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 14:05:06 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140902135052.20413.975133368.divmod.xquotient.1791@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> Message-ID: On 2 September 2014 14:50, wrote: > On 01:38 pm, jmp at editshare.com wrote: >> [snip] >> >> Without my workaround, I continue to have the problem with the >> gtk2reactor. > > Have you reported this bug? > > Jean-Paul In my initial use case, signal handlers were not installed since I wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can use my custom Unix Daemon and Windows Service adapters. I understand why reactor.run has the installSignalHandlers argument. Do you think it would make sense to have some kind of public API which would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do reactor.stop()) but would install the SIGCHLD handler? Thanks! -- Adi Roiban From exarkun at twistedmatrix.com Wed Sep 3 07:39:46 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 13:39:46 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> Message-ID: <20140903133946.20413.1728050305.divmod.xquotient.1804@top> On 01:05 pm, adi at roiban.ro wrote: >On 2 September 2014 14:50, wrote: >>On 01:38 pm, jmp at editshare.com wrote: >>> >[snip] >>> >>>Without my workaround, I continue to have the problem with the >>>gtk2reactor. >> >>Have you reported this bug? >> >>Jean-Paul > >In my initial use case, signal handlers were not installed since I >wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can >use my custom Unix Daemon and Windows Service adapters. > >I understand why reactor.run has the installSignalHandlers argument. > >Do you think it would make sense to have some kind of public API which >would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do >reactor.stop()) but would install the SIGCHLD handler? Yes. Providing more fine-grain control over signal handlers would be a fine improvement. Another fine improvement would be making child processes work even if a SIGCHLD handler cannot be installed (for example, by polling children periodically, by using wait() in a sidecar process, or by using a better system-specific child process monitoring API (eg kqueue's EVFILT_PROC)). Jean-Paul From adi at roiban.ro Wed Sep 3 09:27:05 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 16:27:05 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140903133946.20413.1728050305.divmod.xquotient.1804@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> Message-ID: On 3 September 2014 14:39, wrote: > On 01:05 pm, adi at roiban.ro wrote: >> >> On 2 September 2014 14:50, wrote: >>> >>> On 01:38 pm, jmp at editshare.com wrote: >>>> >>>> >> [snip] >>>> >>>> >>>> Without my workaround, I continue to have the problem with the >>>> gtk2reactor. >>> >>> >>> Have you reported this bug? >>> >>> Jean-Paul >> >> >> In my initial use case, signal handlers were not installed since I >> wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can >> use my custom Unix Daemon and Windows Service adapters. >> >> I understand why reactor.run has the installSignalHandlers argument. >> >> Do you think it would make sense to have some kind of public API which >> would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do >> reactor.stop()) but would install the SIGCHLD handler? > > > Yes. Providing more fine-grain control over signal handlers would be a fine > improvement. Do you have any suggestion for how the calls should be made? reactor.run(installSignalHandlers=True, installStopHandlers=False) or reactor.installStopHandlers = False reactor.run() > Another fine improvement would be making child processes work even if a > SIGCHLD handler cannot be installed (for example, by polling children > periodically, by using wait() in a sidecar process, or by using a better > system-specific child process monitoring API (eg kqueue's EVFILT_PROC)). > I see that GlibReactorBase inherits from PosixReactorBase so it should install the SIGCHLD handler... this should not be the reason why gtk2 reactor don't work. As a poor man's fix and Unix independent fix maybe we can call reapAllProcess in a task.LoopingCall... What are the reasons why SIGCHLD handler cannot be installed? I am asking since I hope it could help me understant where and how to enable the poor man's fix... and how to fill the bug report. kqueue's EVFILT_PROC would be great, but I guess that we still need a general fix --------- For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles but keep SIGCHLD I do: # Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK handlers _SignalReactorMixin._handleSignals = lambda self: None reactor.run() -- Adi Roiban From adi at roiban.ro Wed Sep 3 09:49:36 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 16:49:36 +0100 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows Message-ID: Hi, If I call spawnProcess with a bad executable, on Unix the process will exit with exit code 1 while on Windows an OSError is raised. I am working on a multi-OS software and to make my life easier I ended up with this hack inspired by _BackRelay https://gist.github.com/adiroiban/bac493f00ce5e94738ce ---- Is there something already in Twisted doing this? Am I reinventing the wheel? If not, do you think that it would help to update _BackRelay to support timeout and cross-os error handling? -- Adi Roiban From exarkun at twistedmatrix.com Wed Sep 3 10:51:53 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 16:51:53 -0000 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows In-Reply-To: References: Message-ID: <20140903165153.20413.1282979568.divmod.xquotient.1807@top> On 03:49 pm, adi at roiban.ro wrote: >Hi, > >If I call spawnProcess with a bad executable, on Unix the process will >exit with exit code 1 while on Windows an OSError is raised. Ideally, these two implementations of the same interface would have the same error behavior in this case. In other words, this seems like a bug to me. Jean-Paul From adi at roiban.ro Wed Sep 3 11:17:01 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 18:17:01 +0100 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows In-Reply-To: References: Message-ID: On 3 September 2014 16:49, Adi Roiban wrote: > Hi, > > If I call spawnProcess with a bad executable, on Unix the process will > exit with exit code 1 while on Windows an OSError is raised. > > I am working on a multi-OS software and to make my life easier I ended > up with this hack inspired by _BackRelay > > https://gist.github.com/adiroiban/bac493f00ce5e94738ce > > ---- > > Is there something already in Twisted doing this? Am I reinventing the wheel? > > If not, do you think that it would help to update _BackRelay to > support timeout and cross-os error handling? > Searching the Twisted trac I found this ticket http://twistedmatrix.com/trac/ticket/4184 Maybe we can leave spawnProcess as the low-level API and then create another wrapper which will handle/raise the errors in an consistent way. For the case where command is not found, I prefer the behaviour from Windows where an OSError is raise, rather than looking at errReceived and trying to guess is execvpe call failed or not. -- Adi Roiban From adi at roiban.ro Wed Sep 3 11:25:50 2014 From: adi at roiban.ro (Adi Roiban) Date: Wed, 3 Sep 2014 18:25:50 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> Message-ID: Just for reference: For the gkt2 reactor problem I found this bug report https://twistedmatrix.com/trac/ticket/5289 ... I guess that there is a problem with gtk2 and spawnProcess On 3 September 2014 16:27, Adi Roiban wrote: > On 3 September 2014 14:39, wrote: >> On 01:05 pm, adi at roiban.ro wrote: >>> >>> On 2 September 2014 14:50, wrote: >>>> >>>> On 01:38 pm, jmp at editshare.com wrote: >>>>> >>>>> >>> [snip] >>>>> >>>>> >>>>> Without my workaround, I continue to have the problem with the >>>>> gtk2reactor. >>>> >>>> >>>> Have you reported this bug? >>>> >>>> Jean-Paul >>> >>> >>> In my initial use case, signal handlers were not installed since I >>> wanted a custom behaviour for SIGINT, SIGTERM, SIGBREAK so that I can >>> use my custom Unix Daemon and Windows Service adapters. >>> >>> I understand why reactor.run has the installSignalHandlers argument. >>> >>> Do you think it would make sense to have some kind of public API which >>> would skip handlers for SIGINT, SIGTERM, SIGBREAK (as they only do >>> reactor.stop()) but would install the SIGCHLD handler? >> >> >> Yes. Providing more fine-grain control over signal handlers would be a fine >> improvement. > > Do you have any suggestion for how the calls should be made? > > reactor.run(installSignalHandlers=True, installStopHandlers=False) > > or > > reactor.installStopHandlers = False > reactor.run() > > >> Another fine improvement would be making child processes work even if a >> SIGCHLD handler cannot be installed (for example, by polling children >> periodically, by using wait() in a sidecar process, or by using a better >> system-specific child process monitoring API (eg kqueue's EVFILT_PROC)). >> > > I see that GlibReactorBase inherits from PosixReactorBase so it should > install the SIGCHLD handler... this should not be the reason why gtk2 > reactor don't work. > > As a poor man's fix and Unix independent fix maybe we can call > reapAllProcess in a task.LoopingCall... > What are the reasons why SIGCHLD handler cannot be installed? > > I am asking since I hope it could help me understant where and how to > enable the poor man's fix... and how to fill the bug report. > > kqueue's EVFILT_PROC would be great, but I guess that we still need a > general fix > > --------- > > For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles > but keep SIGCHLD I do: > > # Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK handlers > _SignalReactorMixin._handleSignals = lambda self: None > reactor.run() > > > -- > Adi Roiban -- Adi Roiban From vikas.c.kumar at oracle.com Wed Sep 3 11:52:13 2014 From: vikas.c.kumar at oracle.com (vikas kumar) Date: Wed, 03 Sep 2014 23:22:13 +0530 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: References: <5406FC34.9080109@oracle.com> Message-ID: <5407554D.2040509@oracle.com> Hi Jonas, This is nice. But still we are retrying it repeatedly at regular interval(10). Can't we get the notified asynchronously as and when handle is available. Regards Vikas On 9/3/2014 6:10 PM, Jonas Brunsgaard wrote: > I would fiddle around with something like this. > > from twisted.internet.defer import inlineCallbacks, returnValue, Deferred > from twisted.internet import reactor > > > @inlineCallbacks > def foo(self, retries=10, interval=10): > while True: > try: > returnValue((yield self.get_handle())) > except SystemDelayException as e: > if retries > 0: > retries -= 1 > d = Deferred() > reactor.callLater(interval, d.callback, None) > yield d > else: > raise e > > On Wed, Sep 3, 2014 at 1:32 PM, vikas kumar > wrote: > > Hi, > > I am little new to twisted. > I've a function(get_handle) which returns a handle. > > But get_handle() may throw an exception(SystemDelayException) > because of some delay in system. > My requirement is : When I get SystemDelayException re-attempt > get_handle() and return only when handle is available. How do I > use deferred/reactor in this scenario? > > try: > handle = yield self.get_handle() > except SystemDelayException: > // code to re-attempt get_handle and return only when handle > is available > > Regards > Vikas > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Wed Sep 3 11:55:17 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 17:55:17 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> Message-ID: <20140903175517.20413.1988429554.divmod.xquotient.1845@top> On 03:27 pm, adi at roiban.ro wrote: >On 3 September 2014 14:39, wrote: >> >>Yes. Providing more fine-grain control over signal handlers would be >>a fine >>improvement. > >Do you have any suggestion for how the calls should be made? > >reactor.run(installSignalHandlers=True, installStopHandlers=False) Perhaps. >or > >reactor.installStopHandlers = False >reactor.run() Probably not this one. Setting attributes on random things is kind of sad. :) It might be nice to try to be somewhat flexible - in case there's some reason to change what signals the reactor wants to handle in the future. Perhaps: reactor.run(installSignalHandlers={SIGCHLD}) An entirely different direction could be to make this bit of configuration into initialization for the reactor. from twisted.internet.epollreactor import install install(installSignalHandlers={SIGCHLD}) from twisted.internet import reactor ... reactor.run() By keeping these details away from `IReactorCore.run`, that method remains maximally useful. For example, if you could set up the reactor this way, a normal `twistd` plugin would still be able to benefit from your choice, even with twistd's naive call of `reactor.run()` with no extra arguments. Application code calling these `install` functions is already supported (it's how you select a specific reactor, after all). Some of the install functions even accept arguments already. This would actually eliminate another existing issue - `IReactorCore.run` is actually defined to take no arguments. The implementations ignore this because someone thought it was important to be able to disable installation of signal handlers. > >>Another fine improvement would be making child processes work even if >>a >>SIGCHLD handler cannot be installed (for example, by polling children >>periodically, by using wait() in a sidecar process, or by using a >>better >>system-specific child process monitoring API (eg kqueue's >>EVFILT_PROC)). > >I see that GlibReactorBase inherits from PosixReactorBase so it should >install the SIGCHLD handler... this should not be the reason why gtk2 >reactor don't work. Gtk messes with signals too. There are confusing order-of-execution dependencies and Gtk changes subtly from release to release, re-breaking things after we fix them or changing them to be broken in a different way. So that's *why* it probably doesn't work with Gtk2 reactor - if not *how*. ;) I think I missed the explanation of what in particular wasn't working with Gtk2 reactor though. >As a poor man's fix and Unix independent fix maybe we can call >reapAllProcess in a task.LoopingCall... >What are the reasons why SIGCHLD handler cannot be installed? Either because you want to run the reactor in a non-main thread (where Python won't let you install signal handlers for the good reason that mixing signals and threads has crazy behavior) or because you want to use a different library that depends on having its own SIGCHLD handler and you're not interested in Twisted's process support. >I am asking since I hope it could help me understant where and how to >enable the poor man's fix... and how to fill the bug report. > >kqueue's EVFILT_PROC would be great, but I guess that we still need a >general fix Perhaps, perhaps not. A general fix might be less code but having half a dozen specialized fixes, one for each reactor, would also still fix the problem. The different reactor implementations are essentially the big piles of specialized fixes needed to do non-blocking I/O on different platforms. This would just be a little more of the same. The sidecar process is an example of a general fix, though. The idea there is that Twisted itself runs a private child process (perhaps only when the first call to spawnProcess is made). It talks to that process using a file descriptor. That process can install a SIGCHLD handler (because Twisted owns it, application developers don't get to say they don't want one installed) or use another more invasive strategy for child process management. When you want to spawn a process, the main process tells the sidecar to do it. The sidecar relays traffic between the child and the original parent (or does something involving passing file descriptors across processes). This removes the need to ever install a SIGCHLD handler in the main process. It also probably enables some optimizations (reapProcesses is O(N!) on the number of child processes right now) that are very tricky or impossible otherwise. Jean-Paul >--------- > >For the record: Right now, to ignore SIGINT, SIGTERM, SIGBREAK handles >but keep SIGCHLD I do: > ># Patch base reactor to not install SIGINT, SIGTERM and SIGBREAK >handlers >_SignalReactorMixin._handleSignals = lambda self: None >reactor.run() > > >-- >Adi Roiban From exarkun at twistedmatrix.com Wed Sep 3 12:00:48 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 18:00:48 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140903175517.20413.1988429554.divmod.xquotient.1845@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> Message-ID: <20140903180048.20413.1938651844.divmod.xquotient.1848@top> On 05:55 pm, exarkun at twistedmatrix.com wrote: > >Gtk messes with signals too. There are confusing order-of-execution >dependencies and Gtk changes subtly from release to release, re- >breaking things after we fix them or changing them to be broken in a >different way. > >So that's *why* it probably doesn't work with Gtk2 reactor - if not >*how*. ;) > >I think I missed the explanation of what in particular wasn't working >with Gtk2 reactor though. Oh right, that was where Justin's bug was. Jean-Paul From exarkun at twistedmatrix.com Wed Sep 3 12:07:03 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 18:07:03 -0000 Subject: [Twisted-Python] spawnProcess with timeout and same error handling on Unix and Windows In-Reply-To: References: Message-ID: <20140903180703.20413.1947044842.divmod.xquotient.1855@top> On 05:17 pm, adi at roiban.ro wrote: >On 3 September 2014 16:49, Adi Roiban wrote: >>Hi, >> >>If I call spawnProcess with a bad executable, on Unix the process will >>exit with exit code 1 while on Windows an OSError is raised. >> >>I am working on a multi-OS software and to make my life easier I ended >>up with this hack inspired by _BackRelay >> >>https://gist.github.com/adiroiban/bac493f00ce5e94738ce >> >>---- >> >>Is there something already in Twisted doing this? Am I reinventing the >>wheel? >> >>If not, do you think that it would help to update _BackRelay to >>support timeout and cross-os error handling? > >Searching the Twisted trac I found this ticket >http://twistedmatrix.com/trac/ticket/4184 > >Maybe we can leave spawnProcess as the low-level API and then create >another wrapper which will handle/raise the errors in an consistent >way. > >For the case where command is not found, I prefer the behaviour from >Windows where an OSError is raise, rather than >looking at errReceived and trying to guess is execvpe call failed or >not. It is more convenient, certainly. Unfortunately on POSIX the underlying API isn't synchronous - which is why the POSIX implementation doesn't deliver the exception synchronously. So to be uniform we probably need to make the Windows behavior asynchronous. I suggest we can improve the situation by reporting the error more in an easier-to-recognize way when we report it asynchronously, though. For example, instead of forcing the application to parse stderr we could deliver a different exception type to processEnded. A better solution would be to make `spawnProcess` properly asynchronous - have it return a `Deferred` that fires with an `IProcessTransport` and let that `Deferred` fire with a `Failure` if there is a problem creating the new process. But that's probably more like "introduce an API to replace `spawnProcess`" (for which I believe there is also a ticket). Jean-Paul From exarkun at twistedmatrix.com Wed Sep 3 12:07:42 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Wed, 03 Sep 2014 18:07:42 -0000 Subject: [Twisted-Python] Return only when value/handle is available In-Reply-To: <5407554D.2040509@oracle.com> References: <5406FC34.9080109@oracle.com> <5407554D.2040509@oracle.com> Message-ID: <20140903180742.20413.944089773.divmod.xquotient.1856@top> On 05:52 pm, vikas.c.kumar at oracle.com wrote: >Hi Jonas, > >This is nice. >But still we are retrying it repeatedly at regular interval(10). Can't >we get the notified asynchronously as and when handle is available. You tell us. What's a handle? Where do you get it from? Does that system produce an event when it is ready? Jean-Paul From glyph at twistedmatrix.com Wed Sep 3 15:28:40 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 3 Sep 2014 14:28:40 -0700 Subject: [Twisted-Python] what is a non-class class? In-Reply-To: <1709945.N3rAP3M3F9@i5> References: <1522503.xpp1Bo2WKl@i5> <1925033.An9b9qoBL0@i5> <8ED93F16-3B3D-44D6-B0C8-7E46FF9293AA@twistedmatrix.com> <1709945.N3rAP3M3F9@i5> Message-ID: On Sep 3, 2014, at 12:55 AM, Wolfgang Rohdewald wrote: > Am Mittwoch, 3. September 2014, 00:29:59 schrieb Glyph: >>> That is my problem. How do I know if the object is of a non-class class? >> >> isinstance(something, (types.ClassType, type)). > > but that will make it possible to attach a method even to type "int" Yes, that's fine. 'int' has methods, there's no reason that those methods couldn't be serialized by jelly. For example: >>> (3).conjugate The bound method object there is the sort of thing that we're talking about. Particularly since we might be talking about a subclass of 'int' with its own overridden conjugate method, and the im_class attribute says which class the method's function actually came from. > Python2: > >>>> isinstance(int, type) > True > >>>>> isinstance(object, type) >> True >>>>> isinstance(object(), type) >> False > Sure but the test only gets the class and it should certainly not instantiate it. Right; the point is that you get a thing, and that thing may be a class object _or_ it might be an instance object, and if it's an instance that's invalid. > Meanwhile I believe it is best to simply remove the test since python itself > will reject most: > >>>> a=int >>>> a.x=5 > Traceback (most recent call last): > File "", line 1, in > TypeError: can't set attributes of built-in/extension type 'int' This is about serializing and deserializing existing methods, not assigning attributes to instances. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Thu Sep 4 01:26:39 2014 From: adi at roiban.ro (Adi Roiban) Date: Thu, 4 Sep 2014 08:26:39 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140903175517.20413.1988429554.divmod.xquotient.1845@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> Message-ID: On 3 September 2014 18:55, wrote: > On 03:27 pm, adi at roiban.ro wrote: >> >> On 3 September 2014 14:39, wrote: [snip] >> Do you have any suggestion for how the calls should be made? >> >> reactor.run(installSignalHandlers=True, installStopHandlers=False) > > > Perhaps. [snip] > It might be nice to try to be somewhat flexible - in case there's some > reason to change what signals the reactor wants to handle in the future. > Perhaps: > > reactor.run(installSignalHandlers={SIGCHLD}) > > An entirely different direction could be to make this bit of configuration > into initialization for the reactor. > > from twisted.internet.epollreactor import install > install(installSignalHandlers={SIGCHLD}) > > from twisted.internet import reactor > ... > reactor.run() > > By keeping these details away from `IReactorCore.run`, that method remains > maximally useful. For example, if you could set up the reactor this way, a > normal `twistd` plugin would still be able to benefit from your choice, even > with twistd's naive call of `reactor.run()` with no extra arguments. > > Application code calling these `install` functions is already supported > (it's how you select a specific reactor, after all). Some of the install > functions even accept arguments already. > > This would actually eliminate another existing issue - `IReactorCore.run` is > actually defined to take no arguments. The implementations ignore this > because someone thought it was important to be able to disable installation > of signal handlers. I am happy to have a simple reactor.run() and move installSignalHandlers somewhere else. working with install(installSignalHandlers={SIGCHLD}) seems a bit complicated, as I assume that many developers rely on the automatic reactor installation. In the same time, I assume that 'installSignalHandlers' argument would be supported by all reactors this is why maybe we can have something like: from twisted.internet import reactor def customHandler(signum, frame): pass reactor.installSignalHandlers( SIGCHLD=True, # Install default handler SIGTERM=None, # Don't install handler SIGINT=customHandler, # Install custom handler # SIGBREAK is not request so that default handler is installed. ) # reactor.installSignalHandlers() installs all default handlers. reactor.run() ---- reactor.run(InstallSignalHandlers=True|False) would be deprecated. In case reactor.installSignalHandlers is not called before run(), all default handlers will be installed. [snip] > The sidecar process is an example of a general fix, though. The idea there > is that Twisted itself runs a private child process (perhaps only when the > first call to spawnProcess is made). It talks to that process using a file > descriptor. That process can install a SIGCHLD handler (because Twisted > owns it, application developers don't get to say they don't want one > installed) or use another more invasive strategy for child process > management. When you want to spawn a process, the main process tells the > sidecar to do it. The sidecar relays traffic between the child and the > original parent (or does something involving passing file descriptors across > processes). > > This removes the need to ever install a SIGCHLD handler in the main process. > It also probably enables some optimizations (reapProcesses is O(N!) on the > number of child processes right now) that are very tricky or impossible > otherwise. > > Jean-Paul Thanks for the details regarding the side-process dedicated to child process management. Not sure if we need a separate ticket for that, or add it as a comment to https://twistedmatrix.com/trac/ticket/5710 Thanks! -- Adi Roiban From sangiovanni at nweb.it Thu Sep 4 04:52:07 2014 From: sangiovanni at nweb.it (Fabio Sangiovanni) Date: Thu, 4 Sep 2014 12:52:07 +0200 Subject: [Twisted-Python] Graceful shutdown of twistd application Message-ID: Hello list, I need to implement a graceful shutdown procedure for a twistd application. The application is made up of two services: an internet.TCPClient and an internet.TCPServer. They're glued together with a MultiService instance, which is in turn set to have 'application' as parent. The server and the client work together, making a proxy (SMTP server and AMQP client). My goal is the following: - intercept a SIGTERM signal - 'block' on the server side: since it's SMTP I get this by setting a variable that makes the server return tempfails (4xx) for new messages, while keeping current sessions active - wait until current requests are satisfied (I keep a dictionary of current pending messages) - shut the whole thing down What is the best solution for this use case? It's not really clear to me how to catch SIGTERM and handle pending requests *before* the underlying services start to shutdown (i.e. even addSystemEventTrigger('before', 'shutdown', callable) is called too late for my needs). Thank you very much for your help! Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Thu Sep 4 05:59:15 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 11:59:15 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> Message-ID: <20140904115915.20413.342593090.divmod.xquotient.1871@top> On 07:26 am, adi at roiban.ro wrote: >On 3 September 2014 18:55, wrote: >>On 03:27 pm, adi at roiban.ro wrote: >>> >>>On 3 September 2014 14:39, wrote: >[snip] >>>Do you have any suggestion for how the calls should be made? >>> >>>reactor.run(installSignalHandlers=True, installStopHandlers=False) >> Also note there is an old, widely scoped ticket: https://twistedmatrix.com/trac/ticket/2415 with some more stuff (not necessarily directly related to your comments on signal handling) on it. What would be really nice is if someone collected *all* of the complaints about `spawnProcess` into one place and integrated solutions to them into a design for a replacement. :) Jean-Paul From exarkun at twistedmatrix.com Thu Sep 4 06:02:48 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 12:02:48 -0000 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: References: Message-ID: <20140904120248.20413.662616188.divmod.xquotient.1874@top> On 10:52 am, sangiovanni at nweb.it wrote: >Hello list, > >I need to implement a graceful shutdown procedure for a twistd >application. >The application is made up of two services: an internet.TCPClient and >an internet.TCPServer. >They're glued together with a MultiService instance, which is in turn >set >to have 'application' as parent. >The server and the client work together, making a proxy (SMTP server >and >AMQP client). > >My goal is the following: >- intercept a SIGTERM signal >- 'block' on the server side: since it's SMTP I get this by setting a >variable that makes the server return tempfails (4xx) for new messages, >while keeping current sessions active >- wait until current requests are satisfied (I keep a dictionary of >current >pending messages) >- shut the whole thing down This is exactly what before shutdown triggers are for. Alternatively, use the higher-level API and implement `stopService` on one of your services. Either way, return a `Deferred` that only fires when you're satisfied it is time for shutdown to proceed. You said before shutdown triggers are too late but you didn't say why. I think that's based on a misunderstanding - but if not, then explain why it doesn't work for your scenario. Jean-Paul >What is the best solution for this use case? It's not really clear to >me >how to catch SIGTERM and handle pending requests *before* the >underlying >services start to shutdown (i.e. even addSystemEventTrigger('before', >'shutdown', callable) is called too late for my needs). > >Thank you very much for your help! > >Fabio From sangiovanni at nweb.it Thu Sep 4 06:36:00 2014 From: sangiovanni at nweb.it (Fabio Sangiovanni) Date: Thu, 4 Sep 2014 14:36:00 +0200 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: <20140904120248.20413.662616188.divmod.xquotient.1874@top> References: <20140904120248.20413.662616188.divmod.xquotient.1874@top> Message-ID: On Thu, Sep 4, 2014 at 2:02 PM, wrote: > > > You said before shutdown triggers are too late but you didn't say why. I > think that's based on a misunderstanding - but if not, then explain why it > doesn't work for your scenario. > Hi, thanks for your reply. I've tried the following: def sleep(secs): log.msg('from within trigger') d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10) This is what I can see in the logs: Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] Received SIGTERM, shutting down. Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] from within trigger Sep 4 14:25:06 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] will retry in 2 seconds Sep 4 14:25:06 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] Stopping factory <__builtin__.RabbitMQClientFactory instance at 0x00000000057172c0> Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] (TCP Port 10025 Closed) Sep 4 14:25:06 prepyproxy01 proxy [4924]: [-] Stopping factory <__builtin__.TempfailingESMTPFactory instance at 0x00000000057172a0> Sep 4 14:25:09 prepyproxy01 proxy [4924]: [-] Starting factory <__builtin__.RabbitMQClientFactory instance at 0x00000000057172c0> Sep 4 14:25:09 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] [rmq01] RabbitMQ connection established Sep 4 14:25:16 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] will retry in 2 seconds Sep 4 14:25:16 prepyproxy01 proxy [4924]: [TwistedProtocolConnection,client] Stopping factory <__builtin__.RabbitMQClientFactory instance at 0x00000000057172c0> Sep 4 14:25:16 prepyproxy01 proxy [4924]: [-] Main loop terminated. Sep 4 14:25:16 prepyproxy01 proxy [4924]: [-] Server Shut Down. It seems to me that the shutdown phase doesn't wait for the deferred to fire before stopping my client and server. To be clear: my expected result is: - SIGTERM - pause 10s - client/server shutdown I am surely missing something, but I really can't figure out what. Oh, for the records: I'm using Twisted 13.2.0 on Pypy. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi.libotean at proatria.com Thu Sep 4 06:51:58 2014 From: adi.libotean at proatria.com (Adi Libotean) Date: Thu, 04 Sep 2014 15:51:58 +0300 Subject: [Twisted-Python] High throughput database logger Message-ID: <5408606E.2030401@proatria.com> Hi, I'm looking at various options for implementing a high throughput database logger that will work with Twisted. My requirements, listed by importance: 1) small memory footprint 2) high speed 3) low garbage generation The application I'm working on runs continuously (24/7). I've experimented a bit with pysqlite and Twisted to see which approach is better suited (see attached example). ---- Question 1: I noticed that all of the Twisted based versions are very slow compared to the plain sqlite3 test. This seems to be caused by atomic transaction management, namely a commit after each insert. Would be interested to know if there is a simple way to avoid this and do my own transaction management (aka batch commit). One other thing is the greatly varying amounts of garbage generated (peak memory) and memory usage between the Twisted variants. ---- Question 2: I would have expected B (Twisted ADBAPI) to behave very similar to C/E since I'm using a connection pool of size 1 and all requests are queued and handled sequentially. Could any of you please give me some pointers as to why this is happening? ---- Question 3: Even though objgraph lists the exact same reference count once the code is ran, the amount of used memory greatly differs. Any ideas what might be causing this? Any suggestions and/or pointers on how to improve/do this are more than welcome. Thank you for your time, Adrian -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: db-insert-test.py URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: results.txt URL: From exarkun at twistedmatrix.com Thu Sep 4 06:55:49 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 12:55:49 -0000 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: References: <20140904120248.20413.662616188.divmod.xquotient.1874@top> Message-ID: <20140904125549.20413.481091133.divmod.xquotient.1886@top> On 12:36 pm, sangiovanni at nweb.it wrote: >On Thu, Sep 4, 2014 at 2:02 PM, wrote: >> >> >>You said before shutdown triggers are too late but you didn't say why. >>I >>think that's based on a misunderstanding - but if not, then explain >>why it >>doesn't work for your scenario. > >Hi, thanks for your reply. > >I've tried the following: > >def sleep(secs): > log.msg('from within trigger') > d = defer.Deferred() > reactor.callLater(secs, d.callback, None) > return d > >reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10) All 'before' trigger are run concurrently. If you're using `Application` then your `sleep` trigger runs concurrently with the application's `stopService` trigger (because `Application` has its stopService added as another 'before' shutdown' trigger alongside yours). If you want to delay your application shutdown, you need to cooperate a little more closely with it. Either attach your application shutdown code as a callback to the sleep Deferred or move the sleep into the stopService implementation of one of the services on your application and trigger the remaining stopService calls (eg the stopService call on the MultiService you mentioned) when the sleep Deferred fires there. Jean-Paul From exarkun at twistedmatrix.com Thu Sep 4 07:20:15 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 04 Sep 2014 13:20:15 -0000 Subject: [Twisted-Python] High throughput database logger In-Reply-To: <5408606E.2030401@proatria.com> References: <5408606E.2030401@proatria.com> Message-ID: <20140904132015.20413.679407771.divmod.xquotient.1912@top> On 12:51 pm, adi.libotean at proatria.com wrote: >Hi, > >I'm looking at various options for implementing a high throughput >database logger that will work with Twisted. > >My requirements, listed by importance: > >1) small memory footprint >2) high speed >3) low garbage generation > >The application I'm working on runs continuously (24/7). I've >experimented a bit with pysqlite and Twisted to see which approach is >better suited (see attached example). > >---- > >Question 1: I noticed that all of the Twisted based versions are very >slow compared to the plain sqlite3 test. This seems to be caused by >atomic transaction management, namely a commit after each insert. Not only this but in some of the Twisted versions you've introduced a round-trip communication from the reactor thread to a worker thread between each operation. This will greatly impact throughput by adding lots of latency to each insert. >Would be interested to know if there is a simple way to avoid this and >do my own transaction management (aka batch commit). Using twisted.enterprise.adbapi? You could probably hack something horrible together but it would definitely be a hack. I suggest you take a look at adbapi2 instead - . >One other thing is the greatly varying amounts of garbage generated >(peak memory) and memory usage between the Twisted variants. "Garbage" and "peak memory" are different things. The Twisted-using version does a lot more - and some of your Twisted-using versions put the *entire* data set into memory (in a vastly expanded form, where each insert is represented by multiple large objects including Deferreds). So it's not too surprising the memory usage is greater. >---- > >Question 2: I would have expected B (Twisted ADBAPI) to behave very >similar to C/E since I'm using a connection pool of size 1 and all >requests are queued and handled sequentially. > >Could any of you please give me some pointers as to why this is >happening? You didn't actually label the code with these letters. :) I'm guessing B is the `adbapi` function, C is `inline_callbacks`, and E is `cooperator`. Also you didn't say in what respect you expected them to behavior similarly. You expected their memory usage to be the same? You expected their runtime to be the same? You expected them to put the same data into the database? As far as memory usage goes, B uses lots of memory for the same reason `semaphore` (D?) uses lots of memory. You queue up the entire dataset in memory as piles of tuples, lists, Deferreds, etc. adbapi might be executing the operations one at a time, but the *loop* inside `adbapi` runs all the way to the end all in one go. It starts every one of those `runOperation`s before any of them (probably) has a chance to execute. >---- > >Question 3: Even though objgraph lists the exact same reference count >once the code is ran, the amount of used memory greatly differs. Any >ideas what might be causing this? Hopefully the above helps explain this. Something else you might consider is batching up your inserts (not just only committing after a batch of inserts). Since SQLite3 can only write from a single thread at a time, you're effectively limited to serialized inserts - so it doesn't make sense to try to start a second insert before the first has finished. When the first finishes, if 50 more data points have arrived, you should do one insert for all 50 of those - not 50 inserts each for one piece of data. This cuts off a bunch of your overhead - Python objects, round- trip latency for inter-thread communication, function calls, etc. Jean-Paul >Any suggestions and/or pointers on how to improve/do this are more than >welcome. > >Thank you for your time, >Adrian From adi at roiban.ro Fri Sep 5 00:30:35 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 5 Sep 2014 07:30:35 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140904115915.20413.342593090.divmod.xquotient.1871@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> Message-ID: On 4 September 2014 12:59, wrote: [snip] > What would be really nice is if someone collected *all* of the complaints > about `spawnProcess` into one place and integrated solutions to them into a > design for a replacement. :) > > Jean-Paul Since I can not create wiki pages here are my notes: To do: * Always return a deferred. #2415 * Provide the option to timeout the execution... and on timeout errback with a dedicated failure #2415 * Add childs to the same process group #2415 * allways errback when failing to spawn (rather than raising OSError) and errback with a different failure when failing to spawn the executable #4184 * on Unix provide a method to install SIGCHLD handler, independent of the general reactor.run() signal handlers. ... maybe related to #5710 * Allow passing Unicode environment (at least on Windows) maybe needed for Python3 #5968 * Allow passing Unicode command and arguments on Windows (current not supported by pywin32 CreateProcess) #6470... maybe needed for Python3 #5968 * Improve fork/exec for speed and memory usage #5710 .... maybe by creating a auxiliary process which handles forks and child management. * Execute as a different account in Windows * Spawn only after reactor starts ? I am ok with forking before. * Support client endpoint? ------- For a new design: * I like the ProcessProtocol API * spawnProcess will always return a deferred... for backward compatibility we need to decide a name for this new method which returns a deferred. Maybe createProcess or connectProcess * create spawnUnixProcess which has childFDs uid/gid * create spawnWindowsProcess which has Windows specific options * createProcess / connectProcess will call spawnUnixProcess or spawnWindowsProcess depending on current OS * ProcessProtocol.processEnded is similar to connectionLost.. so maybe it can be rename it and add other methods so that IProcessProtocol can inherit from IProtocol. I guess that this will help with connectors. ------ I think that child management problem, does not affect the public API so I am not discussing it here. Timeouts are handled at protocol level and I see that we have 2 timeout types: total execution time and timeout after last child data was received. ----- Does it make sense to put it on wiki or what should be the Twisted Enhancement Proposal process? Can I get 'create page' rights on Trac? Thanks! -- Adi Roiban From sangiovanni at nweb.it Fri Sep 5 02:42:17 2014 From: sangiovanni at nweb.it (Fabio Sangiovanni) Date: Fri, 5 Sep 2014 10:42:17 +0200 Subject: [Twisted-Python] Graceful shutdown of twistd application In-Reply-To: <20140904125549.20413.481091133.divmod.xquotient.1886@top> References: <20140904120248.20413.662616188.divmod.xquotient.1874@top> <20140904125549.20413.481091133.divmod.xquotient.1886@top> Message-ID: Hi, I just wanted to give an update about the way I got it, in case this could be useful for somebody. I essentially inherited from service.MultiService like this: class GracefulMultiService(service.MultiService): def stopService(self): d = defer.succeed(None) d.addCallback(self._wait_for_your_cleanup_code_to_complete) d.addCallback(lambda _: self) d.addCallback(service.MultiService.stopService) return d @defer.inlineCallbacks def _wait_for_your_cleanup_code_to_complete(self): [...] yield sleep(10) # cleanup code example def sleep(secs): d = defer.Deferred() reactor.callLater(secs, d.callback, None) return d Thanks again for your help! On Thu, Sep 4, 2014 at 2:55 PM, wrote: > On 12:36 pm, sangiovanni at nweb.it wrote: > >> On Thu, Sep 4, 2014 at 2:02 PM, wrote: >> >>> >>> >>> You said before shutdown triggers are too late but you didn't say why. I >>> think that's based on a misunderstanding - but if not, then explain why >>> it >>> doesn't work for your scenario. >>> >> >> Hi, thanks for your reply. >> >> I've tried the following: >> >> def sleep(secs): >> log.msg('from within trigger') >> d = defer.Deferred() >> reactor.callLater(secs, d.callback, None) >> return d >> >> reactor.addSystemEventTrigger('before', 'shutdown', sleep, 10) >> > > All 'before' trigger are run concurrently. If you're using `Application` > then your `sleep` trigger runs concurrently with the application's > `stopService` trigger (because `Application` has its stopService added as > another 'before' shutdown' trigger alongside yours). > > If you want to delay your application shutdown, you need to cooperate a > little more closely with it. Either attach your application shutdown code > as a callback to the sleep Deferred or move the sleep into the stopService > implementation of one of the services on your application and trigger the > remaining stopService calls (eg the stopService call on the MultiService > you mentioned) when the sleep Deferred fires there. > > Jean-Paul > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -- *Fabio Sangiovanni | System Integrator* *T* +39 0372 24525* F* +39 0372 800725 Via Dei Comizi Agrari 10, 26100 Cremona - Italia Milano - Cremona - San Francisco MailUp® La soluzione per l'invio di email e SMS mailup.it -------------- next part -------------- An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Fri Sep 5 06:13:55 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 05 Sep 2014 12:13:55 -0000 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> Message-ID: <20140905121355.20413.575214074.divmod.xquotient.2052@top> On 06:30 am, adi at roiban.ro wrote: >On 4 September 2014 12:59, wrote: >[snip] >>What would be really nice is if someone collected *all* of the >>complaints >>about `spawnProcess` into one place and integrated solutions to them >>into a >>design for a replacement. :) >> >>Jean-Paul > >Since I can not create wiki pages here are my notes: > >To do: > >* Always return a deferred. #2415 > >* Provide the option to timeout the execution... and on timeout errback >with > a dedicated failure #2415 > >* Add childs to the same process group #2415 > >* allways errback when failing to spawn (rather than raising OSError) >and > errback with a different failure when failing to spawn the executable >#4184 > >* on Unix provide a method to install SIGCHLD handler, independent of >the > general reactor.run() signal handlers. ... maybe related to #5710 > >* Allow passing Unicode environment (at least on Windows) maybe needed >for Python3 #5968 > >* Allow passing Unicode command and arguments on Windows (current not >supported by pywin32 CreateProcess) #6470... maybe needed for Python3 >#5968 > >* Improve fork/exec for speed and memory usage #5710 .... maybe by >creating > a auxiliary process which handles forks and child management. > >* Execute as a different account in Windows > >* Spawn only after reactor starts ? I am ok with forking before. > >* Support client endpoint? > >------- > >For a new design: > >* I like the ProcessProtocol API I don't. :) ProcessProtocol means you can't re-use any of your existing IProtocol implementations. Oops. It seems like a better API would let you say "Here is a protocol, hook its output up to fd 0 in the process, hook fd 1 in the process up to its input. Here is another protocol, hook fd 2 in the process up to its input" (this would probably be a common configuration - "speak some regular protocol over stdin/stdout, have a little special logic (probably logging) for stderr). On the other hand, it's possible to build this on top of ProcessProtocol and no one ever has... >* spawnProcess will always return a deferred... for backward >compatibility > we need to decide a name for this new method which returns a deferred. > Maybe createProcess or connectProcess > >* create spawnUnixProcess which has childFDs uid/gid > >* create spawnWindowsProcess which has Windows specific options > >* createProcess / connectProcess will call spawnUnixProcess or >spawnWindowsProcess > depending on current OS > >* ProcessProtocol.processEnded is similar to connectionLost.. > so maybe it can be rename it and add other methods so that >IProcessProtocol > can inherit from IProtocol. I guess that this will help with >connectors. Let's not introduce new cases where inheritance is encouraged. >------ > >I think that child management problem, does not affect the public API >so I am >not discussing it here. > >Timeouts are handled at protocol level and I see that we have 2 timeout >types: >total execution time and timeout after last child data was received. > >----- > >Does it make sense to put it on wiki or what should be the Twisted >Enhancement Proposal process? I think it's easier to maintain a document on the wiki than in threads on the mailing list. >Can I get 'create page' rights on Trac? Hm. I thought everyone had wiki edit rights (except for a small number of privileged pages) by default - so I'm not sure what's going on here or how to give you permissions. Perhaps something changed in the trac upgrade or as part of the recent ticket permission reconfiguration. Jean-Paul From adi at roiban.ro Fri Sep 5 07:14:32 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 5 Sep 2014 14:14:32 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140905121355.20413.575214074.divmod.xquotient.2052@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> <20140905121355.20413.575214074.divmod.xquotient.2052@top> Message-ID: On 5 September 2014 13:13, wrote: > On 06:30 am, adi at roiban.ro wrote: [snip] >> For a new design: >> >> * I like the ProcessProtocol API > > > I don't. :) ProcessProtocol means you can't re-use any of your existing > IProtocol implementations. Oops. > > It seems like a better API would let you say "Here is a protocol, hook its > output up to fd 0 in the process, hook fd 1 in the process up to its input. > Here is another protocol, hook fd 2 in the process up to its input" (this > would probably be a common configuration - "speak some regular protocol over > stdin/stdout, have a little special logic (probably logging) for stderr). > > On the other hand, it's possible to build this on top of ProcessProtocol and > no one ever has... I lied about ProcessProtocol API... I don't like the current API as it is ... but I like the idea of a single protocol. Maybe is not that hard to convert IProcessProtocol into IProtocol. create makeConnection, rename outRecevied to dataReceived, rename processEnded to connectionLost... rename errReceived to errorReceived I assume that most executables will implement a single protocol and adding another IProtocol to handle stderror seems complicated. I assume that the stderr protocol will want to share a lot of state/data with the stdout/stdin protocol. [snip] >> ----- >> >> Does it make sense to put it on wiki or what should be the Twisted >> Enhancement Proposal process? > > > I think it's easier to maintain a document on the wiki than in threads on > the mailing list. >> >> Can I get 'create page' rights on Trac? > > > Hm. I thought everyone had wiki edit rights (except for a small number of > privileged pages) by default - so I'm not sure what's going on here or how > to give you permissions. Perhaps something changed in the trac upgrade or > as part of the recent ticket permission reconfiguration. My bad... I tried trac/spawnProcess instead of trac/wiki/spawnProcess and I did not read the error message and assume that somehow I don't have WIKI_CREATE permissions. Here it is: https://twistedmatrix.com/trac/wiki/spawnProcess-refactoring -- Adi Roiban From p.mayers at imperial.ac.uk Fri Sep 5 08:09:37 2014 From: p.mayers at imperial.ac.uk (Phil Mayers) Date: Fri, 05 Sep 2014 15:09:37 +0100 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140905121355.20413.575214074.divmod.xquotient.2052@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> <20140905121355.20413.575214074.divmod.xquotient.2052@top> Message-ID: <5409C421.30306@imperial.ac.uk> On 05/09/14 13:13, exarkun at twistedmatrix.com wrote: > I don't. :) ProcessProtocol means you can't re-use any of your existing > IProtocol implementations. Oops. > > On the other hand, it's possible to build this on top of ProcessProtocol > and no one ever has... I do this about 3 times a year. Yes - ProcessProtocol must die! From glyph at twistedmatrix.com Fri Sep 5 12:47:09 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 5 Sep 2014 11:47:09 -0700 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: <20140905121355.20413.575214074.divmod.xquotient.2052@top> References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> <20140905121355.20413.575214074.divmod.xquotient.2052@top> Message-ID: <40FEB580-7E6A-4F19-9D40-DA4CA21C9A03@twistedmatrix.com> On Sep 5, 2014, at 5:13 AM, exarkun at twistedmatrix.com wrote: > It seems like a better API would let you say "Here is a protocol, hook its output up to fd 0 in the process, hook fd 1 in the process up to its input. Here is another protocol, hook fd 2 in the process up to its input" (this would probably be a common configuration - "speak some regular protocol over stdin/stdout, have a little special logic (probably logging) for stderr). +1 > On the other hand, it's possible to build this on top of ProcessProtocol and no one ever has... Of course we have _sort of_ done it a gajillion times: -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Fri Sep 5 12:50:15 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 5 Sep 2014 11:50:15 -0700 Subject: [Twisted-Python] spawnProcess - reapProcess not retrying on failures In-Reply-To: References: <5405A47D.3020106@editshare.com> <5405C86F.2000609@editshare.com> <20140902135052.20413.975133368.divmod.xquotient.1791@top> <20140903133946.20413.1728050305.divmod.xquotient.1804@top> <20140903175517.20413.1988429554.divmod.xquotient.1845@top> <20140904115915.20413.342593090.divmod.xquotient.1871@top> Message-ID: <41C3CC5A-4F38-440E-9CDC-423A40CAAA5B@twistedmatrix.com> On Sep 4, 2014, at 11:30 PM, Adi Roiban wrote: > Since I can not create wiki pages here are my notes: I went to add you to wiki_editors, and I found that someone had already done so ;). Please go ahead and create a wiki page - perhaps under - so we can update it as consensus develops in this thread? Thanks for collecting all this information Adi, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac at twistedmatrix.com Sun Sep 7 00:05:15 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 07 Sep 2014 06:05:15 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1410069915.49.1225728082364432683@twistedmatrix.com An HTML attachment was scrubbed... URL: From kebin70 at gmail.com Sun Sep 7 01:51:36 2014 From: kebin70 at gmail.com (Kevin Mcintyre) Date: Sun, 7 Sep 2014 00:51:36 -0700 Subject: [Twisted-Python] pb.Copyable knowledge barrier In-Reply-To: References: <23AED729-A718-4887-A755-A9A044B1E898@twistedmatrix.com> Message-ID: Jelly performance factors below expectations. Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. At this point I'm unsure of what jelly actually does well. who's the JellyCon2014 keynote speaker? On Sat, Aug 9, 2014 at 6:58 PM, Glyph wrote: > On Aug 8, 2014, at 9:17 PM, Kevin Mcintyre wrote: > > correction. I was stupidly reprocessing list on remote copy side. Seeing > much better results now ~10K per second. > > > Whew. I don't know exactly how well I would expect this perform, but > those other results looked off by a couple orders of magnitude :-). > > -glyph > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Sep 7 10:43:18 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 7 Sep 2014 09:43:18 -0700 Subject: [Twisted-Python] pb.Copyable knowledge barrier In-Reply-To: References: <23AED729-A718-4887-A755-A9A044B1E898@twistedmatrix.com> Message-ID: <3CDFCECA-735B-4F59-A361-53C9760C529B@twistedmatrix.com> On Sep 7, 2014, at 12:51 AM, Kevin Mcintyre wrote: > Jelly performance factors below expectations. I am not clear on the significance of this statement. > Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. Asking whether your copy is good any more is a PB-level task. Jelly itself is a separate layer which is about getting the right data to the right place, not keeping it updated. > My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. "problematic" is passive voice :-). What are the problems? > At this point I'm unsure of what jelly actually does well. Malheureusement, Jelly's main claim to fame is "it's better than pickle". But with all this renewed interest in PB perhaps we'll get pre-deserialization schema enforcement and type checking, and then it will have some real advantages :). > who's the JellyCon2014 keynote speaker? One day we will have a conference - Tx/Rx, of course - and there will be a talk on this. But first we need several people in the community who wants their contribution to Twisted to be organization and community oriented rather than code ;-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Sun Sep 7 19:26:53 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 03:26:53 +0200 Subject: [Twisted-Python] Python3: should paths be bytes or str? Message-ID: <1469549.Iq1ZMIjpzY@i5> The porting guide says No byte paths in sys.path. I am not sure what this means, I would assume that file paths should always be native strings. Or does it mean that sys.path must only contain ascii bytes? doc for FilePath says On both Python 2 and Python 3, paths can only be bytes. and svn commit 35410 by itamarst changed the doc for some path functions in python/filepath.py from str to bytes but not all of them: fgrep 'type path:' filepath.py @type path: L{str} @type path: L{str} @type path: L{bytes} @type path: L{bytes} @type path: L{bytes} I stumbled upon this while trying to find out how much work it might be to make bin/trial run with python3 admin/run-python3-tests already passes for all twisted.spread related tests but I still need to clean up a lot. after adding an assert to FilePath.__init__, python3 bin/trial ... gives File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run config.parseOptions() File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in parseOptions self.postOptions() File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in postOptions _BasicOptions.postOptions(self) File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in postOptions self['reporter'] = self._loadReporterByName(self['reporter']) File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in _loadReporterByName for p in plugin.getPlugins(itrial.IReporter): File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins allDropins = getCache(package) File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache mod = getModule(module.__name__) File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in getModule return theSystemPath[moduleName] File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in __getitem__ self._findEntryPathString(moduleObject)), File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in _findEntryPathString if _isPackagePath(FilePath(topPackageObj.__file__)): File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in __init__ assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) AssertionError: path must be bytes: '/home/wr/ssdsrc/Twisted/twisted/__init__.py' -- Wolfgang From exarkun at twistedmatrix.com Sun Sep 7 20:14:10 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 02:14:10 -0000 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <1469549.Iq1ZMIjpzY@i5> References: <1469549.Iq1ZMIjpzY@i5> Message-ID: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> On 01:26 am, wolfgang.kde at rohdewald.de wrote: >The porting guide says > >No byte paths in sys.path. What porting guide is that? > >doc for FilePath says > On both Python 2 and Python 3, paths can only be bytes. > > >I stumbled upon this while trying to find out how much work it might be >to make bin/trial run with python3 > >admin/run-python3-tests already passes for all twisted.spread related >tests but I still need to clean up a lot. > >after adding an assert to FilePath.__init__, python3 bin/trial ... >gives > > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in >run > config.parseOptions() > File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in >parseOptions > self.postOptions() > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in >postOptions > _BasicOptions.postOptions(self) > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in >postOptions > self['reporter'] = self._loadReporterByName(self['reporter']) > File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in >_loadReporterByName > for p in plugin.getPlugins(itrial.IReporter): > File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in >getPlugins > allDropins = getCache(package) > File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in >getCache > mod = getModule(module.__name__) > File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in >getModule > return theSystemPath[moduleName] > File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in >__getitem__ > self._findEntryPathString(moduleObject)), > File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in >_findEntryPathString > if _isPackagePath(FilePath(topPackageObj.__file__)): > File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, >in __init__ > assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) >AssertionError: path must be bytes: >'/home/wr/ssdsrc/Twisted/twisted/__init__.py' If paths are being represented using unicode somewhere and you want to use them with FilePath then you have to encode them (or you have to add unicode path support to FilePath and let FilePath encode them). Unfortunately it's not entirely obvious how to make FilePath support unicode paths since not all platforms Twisted supports represent filesystem paths using unicode. The choice python-dev made to bridge this gap was the creation of the "surrogateescape" error handler for the UTC-8 codec. This lets you pretend that any time you need to convert between bytes and unicode the correct codec is UTF-8 (with this special error handler). It's not clear this was a good choice (since the result is unicode strings that may contain garbage which will confuse other software) but it's also not clear it's possible for Twisted to try to make any other choice (at some point Twisted has to interoperate with the path-related APIs in Python itself - `sys.path`, for example). Not sure if that helps you at all. Maybe it outlines the problem a little more clearly, at least. Jean-Paul From wolfgang.kde at rohdewald.de Mon Sep 8 03:19:37 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 11:19:37 +0200 Subject: [Twisted-Python] PEP3131: non-ascii identifiers Message-ID: <2095383.3s724Eqe0K@i5> This does not seem to be supported by Python yet. Should that be enabled at all? If one process with PY3 sends such identifiers to a separate process with PY2, that will fail. I am not sure if that would be a problem, whoever uses this must make sure PY3 is used everywhere. If this should be forbidden, I will add a test to test_pb for this. And of course somebody should document that somewhere. There more PEP3131 is used, the more users will fall into this trap. If this should be enabled (which I think is not difficult, at least for pb): At least the patch below will be needed (only for PY3), maybe it is already sufficient. Given that nativeString and networkString are always used (done that for pb). networkString may then return bytes with the high bit set But since networkString is called in many places I want to ask and make sure that it may really be changed this way. https://twistedmatrix.com/documents/14.0.0/core/specifications/banana.html does not speak against it, so I wonder why networkString has that limitation to 7bit. concrete banana-encoded example, from modified test_pb: (the method name is getSimpleä) test_pb still passes with patched nativeString/networkString (but I only have one test for this so far, test_refcount). b'\x07\x80\x07\x82message\x01\x81\x03\x82foo\x0b\x82getSimple\xc3\xa4\x01\x81\x01\x80\x05\x82tuple\x01\x80\n\x82dictionary' diff --git twisted/python/compat.py twisted/python/compat.py index 6f76c39..6919cf6 100644 --- twisted/python/compat.py +++ twisted/python/compat.py @@ -348,10 +348,9 @@ def nativeString(s): raise TypeError("%r is neither bytes nor unicode" % s) if _PY3: if isinstance(s, bytes): - return s.decode("ascii") + return s.decode("utf-8") else: - # Ensure we're limited to ASCII subset: - s.encode("ascii") + return s else: if isinstance(s, unicode): return s.encode("ascii") @@ -428,7 +427,7 @@ if _PY3: def networkString(s): if not isinstance(s, unicode): raise TypeError("Can only convert text to bytes on Python 3, I got %r" % (s,)) - return s.encode('ascii') + return s.encode('utf-8') def networkChar(integer): """ -- Wolfgang From wolfgang.kde at rohdewald.de Mon Sep 8 03:20:05 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 11:20:05 +0200 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: <1726686.JYd9teDfKp@i5> Am Montag, 8. September 2014, 02:14:10 schrieb exarkun at twistedmatrix.com: > On 01:26 am, wolfgang.kde at rohdewald.de wrote: > >The porting guide says > > > >No byte paths in sys.path. > > What porting guide is that? https://twistedmatrix.com/trac/wiki/Plan/Python3 see the reviewer check list > it's also not clear it's possible for Twisted to try to make any other > choice (at some point Twisted has to interoperate with the path-related > APIs in Python itself - `sys.path`, for example). Am Montag, 8. September 2014, 02:14:10 schrieb exarkun at twistedmatrix.com: > If paths are being represented using unicode somewhere and you want to > use them with FilePath then you have to encode them (or you have to add > unicode path support to FilePath and let FilePath encode them). I always thought module names must be ascii-only but now found PEP3131 So we should do the same for twisted.python.modules as in those other places grepped below. And add that assert isinstance(path, bytes) for PY3 in FilePath. And maybe this should go into the above check list? I have no edit rights in the Wiki. BUT – I will stop trying to port python/modules.py, the usage of the same strings for both module names and file paths is too much interwoven, I do not want to touch that. My feeling is that file names should all be unicode, converting them only where needed. But then I am not an expert about this. Next problem - PEP3131. See separate mail. grep -r __file__ | grep encode web/test/test_webclient.py:serverPEM = FilePath(test.__file__.encode("utf-8")).sibling(b'server.pem') test/ssl_helpers.py:certPath = nativeString(FilePath(__file__.encode("utf-8") test/test_setup.py:if not FilePath(twisted.__file__.encode('utf-8')).sibling(b'topfiles').child(b'setup.py').exists(): python/test/test_deprecate.py: self.assertEqual(FilePath(module.__file__.encode("utf-8")), internet/test/test_gireactor.py: path = FilePath(__file__.encode("utf-8")).sibling( -- Wolfgang From techtonik at gmail.com Mon Sep 8 04:26:58 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 8 Sep 2014 13:26:58 +0300 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: On Mon, Sep 8, 2014 at 5:14 AM, wrote: > On 01:26 am, wolfgang.kde at rohdewald.de wrote: >> >> The porting guide says >> >> No byte paths in sys.path. > > > What porting guide is that? >> >> >> doc for FilePath says >> On both Python 2 and Python 3, paths can only be bytes. >> >> >> I stumbled upon this while trying to find out how much work it might be >> to make bin/trial run with python3 >> >> admin/run-python3-tests already passes for all twisted.spread related >> tests but I still need to clean up a lot. >> >> after adding an assert to FilePath.__init__, python3 bin/trial ... gives >> >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run >> config.parseOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in >> parseOptions >> self.postOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in >> postOptions >> _BasicOptions.postOptions(self) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in >> postOptions >> self['reporter'] = self._loadReporterByName(self['reporter']) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in >> _loadReporterByName >> for p in plugin.getPlugins(itrial.IReporter): >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins >> allDropins = getCache(package) >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache >> mod = getModule(module.__name__) >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in >> getModule >> return theSystemPath[moduleName] >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in >> __getitem__ >> self._findEntryPathString(moduleObject)), >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in >> _findEntryPathString >> if _isPackagePath(FilePath(topPackageObj.__file__)): >> File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in >> __init__ >> assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) >> AssertionError: path must be bytes: >> '/home/wr/ssdsrc/Twisted/twisted/__init__.py' > > > If paths are being represented using unicode somewhere and you want to use > them with FilePath then you have to encode them (or you have to add unicode > path support to FilePath and let FilePath encode them). > > Unfortunately it's not entirely obvious how to make FilePath support unicode > paths since not all platforms Twisted supports represent filesystem paths > using unicode. It really depends on filesystem, not on a platform. Platform just makes sure that you won't shoot it in the foot. So to behave good you need to translate you path knowledge to platform knowledge when you have to make change. In data transformation theory that may mean: [ ] get data about path in native format [ ] detect the source encoding of filesystem [ ] figure out if you can work with native format [ ] python 2 way - just work with bytes [ ] python 3 way - look if native filesystem format is convertible to unicode [ ] if conversion is symmetrical - operate in unicode [ ] if not convertible, alternatives (options, switches) [ ] fail and explain why to user in user actionable manner (don't use ?) [ ] use some symmetrical mapping to unicode and mark path objects as `mapped` so that there is a trace of hacks on filepaths [ ] provide API on objects without ability to use names directly [ ] transform the name to a "safe" valid value loosing the original name and explain the user why and what happened to the old name > The choice python-dev made to bridge this gap was the creation of the > "surrogateescape" error handler for the UTC-8 codec. This lets you pretend > that any time you need to convert between bytes and unicode the correct > codec is UTF-8 (with this special error handler). > > It's not clear this was a good choice (since the result is unicode strings > that may contain garbage which will confuse other software) but it's also > not clear it's possible for Twisted to try to make any other choice (at some > point Twisted has to interoperate with the path-related APIs in Python > itself - `sys.path`, for example). > > Not sure if that helps you at all. Maybe it outlines the problem a little > more clearly, at least. I think that it should be a choice of application maintainers. If they want to create files with dots at the end, Windows allows this, but doesn't support it through standard WinAPI calls, because of FAT. But you can use special path transformation prefix \\?\ to do this on NTFS or on a remote machine. In networking OS plays less role, but the new choice for every platform filesystem is not clear for users. They don't realize where the problem comes from. In the end it all depends on FS first, then on OS API (which can be skipped thanks to direct disk access), then on FS library, then on application. Application should be able to opt-in for handling all possible cases, or just "safe" subset or something middle, so a task of a framework is to just describe the problem well and ensure that people can make their choice. -- anatoly t. From exarkun at twistedmatrix.com Mon Sep 8 04:54:44 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 10:54:44 -0000 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <2095383.3s724Eqe0K@i5> References: <2095383.3s724Eqe0K@i5> Message-ID: <20140908105444.20413.527660985.divmod.xquotient.2139@top> On 09:19 am, wolfgang.kde at rohdewald.de wrote: >This does not seem to be supported by Python yet. > >Should that be enabled at all? >If one process with PY3 sends such identifiers to >a separate process with PY2, that will fail. I am not >sure if that would be a problem, whoever uses this must >make sure PY3 is used everywhere. This is why we will *not* change the PB wire protocol as part of the porting work. The wire protocol will remain the same whether you are using Python 2 or Python 3 to run your program or. This is the point of a protocol, after all. It is to let two programs communicate with each other. >If this should be forbidden, I will add a test to >test_pb for this. And of course somebody should document that >somewhere. There more PEP3131 is used, the more users will >fall into this trap. I'm not exactly sure what you mean here. Using unicode where only bytes are allowed is probably already forbidden throughout PB. > >If this should be enabled (which I think is not difficult, >at least for pb): > >At least the patch below will be needed (only for PY3), >maybe it is already sufficient. Given that nativeString >and networkString are always used (done that for pb). > >networkString may then return bytes with the high bit set Definitely not. >But since networkString is called in many places I want to ask and >make sure that it may really be changed this way. > > >https://twistedmatrix.com/documents/14.0.0/core/specifications/banana.html >does not speak against it, so I wonder why networkString has that >limitation >to 7bit. That is the sole purpose of `networkString`. It is a work-around for the inconvenient fact that Python changed the meaning of the string literal syntax from bytes to unicode. > >concrete banana-encoded example, from modified test_pb: (the method >name is getSimpleä) >test_pb still passes with patched nativeString/networkString (but I >only have one test for this so far, test_refcount). > >b'\x07\x80\x07\x82message\x01\x81\x03\x82foo\x0b\x82getSimple\xc3\xa4\x01\x81\x01\x80\x05\x82tuple\x01\x80\n\x82dictionary' > > >diff --git twisted/python/compat.py twisted/python/compat.py >index 6f76c39..6919cf6 100644 >--- twisted/python/compat.py >+++ twisted/python/compat.py >@@ -348,10 +348,9 @@ def nativeString(s): > raise TypeError("%r is neither bytes nor unicode" % s) > if _PY3: > if isinstance(s, bytes): >- return s.decode("ascii") >+ return s.decode("utf-8") > else: >- # Ensure we're limited to ASCII subset: >- s.encode("ascii") >+ return s > else: > if isinstance(s, unicode): > return s.encode("ascii") >@@ -428,7 +427,7 @@ if _PY3: > def networkString(s): > if not isinstance(s, unicode): > raise TypeError("Can only convert text to bytes on Python >3, I got %r" % (s,)) >- return s.encode('ascii') >+ return s.encode('utf-8') > > def networkChar(integer): > """ This change definitely won't be acceptable. It completely removes the feature `networkString` exists to provide: verifying that strings that might be either unicode or bytes can still be implicitly combined into bytes. Can you point out the specific places where you think PB needs to start using UTF-8 instead of ASCII? Those are the places that need to be fixed, not the underlying porting helpers they happen to use. Jean-Paul From wolfgang.kde at rohdewald.de Mon Sep 8 05:44:30 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 13:44:30 +0200 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <20140908105444.20413.527660985.divmod.xquotient.2139@top> References: <2095383.3s724Eqe0K@i5> <20140908105444.20413.527660985.divmod.xquotient.2139@top> Message-ID: <1572241.mjvUuZoz8Z@i5> Am Montag, 8. September 2014, 10:54:44 schrieb exarkun at twistedmatrix.com: > Can you point out the specific places where you think PB needs to start > using UTF-8 instead of ASCII? Those are the places that need to be > fixed, not the underlying porting helpers they happen to use. My question is whether PEP3131 should be supported or not. After all, it is an integral part of Python3. I have no opinion about this, I just want to point it out as a potential problem. So you say it should not be supported, and I have no problem with that. Just to be clear about it: In Python3 you can legally write def ÜnicödeMethod€Name(self): or see https://mail.python.org/pipermail/python-3000/2007-June/008172.html for a nice Russian example. but PB cannot transfer those identifiers. Which is not what the user would expect. This should be clearly communicated somewhere in the docs, IMHO. -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 8 06:04:46 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 12:04:46 -0000 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <1572241.mjvUuZoz8Z@i5> References: <2095383.3s724Eqe0K@i5> <20140908105444.20413.527660985.divmod.xquotient.2139@top> <1572241.mjvUuZoz8Z@i5> Message-ID: <20140908120446.20413.1993708093.divmod.xquotient.2146@top> On 11:44 am, wolfgang.kde at rohdewald.de wrote: >Am Montag, 8. September 2014, 10:54:44 schrieb >exarkun at twistedmatrix.com: >>Can you point out the specific places where you think PB needs to >>start >>using UTF-8 instead of ASCII? Those are the places that need to be >>fixed, not the underlying porting helpers they happen to use. > >My question is whether PEP3131 should be supported or not. After all, >it is an integral part of Python3. I have no opinion about this, >I just want to point it out as a potential problem. > >So you say it should not be supported, and I have no problem with that. > >Just to be clear about it: > >In Python3 you can legally write > >def ÜnicödeMethod€Name(self): > >or see >https://mail.python.org/pipermail/python-3000/2007-June/008172.html >for a nice Russian example. > >but PB cannot transfer those identifiers. Which is not what the >user would expect. PB supports unicode perfectly well and has for many years. This is why I asked which specific part of PB has a problem. I don't see why you think unicode is a problem here. This doesn't mean there isn't a problem - after all, you've been looking at the code and trying to make it work on Python 3 and I haven't - but you haven't made it clear what that problem is. A failing unit test or a minimal example () would communicate this most clearly, but perhaps you can just mention a specific API and give an incomplete example of how it will fail when it runs up against the changes defined by PEP 3131. Jean-Paul > >This should be clearly communicated somewhere in the docs, IMHO. > >-- >Wolfgang From wolfgang.kde at rohdewald.de Mon Sep 8 07:07:19 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 08 Sep 2014 15:07:19 +0200 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <20140908120446.20413.1993708093.divmod.xquotient.2146@top> References: <2095383.3s724Eqe0K@i5> <1572241.mjvUuZoz8Z@i5> <20140908120446.20413.1993708093.divmod.xquotient.2146@top> Message-ID: <1573940.73e26bT3Za@i5> Am Montag, 8. September 2014, 12:04:46 schrieb exarkun at twistedmatrix.com: > PB supports unicode perfectly well and has for many years. This is why > I asked which specific part of PB has a problem. PB transfers names of methods and classes as bytes, not as unicode. Which is logical since PY2 does not support unicode identifiers, and bytes is already a native PY2 string. Unicode is only used for content. It not yet always clear to me what is content and what is a formal string like method names or the *_atom strings which must be bytes, this needs more testing. I guess I should patch banana.py such that it dumps all it encodes or decodes into one file, so I can compare output from PY2/PY3 tests. I was assuming that suddenly transferring method names as unicode would really be a break of wire protocol stability, or do you think otherwise? If you think this is acceptable, I will check if the existing twisted code can handle getting those as unicode without source code changes. Not sure. Just tested this with Python2.6, and I am surprised that it works: >>> getattr(A,u'x') Supporting PEP3131 would only introduce a backward-incompatibility. Of course you are right that this is not part of porting. Right now I have a long list of small unsorted git commits, I will have to do a lot of reshuffling and cleaning before I will ask you how to get it into the official codebase. Not all of those commits are strictly porting, some just clean the code, making the porting commits simpler. > A failing unit test or a minimal example () would > communicate this most clearly, but perhaps you can just mention a > specific API and give an incomplete example of how it will fail when it > runs up against the changes defined by PEP 3131. see my first mail in this thread: take test_pb.py, rename getSimple to getSimpleä, run the test. -- Wolfgang From exarkun at twistedmatrix.com Mon Sep 8 08:26:19 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 08 Sep 2014 14:26:19 -0000 Subject: [Twisted-Python] PEP3131: non-ascii identifiers In-Reply-To: <1573940.73e26bT3Za@i5> References: <2095383.3s724Eqe0K@i5> <1572241.mjvUuZoz8Z@i5> <20140908120446.20413.1993708093.divmod.xquotient.2146@top> <1573940.73e26bT3Za@i5> Message-ID: <20140908142619.20413.1258694452.divmod.xquotient.2237@top> On 01:07 pm, wolfgang.kde at rohdewald.de wrote: >Am Montag, 8. September 2014, 12:04:46 schrieb >exarkun at twistedmatrix.com: >>PB supports unicode perfectly well and has for many years. This is >>why >>I asked which specific part of PB has a problem. > >PB transfers names of methods and classes as bytes, not as unicode. PB actually transfers *everything* as bytes. ;) Bytes are the only thing you can send over a socket. If you have anything else - integers, unicode, whatever - you have to encode them as bytes first. As far as I know, the PB protocol is not specified apart from the implementation in Twisted (and this being the only implementation (I am intentionally disregarding TwistedJava) it must serve as the specification, I think). What you meant above, I think, is that PB represents method names as bytes at the banana layer. That is, when you want to call a remote method, you indicate its name by supplying bytes to the banana encoding layer - not unicode (which is good because banana doesn't actually support unicode at all, that's a jelly feature). This does indeed mean we don't simply want to start sending unicode to refer to methods by name - because we can't! At least, not unless we extend banana to support a new type which we probably don't want to do - that would be another incompatible protocol change and so not allowed (since it could break interoperability between different implementations of PB). So, it is necessary to continue to represent method names using bytes. This is fairly easily done. On Python 3, encode any unicode strings which represent method names (using a well-known encoding, probably UTF-8) when making the call and decode them in the same way when dispatching those calls. This can almost be done at the application level: # Some Python 3 code def remote_ä(self): pass ref.callRemote(u"ä".encode("utf-8")) except that Python 3 has actually changed to enforce the type of the second argument of getattr - if it is not a unicode string then a TypeError is raised - so it's not possible to make the decoding step happen (which one might otherwise have done using `__getattr__` or by adding the encoded name of the method to the class dictionary). So if it is going to be supported in Twisted's PB API then that support probably needs to be in Twisted's PB implementation. The same general idea applies, though. Just move the encoding into the implementation of `callRemote`: def callRemote(self, _name, ...): ... _name.encode("utf-8") ... And add a corresponding decode to the other side (probably in `_recvMessage`). This would make the Python 3 PB API be "method names are unicode strings" which makes sense considering the decisions that were made for Python 3. Note that it does not change the wire protocol - method names are still bytes at the banana level. Or does it? These bytes were previously always an ASCII subset. Is expanding out of the ASCII range an incompatible change? What could this break? Let's say you have Python 2 talking to Python 2. It's already possible to construct a method call like this: class Foo(Referenceable): def remote_foo(self): pass setattr(Foo, u"remote_fooä".encode("utf-8"), Foo.remote_foo) ... ref.callRemote(u"remote_fooä".encode("utf-8")) This actually works. Python 2 doesn't much care about how you name your class attributes. PB doesn't care that the high bit is set in one or more of the bytes in the method name. It all just works. So let's say you have Python 2 talking to Python 3 instead. In Python 3, you can't do that setattr() call (the language and runtime disallow it). But you can have `def remote_fooä(self)` instead. If PB on Python 3 decodes the method name before dispatching it (using UTF-8) then again things work. And if you reverse the situation and PB on Python 3 encodes the method name before sending it, then Python 2 is still happen because it can operate on that UTF-8 encoded byte string. Finally, if Python 3 talks to Python 3 then it also works because the sending side encodes and the receiving side decodes. So we get to make a judgement call here, I think. Without a specification there's no objectively correct answer. So, because the current implementation is actually perfectly compatible with non-ASCII bytes - even though the intent is clearly that you would never have those - combined with the point that I made above, that there are no other PB implementations, I suspect it's fine to expand beyond ASCII here because it won't actually break any real world programs. The only case that I can think of that actually would be a problem is the case where someone is already sending non-ASCII, non-UTF-8 method names around. These might decode wrong or might fail to decode at all. I don't think this is likely enough to worry about - but maybe someone who is doing this will speak up and prove me wrong. ;) And to repeat myself a bit, none of this should change the Python 2 PB API. It should continue accepting bytes - because that's what it always accepted. Separately, we could introduce a new feature to support unicode on Python 2. This would be done in the usual way for introducing new features into any Twisted APIs (and there aren't reasonable backwards compatibility considerations here as far as I can tell). The point would be to make it a little more convenient for Python 2 applications to interact with other PB applications that have decided to use unicode method names. Lastly, on another topic, I am subscribed to the mailing list - you don't have to cc me on your replies. Jean-Paul > >Which is logical since PY2 does not support unicode identifiers, >and bytes is already a native PY2 string. Unicode is only used >for content. It not yet always clear to me what is content and >what is a formal string like method names or the *_atom strings >which must be bytes, this needs more testing. > >I guess I should patch banana.py such that it dumps all it encodes >or decodes into one file, so I can compare output from PY2/PY3 tests. > >I was assuming that suddenly transferring method names as unicode >would really be a break of wire protocol stability, or do you >think otherwise? If you think this is acceptable, I will check >if the existing twisted code can handle getting those as unicode >without source code changes. Not sure. Just tested this with >Python2.6, and I am surprised that it works: >>>>getattr(A,u'x') > > > >Supporting PEP3131 would only introduce a backward-incompatibility. > >Of course you are right that this is not part of porting. > >Right now I have a long list of small unsorted git commits, I will >have to do a lot of reshuffling and cleaning before I will ask >you how to get it into the official codebase. >Not all of those commits are strictly porting, some just clean >the code, making the porting commits simpler. >>A failing unit test or a minimal example () would >>communicate this most clearly, but perhaps you can just mention a >>specific API and give an incomplete example of how it will fail when >>it >>runs up against the changes defined by PEP 3131. > >see my first mail in this thread: take test_pb.py, rename getSimple >to getSimpleä, run the test. > >-- >Wolfgang > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python at twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From kebin70 at gmail.com Mon Sep 8 15:36:53 2014 From: kebin70 at gmail.com (Kevin Mcintyre) Date: Mon, 8 Sep 2014 14:36:53 -0700 Subject: [Twisted-Python] pb.Copyable knowledge barrier In-Reply-To: <3CDFCECA-735B-4F59-A361-53C9760C529B@twistedmatrix.com> References: <23AED729-A718-4887-A755-A9A044B1E898@twistedmatrix.com> <3CDFCECA-735B-4F59-A361-53C9760C529B@twistedmatrix.com> Message-ID: Jelly performance factors below expectations. I am not clear on the significance of this statement. Not significant, just an observation. I've made this statement before, backed-off after finding errors on my code, but after fixing I see performance across the bus that still seem slow. "Seem slow" lacks any scientific backing. I would be curious to hear use cases for PB/Jelly that went beyond the docs. Can we say Copyable is the lowest order jelly? The notion that a copy holder can't ask "is my copy good anymore?" makes it so. Essentially root says, I'd prefer not to repeat unit of work nor keep track of the resulting copies, here have the original or resulting copy. Asking whether your copy is good any more is a PB-level task. Jelly itself is a separate layer which is about getting the right data to the right place, not keeping it updated. Yes PB level, I guess I'm looking for a convention where 1 does not exist. In the renewed interest the comments have tended to overlook copyable, or see copyable as being flushed out. My main issue is a copy-holder calling for a copy to determine is the copy is good anymore. I know, see cacheable but it's problematic. "problematic" is passive voice :-). What are the problems? Only parroting what I've read on cacheable, haven't delved into yet. At this point I'm unsure of what jelly actually does well. Malheureusement, Jelly's main claim to fame is "it's better than pickle". But with all this renewed interest in PB perhaps we'll get pre-deserialization schema enforcement and type checking, and then it will have some real advantages :). Agreed. This is what I looked to Jelly/PB for, having found so much in twisted that simplifies. who's the JellyCon2014 keynote speaker? *One* day we will have a conference - Tx/Rx, of course - and there will be a talk on this. But first we need several people in the community who wants their contribution to Twisted to be organization and community oriented rather than code ;-). On Sun, Sep 7, 2014 at 9:43 AM, Glyph wrote: > > On Sep 7, 2014, at 12:51 AM, Kevin Mcintyre wrote: > > Jelly performance factors below expectations. > > > I am not clear on the significance of this statement. > > Can we say Copyable is the lowest order jelly? The notion that a copy > holder can't ask "is my copy good anymore?" makes it so. Essentially root > says, I'd prefer not to repeat unit of work nor keep track of the resulting > copies, here have the original or resulting copy. > > > Asking whether your copy is good any more is a PB-level task. Jelly > itself is a separate layer which is about getting the right data to the > right place, not keeping it updated. > > My main issue is a copy-holder calling for a copy to determine is the copy > is good anymore. I know, see cacheable but it's problematic. > > > "problematic" is passive voice :-). What are the problems? > > At this point I'm unsure of what jelly actually does well. > > > Malheureusement, Jelly's main claim to fame is "it's better than pickle". > But with all this renewed interest in PB perhaps we'll get > pre-deserialization schema enforcement and type checking, and then it will > have some real advantages :). > > who's the JellyCon2014 keynote speaker? > > > *One* day we will have a conference - Tx/Rx, of course - and there will > be a talk on this. But first we need several people in the community who > wants their contribution to Twisted to be organization and community > oriented rather than code ;-). > > -glyph > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Sep 9 00:01:50 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 8 Sep 2014 23:01:50 -0700 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <20140908021410.20413.2139896976.divmod.xquotient.2126@top> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: On Sep 7, 2014, at 7:14 PM, exarkun at twistedmatrix.com wrote: > On 01:26 am, wolfgang.kde at rohdewald.de wrote: >> The porting guide says >> >> No byte paths in sys.path. > > What porting guide is that? >> >> doc for FilePath says >> On both Python 2 and Python 3, paths can only be bytes. >> >> >> I stumbled upon this while trying to find out how much work it might be >> to make bin/trial run with python3 >> >> admin/run-python3-tests already passes for all twisted.spread related >> tests but I still need to clean up a lot. >> >> after adding an assert to FilePath.__init__, python3 bin/trial ... gives >> >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 601, in run >> config.parseOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/python/usage.py", line 277, in parseOptions >> self.postOptions() >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 472, in postOptions >> _BasicOptions.postOptions(self) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 382, in postOptions >> self['reporter'] = self._loadReporterByName(self['reporter']) >> File "/home/wr/ssdsrc/Twisted/twisted/scripts/trial.py", line 369, in _loadReporterByName >> for p in plugin.getPlugins(itrial.IReporter): >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 209, in getPlugins >> allDropins = getCache(package) >> File "/home/wr/ssdsrc/Twisted/twisted/plugin.py", line 134, in getCache >> mod = getModule(module.__name__) >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 781, in getModule >> return theSystemPath[moduleName] >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 702, in __getitem__ >> self._findEntryPathString(moduleObject)), >> File "/home/wr/ssdsrc/Twisted/twisted/python/modules.py", line 627, in _findEntryPathString >> if _isPackagePath(FilePath(topPackageObj.__file__)): >> File "/home/wr/ssdsrc/Twisted/twisted/python/filepath.py", line 664, in __init__ >> assert isinstance(path, bytes), 'path must be bytes: %r' % (path,) >> AssertionError: path must be bytes: '/home/wr/ssdsrc/Twisted/twisted/__init__.py' > > If paths are being represented using unicode somewhere and you want to use them with FilePath then you have to encode them (or you have to add unicode path support to FilePath and let FilePath encode them). > > Unfortunately it's not entirely obvious how to make FilePath support unicode paths since not all platforms Twisted supports represent filesystem paths using unicode. > > The choice python-dev made to bridge this gap was the creation of the "surrogateescape" error handler for the UTC-8 codec. This lets you pretend that any time you need to convert between bytes and unicode the correct codec is UTF-8 (with this special error handler). > > It's not clear this was a good choice (since the result is unicode strings that may contain garbage which will confuse other software) but it's also not clear it's possible for Twisted to try to make any other choice (at some point Twisted has to interoperate with the path-related APIs in Python itself - `sys.path`, for example). > > Not sure if that helps you at all. Maybe it outlines the problem a little more clearly, at least. The problem with making FilePath support unicode is that we want to provide an interface that applications can rely upon, specified in terms of specific types (bytes or text) so that when you get an IFilePath you know what you can do with it. As it is currently implemented, FilePath exposes its internal representation fairly directly, most notably as the ‘.path’ attribute, but also in the return-type of methods like "basename" and "segmentsFrom". FilePath doesn't exactly "support" unicode, in that it's specifically documented not to, but it's sort of hard to tell, since you can instantiate one with a unicode string in both python 2 and python 3, and get (apparently) correct results out of it for some methods. However, methods that need a string constant as part of their implementation, like siblingExtensionSearch and globChildren, will break unceremoniously when presented with unicode. Another decision that python-dev made to bridge the gap was to randomly allow different string types be passed to platform APIs, like this: >>> import os >>> os.listdir(u".") ['a', 'b', 'c'] >>> os.listdir(b".") [b'a', b'b', b'c'] >>> os.path.basename(b".") b'.' >>> os.path.basename(".") '.' This implies a parallel structure might be possible for FilePath: if you pass its constructor bytes, you get a BytesFilePath; if you pass it text, you get a TextFilePath. You can't mix the two, and once you've chosen a path you can't choose a different one. IFilePath could then document that all of its existing methods have the return type of "whatever got passed to __init__" (which is what the current implementation does about 2/3 of the time anyway on py3, and about 9/10 of the time on py2; we would just be making it work intentionally, all the way). But, it would then be possible to give BytesFilePath a "asText()" method and vice versa "asBytes()" - since it's the filesystem, metadata about encodings exists outside your program and you would not need to guess at encodings, you'd simply indicate what return value you'd like from methods like .basename() et. al. The more I think about this, the more I like it - it's a bit of annoying and subtle implementation work, but I think it would supply the behavior that most people want, remain compatible with most of the existing unspecified behavior, and it would address clean text/bytes separation without having a giant deprecation cycle and inventing a new interface. It's also the sort of implementation work which, after some discussion and consideration, we could be reasonably sure is *correct* rather than guessing at things. Thoughts? -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Sep 9 15:27:35 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 9 Sep 2014 14:27:35 -0700 Subject: [Twisted-Python] a brief note about buildbots Message-ID: Hi Twistosophists, Some of you doing reviews (you ARE all doing code reviews, right!?!?) might have noticed that some of the boxes on the left side of of the supported builds page were dark grey, meaning that they were offline. I spent some time over the last couple of days doing some administrative housekeeping and rebooting computers, and all of our supported builders are now back online! This makes me feel much better about the possibility of doing a release again someday :-). Speaking of colors of boxes on that page, though, we should probably try to turn some of them from red to green... -glyph P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're running many these tests on, and (B) my time to do this sort of maintenance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Tue Sep 9 16:56:18 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 10 Sep 2014 01:56:18 +0300 Subject: [Twisted-Python] a brief note about buildbots In-Reply-To: References: Message-ID: On Wed, Sep 10, 2014 at 12:27 AM, Glyph Lefkowitz wrote: > Hi Twistosophists, > > Some of you doing reviews (you ARE all doing code reviews, right!?!?) might > have noticed that some of the boxes on the left side of of the supported > builds page were dark grey, meaning that they were offline. > > I spent some time over the last couple of days doing some administrative > housekeeping and rebooting computers, and all of our supported builders are > now back online! This makes me feel much better about the possibility of > doing a release again someday :-). > > Speaking of colors of boxes on that page, though, we should probably try to > turn some of them from red to green... > > -glyph > > P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're > running many these tests on, and (B) my time to do this sort of maintenance. Long Live Glyph! (Rackspace probably receives their credits in the market share) I presume you're using Firefox for all browsing needs. Am I right? =) In Chrome I've noticed an empty box with missing image at the highscore page leading to https://sm7.sitemeter.com/meter.asp?site=sm7twistedmatrix And when I open linked buildbot page at: https://buildbot.twistedmatrix.com/boxes-supported?branch=trunk&num_builds=4 Chrome says that "Attackers might be trying to steal your information from buildbot.twistedmatrix.com (for example, passwords, messages, or credit cards)." Not your, of course, but mine, so I am double worried about that invalid security certificate issued to www.twistedmatrix.com and deployed on buildbot.t.c BTW, how do you count tests? Is this config available somewhere? -- anatoly t. From wolfgang.kde at rohdewald.de Tue Sep 9 20:40:00 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 10 Sep 2014 04:40 +0200 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> Message-ID: <5928596.KCb0BbjCGC@i5> Am Montag, 8. September 2014, 23:01:50 schrieb Glyph: > This implies a parallel structure might be possible for FilePath: > if you pass its constructor bytes, you get a BytesFilePath; > if you pass it text, you get a TextFilePath. You can't mix the two, > and once you've chosen a path you can't choose a different one. this sounds good. After the port of pb to PY3 is done, I might have a look at it, it probably should be done before trying to port modules.py and trial to PY3. -- Wolfgang From wolfgang.kde at rohdewald.de Tue Sep 9 20:50:35 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Wed, 10 Sep 2014 04:50:35 +0200 Subject: [Twisted-Python] jelly and instances Message-ID: <4250196.vmN52vaFvk@i5> doc says: Instance: s is an instance of UserString.UserString, with a __dict__ {'data': 'hello'}:: [\"UserString.UserString\", ['dictionary', ['data', 'hello']]] There is no test for instances. To be precise, no test ever enters jelly._unjelly_instance() I also cannot find any place in jelly.py where an Instance would be jellied. There is instance_atom = 'instance' but instance_atom is not used anywhere. Is this outdated code? If not, could you please give me a code example using this? -- Wolfgang From glyph at twistedmatrix.com Wed Sep 10 01:18:09 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 10 Sep 2014 00:18:09 -0700 Subject: [Twisted-Python] jelly and instances In-Reply-To: <4250196.vmN52vaFvk@i5> References: <4250196.vmN52vaFvk@i5> Message-ID: On Sep 9, 2014, at 7:50 PM, Wolfgang Rohdewald wrote: > doc says: > > Instance: s is an instance of UserString.UserString, with a __dict__ > {'data': 'hello'}:: > [\"UserString.UserString\", ['dictionary', ['data', 'hello']]] > > There is no test for instances. To be precise, no test ever enters > jelly._unjelly_instance() > > I also cannot find any place in jelly.py where an Instance would > be jellied. There is > > instance_atom = 'instance' > > but instance_atom is not used anywhere. > > Is this outdated code? If not, could you please give me a code > example using this? PB was developed during a period of far less rigorous development practices for Twisted. Particularly, we didn't do test-driven development. It's quite possible that this code was never used. Most likely I looked at the Python type system when originally developing PB and thought that I'd want to jelly types.InstanceType, then thought the better of it because types should be types and InstanceType was a wart (which python-dev apparently agreed with since it is gone from the new-style object model). I imagine the thing to do with this is simply deprecate->remove, if it's already unused. It's certainly not something we'd want to resurrect. Thanks, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Sep 10 01:24:54 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 10 Sep 2014 00:24:54 -0700 Subject: [Twisted-Python] a brief note about buildbots In-Reply-To: References: Message-ID: <96E5EA0F-40AE-4F7C-B7FE-3F25F25FA07A@twistedmatrix.com> On Sep 9, 2014, at 3:56 PM, anatoly techtonik wrote: > On Wed, Sep 10, 2014 at 12:27 AM, Glyph Lefkowitz > wrote: >> Hi Twistosophists, >> >> Some of you doing reviews (you ARE all doing code reviews, right!?!?) might >> have noticed that some of the boxes on the left side of of the supported >> builds page were dark grey, meaning that they were offline. >> >> I spent some time over the last couple of days doing some administrative >> housekeeping and rebooting computers, and all of our supported builders are >> now back online! This makes me feel much better about the possibility of >> doing a release again someday :-). >> >> Speaking of colors of boxes on that page, though, we should probably try to >> turn some of them from red to green... >> >> -glyph >> >> P.S.: Thanks to Rackspace for providing both (A) the cloud resources we're >> running many these tests on, and (B) my time to do this sort of maintenance. > > Long Live Glyph! > (Rackspace probably receives their credits in the market share) Thanks for saying so :-). > I presume you're using Firefox for all browsing needs. Am I right? =) In Chrome > I've noticed an empty box with missing image at the highscore page leading to > https://sm7.sitemeter.com/meter.asp?site=sm7twistedmatrix Nope, I have just grown blind to that box. We should really just give up on sitemeter and delete it everywhere. They were injecting some nasty stuff into our pages on some other areas of the site and it has been removed. > And when I open linked buildbot page at: > https://buildbot.twistedmatrix.com/boxes-supported?branch=trunk&num_builds=4 > Chrome says that "Attackers might be trying to steal your information from > buildbot.twistedmatrix.com (for example, passwords, messages, or credit cards)." > Not your, of course, but mine, so I am double worried about that > invalid security > certificate issued to www.twistedmatrix.com and deployed on buildbot.t.c buildbot.t.c and www.t.c are the same host and, upon closer inspection, you will see they're using the same certificate. The problem is that it's not valid for buildbot.t.c because of dumb rules. I rolled a cert for buildbot some time ago, but it was never installed. In the meanwhile I have written (also thanks to Rackspace!), which makes setting up SNI sites _much_ easier. If someone wanted to submit a pull request to and and the other attendant files in t-web, I'm sure I could deploy it. Or I will (eventually) get around to it. > BTW, how do you count tests? Is this config available somewhere? Not sure. Hopefully someone more familiar with the buildbot configuration will opine :-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Sep 10 01:26:13 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 10 Sep 2014 00:26:13 -0700 Subject: [Twisted-Python] Python3: should paths be bytes or str? In-Reply-To: <5928596.KCb0BbjCGC@i5> References: <1469549.Iq1ZMIjpzY@i5> <20140908021410.20413.2139896976.divmod.xquotient.2126@top> <5928596.KCb0BbjCGC@i5> Message-ID: <937D5E16-5F36-4BFE-A6D9-7EC5AF8CBD97@twistedmatrix.com> On Sep 9, 2014, at 7:40 PM, Wolfgang Rohdewald wrote: > Am Montag, 8. September 2014, 23:01:50 schrieb Glyph: >> This implies a parallel structure might be possible for FilePath: >> if you pass its constructor bytes, you get a BytesFilePath; >> if you pass it text, you get a TextFilePath. You can't mix the two, >> and once you've chosen a path you can't choose a different one. > > this sounds good. After the port of pb to PY3 is done, I might have > a look at it, it probably should be done before trying to port > modules.py and trial to PY3. Thank you very much for putting all this work in. I encourage you to also get involved with doing code reviews, so that the >50 tickets currently on won't distract other reviewers from ever getting to your contributions! :) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From reubenhwk at gmail.com Thu Sep 11 01:37:07 2014 From: reubenhwk at gmail.com (Reuben Hawkins) Date: Thu, 11 Sep 2014 00:37:07 -0700 Subject: [Twisted-Python] epoll 2.4 kernel Message-ID: Not sure if this is the right mailing list for this.... epoll should *not* be used on 2.4 kernels... I have 2.4 kernel running on a 32-bit x86 machine and another 2.4 kernel running on 64-bit x64 and I'm having an issue with twisted (via buildbot). I can import epoll from select on the 64-bit machine (it *should* fail but doesn't) and this later causes twisted to fail... >>> from select import epoll >>> epoll(1) Traceback (most recent call last): File "", line 1, in IOError: [Errno 38] Function not implemented On the 32-bit machine, however, importing epoll is properly failing... >>> from select import epoll Traceback (most recent call last): File "", line 1, in ImportError: cannot import name epoll >>> twisted/internet/default.py is hitting this same problem. It's incorrectly picking epoll when it shouldn’t because import epoll is working when it shouldn't. I 'fixed' the problem on my machine by adding a check method in epollreactor.py which does this... def check(): try: epoll(1) except IOError: return False return True and I import and call the check method in default.py. An alternate fix can be... in twisted/internet/default.py try: if platform.isLinux(): try: from select import epoll epoll(1) from twisted.internet.epollreactor import install except (IOError, ImportError): from twisted.internet.pollreactor import install Any insight would be good. I had to compile my own Python 2.7.8 on these old computers...maybe python is incorrectly detecting epoll in the build process? Thanks in advance, Reuben From glyph at twistedmatrix.com Thu Sep 11 03:42:26 2014 From: glyph at twistedmatrix.com (Glyph) Date: Thu, 11 Sep 2014 02:42:26 -0700 Subject: [Twisted-Python] epoll 2.4 kernel In-Reply-To: References: Message-ID: <98068ED4-68BB-44E2-A6FE-34466DF6ED03@twistedmatrix.com> On Sep 11, 2014, at 12:37 AM, Reuben Hawkins wrote: > Not sure if this is the right mailing list for this.... > > epoll should *not* be used on 2.4 kernels... I'm sorry to say that Twisted does not currently support 2.4 kernels. Our set of supported platforms is defined by the set of buildbots that we maintain: As far as I know none of those platforms use a 2.4 kernel. If you wanted to contribute or fund some resources for a 2.4 buildbot, we could consider it, but... it seems like this would be a waste of time. Even my ancient WiFi routers which I can never upgrade because of proprietary firmware drivers still have a 2.6 kernel at least. As I understand it, 2.4 was end-of-lifed 8 years ago, and even at that point it was pretty well into its decline. Why are you using it? Is "just use a really old version of Twisted" an acceptable answer for you? I ask because I am wondering if supporting it is something it would be reasonable to entertain. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at itamarst.org Thu Sep 11 04:14:10 2014 From: itamar at itamarst.org (Itamar Turner-Trauring) Date: Thu, 11 Sep 2014 06:14:10 -0400 Subject: [Twisted-Python] epoll 2.4 kernel In-Reply-To: References: Message-ID: <541175F2.4060902@itamarst.org> Easiest thing for you to do is install a poll reactor as the custom reactor: https://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html#core-howto-choosing-reactor-poll From sank.daniel at gmail.com Fri Sep 12 00:20:17 2014 From: sank.daniel at gmail.com (Daniel Sank) Date: Thu, 11 Sep 2014 23:20:17 -0700 Subject: [Twisted-Python] workflow Message-ID: I would greatly benefit from some direction regarding workflow. In particular, how do you guys deal with having an installed version of twisted, and a version (versions?) with which to tinker? Currently, I have twisted installed in the normal end user way, and I have a clone of my fork of the git repo. I mess around in the git repo clone, but I don't know how to e.g. run the tests or run applications using the git clone. glyph explained this to me on irc a while back. Are the instructions from that recorded somehwere? Why am I asking? Because I want to do reviews and I'm not doing it because I don't know how to set up a sane environment. -- Daniel Sank -------------- next part -------------- An HTML attachment was scrubbed... URL: From hs at ox.cx Fri Sep 12 03:29:04 2014 From: hs at ox.cx (Hynek Schlawack) Date: Fri, 12 Sep 2014 11:29:04 +0200 Subject: [Twisted-Python] workflow In-Reply-To: References: Message-ID: On 12 Sep 2014, at 8:20, Daniel Sank wrote: > I would greatly benefit from some direction regarding workflow. In > particular, how do you guys deal with having an installed version of > twisted, and a version (versions?) with which to tinker? > > Currently, I have twisted installed in the normal end user way, and I > have > a clone of my fork of the git repo. I mess around in the git repo > clone, > but I don't know how to e.g. run the tests or run applications using > the > git clone. > > glyph explained this to me on irc a while back. Are the instructions > from > that recorded somehwere? > > Why am I asking? Because I want to do reviews and I'm not doing it > because > I don't know how to set up a sane environment. Like any other project: I have a virtualenv where I install Twisted’s git checkout with “pip -e .” plus all the dependencies I need. I also have a tox.ini for common permutations of deps and Python 3, but unfortunately the tests tend to fail with weird heisenerrors at least on OS X so I can’t rely on that. From netwotkstudent at yahoo.com Fri Sep 12 15:26:37 2014 From: netwotkstudent at yahoo.com (User User) Date: Fri, 12 Sep 2014 14:26:37 -0700 Subject: [Twisted-Python] How to send data from another reactor instance Message-ID: <1410557197.78081.YahooMailNeo@web160406.mail.bf1.yahoo.com> Hi, I have the codes below and I would like to know how is it possible to send out data which came to "a" instance from "b" i mean data to port 2000 go out from port "2001" !? class fact(DatagramProtocol): def datagramReceived(self, datagram, addr): self.transport.write( datagram, some_where ) a = reactor.listenUDP(2001, fact()) b = reactor.listenUDP(2002, fact()) -------------- next part -------------- An HTML attachment was scrubbed... URL: From werner at thieprojects.ch Fri Sep 12 18:21:53 2014 From: werner at thieprojects.ch (Werner Thie) Date: Fri, 12 Sep 2014 14:21:53 -1000 Subject: [Twisted-Python] How to send data from another reactor instance In-Reply-To: <1410557197.78081.YahooMailNeo@web160406.mail.bf1.yahoo.com> References: <1410557197.78081.YahooMailNeo@web160406.mail.bf1.yahoo.com> Message-ID: <54138E21.70709@thieprojects.ch> Hi consider this for TCP, adapt to UDP http://stackoverflow.com/questions/15640640/python-twisted-man-in-the-middle-implementation/15645169#15645169 Werner On 9/12/14 11:26 AM, User User wrote: > Hi, > I have the codes below and I would like to know how is it possible to > send out data which came to "a" instance from "b" i mean data to port > 2000 go out from port "2001" !? Is there anybody listening on port 2000? > > class fact(DatagramProtocol): > def datagramReceived(self, datagram, addr): > self.transport.write( datagram, some_where ) > > a = reactor.listenUDP(2001, fact()) > b = reactor.listenUDP(2002, fact()) > > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > From trac at twistedmatrix.com Sun Sep 14 00:05:15 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 14 Sep 2014 06:05:15 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1410674715.71.5935457979269114748@twistedmatrix.com An HTML attachment was scrubbed... URL: From exarkun at twistedmatrix.com Sun Sep 14 18:09:17 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 15 Sep 2014 00:09:17 -0000 Subject: [Twisted-Python] Python 3.3 buildslaves Message-ID: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Hello, I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot configuration/availability issues. Who can look at why one of the Python 3.3 slaves is offline and why the two slaves are configured differently? Thanks, Jean-Paul From adi at roiban.ro Mon Sep 15 00:07:12 2014 From: adi at roiban.ro (Adi Roiban) Date: Mon, 15 Sep 2014 07:07:12 +0100 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: <20140915000917.31272.1552860934.divmod.xquotient.92@top> References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 September 2014 01:09, wrote: > Hello, > > I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker > for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot > configuration/availability issues. > > Who can look at why one of the Python 3.3 slaves is offline and why > the two slaves are configured differently? > > Thanks, > Jean-Paul I would llike to help, but I don't know if I have the permission to do it. In the past, I took a look at how buildbots are configured, and I discovered that there are a lot of customizations and that there is a separate repo/process for updating the buildbots and that special privileges are required to touch the buildslaves. -------- >From my point of view, #7355 is a simple ticket and the fact that it took 4 months and it is still not merged should raise some serious concerns regarding the sustainability of the current Twisted development process. -- Adi Roiban From glyph at twistedmatrix.com Mon Sep 15 00:16:27 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 14 Sep 2014 23:16:27 -0700 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: <20140915000917.31272.1552860934.divmod.xquotient.92@top> References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On Sep 14, 2014, at 5:09 PM, exarkun at twistedmatrix.com wrote: > Hello, > > I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker > for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot > configuration/availability issues. Thank you for (specifically) bringing this up and (generally) getting the 14.1 blockers cleared out. I'm beginning to notice a concerted effort :-). > Who can look at why one of the Python 3.3 slaves is offline and why > the two slaves are configured differently? I've been gradually accumulating the various credentials that I need for maintaining all the buildbots. On my to-do list is centralizing and documenting how all of these are maintained to make it a bit less obscure. It may surprise you that you personally actually have access to those credentials! For the time being I've sent you a separate personal message explaining how you might get access to it, although I apologize that that's as good as we can do right now. This isn't exactly blocked by the following ticket, but filing this is the last thing I did while trying to figure out the Python 3 buildbot situation: I will probably have another look at this tomorrow whether it's fixed or not, but I certainly wouldn't mind if some other folks bumped that ticket and politely reminded the Fedora infrastructure team that invalid certs are a pretty serious operational issue. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Sep 15 00:38:53 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 14 Sep 2014 23:38:53 -0700 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On Sep 14, 2014, at 11:07 PM, Adi Roiban wrote: > On 15 September 2014 01:09, wrote: >> Hello, >> >> I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker >> for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot >> configuration/availability issues. >> >> Who can look at why one of the Python 3.3 slaves is offline and why >> the two slaves are configured differently? >> >> Thanks, >> Jean-Paul > > I would llike to help, but I don't know if I have the permission to do it. > > In the past, I took a look at how buildbots are configured, and I > discovered that there are a lot of customizations and that there is a > separate repo/process for updating the buildbots and that special > privileges are required to touch the buildslaves. Special privileges are required because one of the aspects of buildslave maintenance is cloud control panel access to the cloud providers that maintain this hardware, which, if abused, could end up billing the non-profit which manages Twisted's funds an unlimited amount of money. The buildbots are also managed via shell accounts, which could of course be used to deploy malware. If you are interested in helping to perform maintenance on the buildslaves, access to those permissions could be arranged for you. It had not occurred to me to ask you before, because if I recall correctly, you'd refused commit privileges when they were offered in the past. -glyph From hawkowl at atleastfornow.net Mon Sep 15 03:51:24 2014 From: hawkowl at atleastfornow.net (HawkOwl) Date: Mon, 15 Sep 2014 17:51:24 +0800 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 Sep 2014, at 14:16, Glyph wrote: > On Sep 14, 2014, at 5:09 PM, exarkun at twistedmatrix.com wrote: > >> Hello, >> >> I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker >> for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot >> configuration/availability issues. > > Thank you for (specifically) bringing this up and (generally) getting the 14.1 blockers cleared out. I'm beginning to notice a concerted effort :-). Thanks, exarkun, Glyph, as well as the others which have helped fix tickets and help things along for the 14.1 release — it’s not going to be long before I can get 14.1pre1 gracing all your hard drives! <3 -hawkowl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From hs at ox.cx Mon Sep 15 05:31:08 2014 From: hs at ox.cx (Hynek Schlawack) Date: Mon, 15 Sep 2014 13:31:08 +0200 Subject: [Twisted-Python] 14.1 & regressions (was: Python 3.3 buildslaves) In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 Sep 2014, at 8:16, Glyph wrote: >> I noticed that https://twistedmatrix.com/trac/ticket/7355 is a >> blocker >> for the release of Twisted 14.1 and #7355 is blocked Python 3.3 >> buildbot >> configuration/availability issues. > > Thank you for (specifically) bringing this up and (generally) getting > the 14.1 blockers cleared out. I'm beginning to notice a concerted > effort :-). Speaking of which: I would greatly appreciate if someone with intimate knowledge of the test plumbing code could help me to figure out https://twistedmatrix.com/trac/ticket/7370 and https://twistedmatrix.com/trac/ticket/7429 *** In any case, as soon as the buildbot’s Python installations are updated to more recent versions of cryptography/pyOpenSSL our buildbots will turn red immediately. Thanks, Hynek From exarkun at twistedmatrix.com Tue Sep 16 07:26:35 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 16 Sep 2014 13:26:35 -0000 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders Message-ID: <20140916132635.31272.1641679653.divmod.xquotient.97@top> Hello, These two builders appear to be having problems. See, for example: https://buildbot.twistedmatrix.com/builders/natty64-py2.7maint/builds/4193 https://buildbot.twistedmatrix.com/builders/twistedchecker/builds/2229 Jean-Paul From glyph at twistedmatrix.com Tue Sep 16 18:14:47 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 16 Sep 2014 17:14:47 -0700 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: <20140916132635.31272.1641679653.divmod.xquotient.97@top> References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> Message-ID: <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> On Sep 16, 2014, at 6:26 AM, exarkun at twistedmatrix.com wrote: > Hello, > > These two builders appear to be having problems. See, for example: > > https://buildbot.twistedmatrix.com/builders/natty64-py2.7maint/builds/4193 > https://buildbot.twistedmatrix.com/builders/twistedchecker/builds/2229 For what it's worth, the EOL on Natty was almost 2 years ago now: I'm inclined to do-release-upgrade on that buildbot and simply remove the natty builder, and work on fixing the twistedchecker builder once everything is on supported versions of stuff. If nobody objects by tomorrow I'll probably just proceed with that, since I don't think it was ever explicitly our intent to support OSes that have been EOL'd by their vendors (except perhaps the unkillable, eternal XP...). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From hawkowl at atleastfornow.net Wed Sep 17 05:57:48 2014 From: hawkowl at atleastfornow.net (HawkOwl) Date: Wed, 17 Sep 2014 19:57:48 +0800 Subject: [Twisted-Python] Twisted 14.0.1 Release Announcement Message-ID: <2FECA0C4-21A1-4F16-A667-E397095C3B0E@atleastfornow.net> On behalf of Twisted Matrix Laboratories, I’m releasing Twisted 14.0.1, a security release for Twisted 14.0. It is strongly suggested that users of 14.0.0 upgrade to this release. This patches a bug in Twisted Web’s Agent, where BrowserLikePolicyForHTTPS would not honour the trust root given, and would use the system trust root instead. This would have broken, for example, attempting to pin the issuer for your HTTPS application because you only trust one issuer. Note: on OS X, with the system OpenSSL, you still can't fully rely on this API for issuer pinning, due to modifications by Apple — please see for more details. You can find the downloads at (or alternatively ). The NEWS file is also available at . Thanks for Alex Gaynor for discovering the bug, Glyph & Alex for developing a patch, and David Reid for reviewing it. Twisted Regards, HawkOwl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 455 bytes Desc: Message signed with OpenPGP using GPGMail URL: From mithrandi at mithrandi.net Wed Sep 17 09:41:36 2014 From: mithrandi at mithrandi.net (Tristan Seligmann) Date: Wed, 17 Sep 2014 17:41:36 +0200 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> Message-ID: On 17 September 2014 02:14, Glyph Lefkowitz wrote: > For what it's worth, the EOL on Natty was almost 2 years ago now: > > Speaking of EOL, squeeze is effectively EOL now too (unless you count the squeeze-lts effort, which is very limited in scope); as such, I would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) which are still running squeeze. I think newer version of Debian are already covered by other buildslaves; so are these still needed at all? -- mithrandi, i Ainil en-Balandor, a faer Ambar From twisted-python at 2xlp.com Wed Sep 17 19:31:16 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Wed, 17 Sep 2014 21:31:16 -0400 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? Message-ID: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> Background: I'm using a 3rd party API client, which uses httplib2 in the internals. ( the google-plus api ) I'm placing this blocking code within a DeferredList, that runs within a twisted.enterprise.adbapi.ConnectionPool.runInteraction In my initial tests, everything worked fine -- but I was only making a single GET/POST, as I kept the size of the DeferredList to 1 while I worked on the code. As soon as I increased the limit and had 2 GET/POST events firing off at the same time, I started getting a ssl/socket error ( httplib2 > httplib > socket > ssl ). It works fine in twisted if I make simultaneous requests otherwise (using twisted or the `requests` package). this seems to be tied to how httplib2/httplib is running within twisted. Has anyone experienced this before? if so, do you know if there is a workaround? The easiest fix, is probably for me to write a new ApiClient tomorrow for the methods I need. I'd just like to avoid that if possible From andrew at bemusement.org Wed Sep 17 19:44:11 2014 From: andrew at bemusement.org (Andrew Bennetts) Date: Thu, 18 Sep 2014 11:44:11 +1000 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? In-Reply-To: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> References: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> Message-ID: <20140918014411.GA29843@flay.puzzling.org> Jonathan Vanasco wrote: > > Background: > I'm using a 3rd party API client, which uses httplib2 in the internals. > ( the google-plus api ) > I'm placing this blocking code within a DeferredList, that runs within a > twisted.enterprise.adbapi.ConnectionPool.runInteraction > > In my initial tests, everything worked fine -- but I was only making a single > GET/POST, as I kept the size of the DeferredList to 1 while I worked on the > code. > > As soon as I increased the limit and had 2 GET/POST events firing off at the > same time, I started getting a ssl/socket error ( httplib2 > httplib > > socket > ssl ). What's the error? > It works fine in twisted if I make simultaneous requests otherwise (using > twisted or the `requests` package). this seems to be tied to how > httplib2/httplib is running within twisted. There shouldn't be anything special about httplib2/httplib “within twisted”. Unless httplib2 is mucking with global socket defaults or something (and I doubt it does) I wouldn't expect any interaction between the two at all. It's no different to using httplib2 in a thread without Twisted. -Andrew. > > Has anyone experienced this before? if so, do you know if there is a > workaround? > > The easiest fix, is probably for me to write a new ApiClient tomorrow for the > methods I need. I'd just like to avoid that if possible > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > From glyph at twistedmatrix.com Wed Sep 17 23:16:49 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 17 Sep 2014 22:16:49 -0700 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> Message-ID: <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> On Sep 17, 2014, at 8:41 AM, Tristan Seligmann wrote: > On 17 September 2014 02:14, Glyph Lefkowitz wrote: >> For what it's worth, the EOL on Natty was almost 2 years ago now: >> >> > > Speaking of EOL, squeeze is effectively EOL now too (unless you count > the squeeze-lts effort, which is very limited in scope); as such, I > would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) > which are still running squeeze. I think newer version of Debian are > already covered by other buildslaves; so are these still needed at > all? Not as far as I know. I think you can feel free to deprovision them (although I think you need to submit a PR for the buildmaster's config to remove them from the buildmaster and supported list) -g From dustin at v.igoro.us Thu Sep 18 06:35:45 2014 From: dustin at v.igoro.us (Dustin J. Mitchell) Date: Thu, 18 Sep 2014 08:35:45 -0400 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? In-Reply-To: <20140918014411.GA29843@flay.puzzling.org> References: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> <20140918014411.GA29843@flay.puzzling.org> Message-ID: Are you by any chance using the same HTTPConnection instance in both threads? Dustin On Wed, Sep 17, 2014 at 9:44 PM, Andrew Bennetts wrote: > Jonathan Vanasco wrote: >> >> Background: >> I'm using a 3rd party API client, which uses httplib2 in the internals. >> ( the google-plus api ) >> I'm placing this blocking code within a DeferredList, that runs within a >> twisted.enterprise.adbapi.ConnectionPool.runInteraction >> >> In my initial tests, everything worked fine -- but I was only making a single >> GET/POST, as I kept the size of the DeferredList to 1 while I worked on the >> code. >> >> As soon as I increased the limit and had 2 GET/POST events firing off at the >> same time, I started getting a ssl/socket error ( httplib2 > httplib > >> socket > ssl ). > > What's the error? > >> It works fine in twisted if I make simultaneous requests otherwise (using >> twisted or the `requests` package). this seems to be tied to how >> httplib2/httplib is running within twisted. > > There shouldn't be anything special about httplib2/httplib “within twisted”. > Unless httplib2 is mucking with global socket defaults or something (and I doubt > it does) I wouldn't expect any interaction between the two at all. It's no > different to using httplib2 in a thread without Twisted. > > -Andrew. > >> >> Has anyone experienced this before? if so, do you know if there is a >> workaround? >> >> The easiest fix, is probably for me to write a new ApiClient tomorrow for the >> methods I need. I'd just like to avoid that if possible >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python at twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From twisted-python at 2xlp.com Thu Sep 18 10:58:55 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Thu, 18 Sep 2014 12:58:55 -0400 Subject: [Twisted-Python] are there known ssl/threading/socket issues with httplib2/httplib and Twisted ? In-Reply-To: References: <44906663-ED82-40D9-8F70-617578C15C9F@2xlp.com> <20140918014411.GA29843@flay.puzzling.org> Message-ID: <5BFD8277-22EF-4764-93E7-DD75268E9A4F@2xlp.com> On Sep 18, 2014, at 8:35 AM, Dustin J. Mitchell wrote: > Are you by any chance using the same HTTPConnection instance in both threads? > > Dustin Ack! It looks like the snippet of sample code that I used as a constructor did that. I did not see that. This api library and docs are... frustrating. i've been trying to keep everything tied to the sample code to avoid problems. Everything is working perfect now. Thanks. ( the error was below, sorry for not posting that last night ). including here for search engines in case someone else uses this. ---- File "build/bdist.macosx-10.6-intel/egg/apiclient/http.py", line 716, in execute File "/Users/jvanasco/webserver/environments/stddev-2.7.5/lib/python2.7/site-packages/httplib2/__init__.py", line 1570, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/Users/jvanasco/webserver/environments/stddev-2.7.5/lib/python2.7/site-packages/httplib2/__init__.py", line 1317, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/Users/jvanasco/webserver/environments/stddev-2.7.5/lib/python2.7/site-packages/httplib2/__init__.py", line 1300, in _conn_request content = response.read() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 548, in read s = self.fp.read() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1311, in read return s + self._file.read() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 351, in read data = self._sock.recv(rbufsize) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 241, in recv return self.read(buflen) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 160, in read return self._sslobj.read(len) ssl.SSLError: [Errno 1] _ssl.c:1363: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number From mithrandi at mithrandi.net Sat Sep 20 12:01:44 2014 From: mithrandi at mithrandi.net (Tristan Seligmann) Date: Sat, 20 Sep 2014 20:01:44 +0200 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> Message-ID: On 18 September 2014 07:16, Glyph wrote: > On Sep 17, 2014, at 8:41 AM, Tristan Seligmann wrote: >> Speaking of EOL, squeeze is effectively EOL now too (unless you count >> the squeeze-lts effort, which is very limited in scope); as such, I >> would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) >> which are still running squeeze. I think newer version of Debian are >> already covered by other buildslaves; so are these still needed at >> all? > > Not as far as I know. I think you can feel free to deprovision them (although I think you need to submit a PR for the buildmaster's config to remove them from the buildmaster and supported list) Is the buildmaster's config in a publically accessible repo? If so, where? (If not, how do I get access?) -- mithrandi, i Ainil en-Balandor, a faer Ambar From weykent at weasyl.com Sat Sep 20 12:31:22 2014 From: weykent at weasyl.com (weykent) Date: Sat, 20 Sep 2014 11:31:22 -0700 Subject: [Twisted-Python] On testing things wrapping other things Message-ID: Hi, Recently I worked on refactoring a project so that it was, in my opinion, more testable. For reference, the diff is available at , and the important parts in the changed version are at . My question: is this a good factoring for testing a wrapper? It seems to make sense to split apart crafting and issuing requests to clients, as it lets you test the contents of a request to see what will happen instead of having to issue the request and test that its effects happened. On a previous iteration of this project, I wrote tests that did exactly that: issued requests and checked the bytes that were sent to a StringTransport. Those tests ended up being harder to read and write, and seemed to rely on the exact implementation of the thing I was wrapping. Thanks in advance for any feedback on the matter, ~weykent From trac at twistedmatrix.com Sun Sep 21 00:05:17 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 21 Sep 2014 06:05:17 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1411279517.16.9147535600233956262@twistedmatrix.com An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Sep 21 11:29:42 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 21 Sep 2014 10:29:42 -0700 Subject: [Twisted-Python] natty64-py2.7maint & twistedchecker builders In-Reply-To: References: <20140916132635.31272.1641679653.divmod.xquotient.97@top> <50CB1DBD-7216-4AC9-9C59-7B126BBC15CC@twistedmatrix.com> <4AF2070E-8CEC-4F9D-AB37-8BD56D56F06B@twistedmatrix.com> Message-ID: <8E15331B-F496-4EC8-B6E3-DDD15837B57F@twistedmatrix.com> On Sep 20, 2014, at 11:01 AM, Tristan Seligmann wrote: > On 18 September 2014 07:16, Glyph wrote: >> On Sep 17, 2014, at 8:41 AM, Tristan Seligmann wrote: >>> Speaking of EOL, squeeze is effectively EOL now too (unless you count >>> the squeeze-lts effort, which is very limited in scope); as such, I >>> would like to upgrade or retire my buildslaves (bot-idnar-debian{,64}) >>> which are still running squeeze. I think newer version of Debian are >>> already covered by other buildslaves; so are these still needed at >>> all? >> >> Not as far as I know. I think you can feel free to deprovision them (although I think you need to submit a PR for the buildmaster's config to remove them from the buildmaster and supported list) > > Is the buildmaster's config in a publically accessible repo? If so, > where? (If not, how do I get access?) It's here: -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Sun Sep 21 11:36:37 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 21 Sep 2014 10:36:37 -0700 Subject: [Twisted-Python] On testing things wrapping other things In-Reply-To: References: Message-ID: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> On Sep 20, 2014, at 11:31 AM, weykent wrote: > Hi, > > Recently I worked on refactoring a project so that it was, in my opinion, more testable. For reference, the diff is available at , and the important parts in the changed version are at . > > My question: is this a good factoring for testing a wrapper? It seems to make sense to split apart crafting and issuing requests to clients, as it lets you test the contents of a request to see what will happen instead of having to issue the request and test that its effects happened. If anything I'd say that the problem with this code is that it's not a public, documented interface. The clients of this library might want to test it as well :). > On a previous iteration of this project, I wrote tests that did exactly that: issued requests and checked the bytes that were sent to a StringTransport. Those tests ended up being harder to read and write, and seemed to rely on the exact implementation of the thing I was wrapping. If you're going to check the bytes in a StringTransport, that means you need an implementation of the other end of the protocol as well. If you're just asserting about the exact bytes, then that's really brittle and will break in response to lots of incidental changes. You can use this as a part of a higher-level testing strategy, like so: but in this case the unit test code doesn't see the response to its requests until it's been parsed into a sensible data structure, and assertions are about that. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From weykent at weasyl.com Sun Sep 21 11:52:12 2014 From: weykent at weasyl.com (weykent) Date: Sun, 21 Sep 2014 10:52:12 -0700 Subject: [Twisted-Python] On testing things wrapping other things In-Reply-To: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> References: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> Message-ID: On Sep 21, 2014, at 10:36 AM, Glyph wrote: > If anything I'd say that the problem with this code is that it's not a public, documented interface. The clients of this library might want to test it as well :). Ah! You are referring to both _wrap and _issueRequest here, right? That makes sense, and it’s easy to make those documented public interfaces. > If you're going to check the bytes in a StringTransport, that means you need an implementation of the other end of the protocol as well. If you're just asserting about the exact bytes, then that's really brittle and will break in response to lots of incidental changes. > > You can use this as a part of a higher-level testing strategy, like so: > > > > but in this case the unit test code doesn't see the response to its requests until it's been parsed into a sensible data structure, and assertions are about that. Okay, this makes sense. Of course, Twisted doesn’t come with an implementation of a memcache server, and I’d rather not implement one myself for the purpose of testing. I think I’ll stick with my latest approach, then. Would you employ the same approach for testing non-wrappers as well? That is, if you were testing the client implementation of some wire protocol, would you still write parsers for the server implementation of the protocol to make assertions about the sent bytes? As far as I can tell, this is not what, say, the existing memcache protocol tests do. From glyph at twistedmatrix.com Sun Sep 21 12:07:53 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 21 Sep 2014 11:07:53 -0700 Subject: [Twisted-Python] On testing things wrapping other things In-Reply-To: References: <3257A578-D2A5-4523-87F2-FC4DBAAC696D@twistedmatrix.com> Message-ID: <04A598A2-0EB5-44A1-AEF6-A937F3244DE8@twistedmatrix.com> On Sep 21, 2014, at 10:52 AM, weykent wrote: > On Sep 21, 2014, at 10:36 AM, Glyph wrote: > >> If anything I'd say that the problem with this code is that it's not a public, documented interface. The clients of this library might want to test it as well :). > > Ah! You are referring to both _wrap and _issueRequest here, right? That makes sense, and it’s easy to make those documented public interfaces. > >> If you're going to check the bytes in a StringTransport, that means you need an implementation of the other end of the protocol as well. If you're just asserting about the exact bytes, then that's really brittle and will break in response to lots of incidental changes. >> >> You can use this as a part of a higher-level testing strategy, like so: >> >> >> >> but in this case the unit test code doesn't see the response to its requests until it's been parsed into a sensible data structure, and assertions are about that. > > Okay, this makes sense. Of course, Twisted doesn’t come with an implementation of a memcache server, and I’d rather not implement one myself for the purpose of testing. I think I’ll stick with my latest approach, then. > > Would you employ the same approach for testing non-wrappers as well? That is, if you were testing the client implementation of some wire protocol, would you still write parsers for the server implementation of the protocol to make assertions about the sent bytes? As far as I can tell, this is not what, say, the existing memcache protocol tests do. If I were testing the wire protocol itself I'd probably assert about the exact bytes directly. At that level, changes to the exact bytes being emitted are not an incidental change, that's exactly what the system under test is supposed to be doing; producing bytes. So changing those tests would definitely be worthwhile. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From sandrieu at greenflex.com Mon Sep 22 08:01:06 2014 From: sandrieu at greenflex.com (Simon Andrieu) Date: Mon, 22 Sep 2014 14:01:06 +0000 Subject: [Twisted-Python] GPRS/Modem protocol through USB support Message-ID: Hi, I work for a company that develops telemetry boxes, and that uses the Twisted framework running on the main board. It is planned to integrate a cellular module connected via USB or serial link to the main board. Could you help me to know if GPRS/3G/Modem protocols through USB are supported with Twisted please? And how it is possible to use this interface? Thanks for your help. Best Regards, Simon Andrieu -------------- next part -------------- An HTML attachment was scrubbed... URL: From bram at diomedia.be Mon Sep 22 08:57:55 2014 From: bram at diomedia.be (Bram Van Steenlandt) Date: Mon, 22 Sep 2014 16:57:55 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: References: Message-ID: <542038F3.5000400@diomedia.be> Hi, I'm using a gsm modem for sending sms messages, the modem is rs232 so I have to use and ftdi adapter. You can use these with pyserial in a thread in twisted. It uses AT commands, I have one from sierra wireless, some phones can do the same. 3G/internet is something else, here you just need something the operating system will support, twisted doesn't care how you are connected. Bram op 22-09-14 16:01, Simon Andrieu schreef: > > Hi, > > I work for a company that develops telemetry boxes, and that uses the > Twisted framework running on the main board. > > It is planned to integrate a cellular module connected via USB or > serial link to the main board. > > Could you help me to know if GPRS/3G/Modem protocols through USB are > supported with Twisted please? And how it is possible to use this > interface? > > Thanks for your help. > > Best Regards, > > Simon Andrieu > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Mon Sep 22 09:05:29 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 22 Sep 2014 17:05:29 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <542038F3.5000400@diomedia.be> References: <542038F3.5000400@diomedia.be> Message-ID: <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> Hi Bram & Simon, I’ll answer both e-mails in one go :-) On 22 Sep 2014, at 16:57, Bram Van Steenlandt wrote: > I'm using a gsm modem for sending sms messages, the modem is rs232 so I have to use and ftdi adapter. > You can use these with pyserial in a thread in twisted. Huh! Interesting; what drove that choice, given twisted’s serial support? > op 22-09-14 16:01, Simon Andrieu schreef: >> >> Could you help me to know if GPRS/3G/Modem protocols through USB are supported with Twisted please? And how it is possible to use this interface? Depends what that thing exposes. Usually it’s a character device, so you end up whistling some bytes. However, if you are *just* interested in a very standard Internet connection setup, it is probably much easier to use the existing OS tools for connecting to the internet using that model, than to write something using Twisted. Twisted would be interesting if you want to do interesting things beyond that; e.g. talk to the device and ask it for all sorts of e.g. telemetry data, data that is not already exposed in some standard, easy to consume way. hth lvh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bram at diomedia.be Mon Sep 22 09:31:33 2014 From: bram at diomedia.be (Bram Van Steenlandt) Date: Mon, 22 Sep 2014 17:31:33 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> Message-ID: <542040D5.9010404@diomedia.be> No specific reason actually, I didn't even know twisted had support for a serial port. This being said: ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. How long does twisted has this serial port support actually ? Bram op 22-09-14 17:05, Laurens Van Houtven schreef: > Hi Bram & Simon, > > I'll answer both e-mails in one go :-) > > On 22 Sep 2014, at 16:57, Bram Van Steenlandt wrote: > >> I'm using a gsm modem for sending sms messages, the modem is rs232 so I have to use and ftdi adapter. >> You can use these with pyserial in a thread in twisted. > Huh! Interesting; what drove that choice, given twisted's serial support? > >> op 22-09-14 16:01, Simon Andrieu schreef: >>> Could you help me to know if GPRS/3G/Modem protocols through USB are supported with Twisted please? And how it is possible to use this interface? > Depends what that thing exposes. Usually it's a character device, so you end up whistling some bytes. However, if you are *just* interested in a very standard Internet connection setup, it is probably much easier to use the existing OS tools for connecting to the internet using that model, than to write something using Twisted. Twisted would be interesting if you want to do interesting things beyond that; e.g. talk to the device and ask it for all sorts of e.g. telemetry data, data that is not already exposed in some standard, easy to consume way. > > hth > lvh > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From twisted-python at 2xlp.com Mon Sep 22 10:00:35 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Mon, 22 Sep 2014 12:00:35 -0400 Subject: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool) Message-ID: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> I've got some database transactional code running within a thread via RunInteraction (it needs to rollback on error). One of the features on the logic is that it interacts with a 3rd party API, which may or may not have a ratelimit in effect. Most 3rd party APIs i consume have a minute/hourly based limit -- so I just do a graceful fail on the transaction to requeue it, and set a local timeout. This particular API has a per-second meter in effect. It is considerably more efficient to just sleep for a second and "try again" a second later. I end up being able to process over twice as many API calls in a given period with that strategy. The problem I ran into, is that I need a second database connection to note that a ratelimit has been created in the database -- and i'd like to pull that db connection out of the twisted connection pool to simplify my connection management. - I can't do it within the existing connection, because I am in a transaction and the limit won't appear to other twisted jobs. - I can't use runInteraction() off the pool, because that will defer to a thread and I am in a thread, and the last time I did something like that... everything broke. - the stopgap I have right now is just creating/closing a raw pscycopg2 connection and using that. it works, but it requires a bit more management of postgres to ensure I can balance the right number of connections. has anyone had a need like this before? if so, what did you do? i basically just need to make two "INSERT" statements from a second connection. From _ at lvh.io Mon Sep 22 10:03:26 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 22 Sep 2014 18:03:26 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <542040D5.9010404@diomedia.be> References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> <542040D5.9010404@diomedia.be> Message-ID: On 22 Sep 2014, at 17:31, Bram Van Steenlandt wrote: > No specific reason actually, I didn't even know twisted had support for a serial port. > > This being said: > ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. > I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). > Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. Use cases vary :-) In a wait-dosomething-wait scenario, I find @inlineCallbacks to typically yield virtually the same code you ended up with. When I was doing this, there was some pretty complex chatter back and forth, and I was multiplexing several connections over a single RS232 device, so Twisted certainly came in handy. In your case: don’t fix what ain’t broken, of course :) > Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. > > How long does twisted has this serial port support actually ? Well over a decade: Mon Dec 16 04:04:25 2002 to be precise. See svn rev trunk at 4528. hth lvh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bram at diomedia.be Mon Sep 22 11:34:04 2014 From: bram at diomedia.be (Bram Van Steenlandt) Date: Mon, 22 Sep 2014 19:34:04 +0200 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> <542040D5.9010404@diomedia.be> Message-ID: <54205D8C.2070606@diomedia.be> op 22-09-14 18:03, Laurens Van Houtven schreef: > On 22 Sep 2014, at 17:31, Bram Van Steenlandt wrote: > >> No specific reason actually, I didn't even know twisted had support for a serial port. >> >> This being said: >> ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. >> I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). >> Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. > Use cases vary :-) In a wait-dosomething-wait scenario, I find @inlineCallbacks to typically yield virtually the same code you ended up with. > > When I was doing this, there was some pretty complex chatter back and forth, and I was multiplexing several connections over a single RS232 device, so Twisted certainly came in handy. In your case: don't fix what ain't broken, of course :) Wouldn't it be difficult to combine the twisted method with external libaries like MinimalModbus, libftdi , libartnet etc ? Or is there some magic that I'm not aware of ? I there a disadvantage to using a thread ? >> Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. >> >> How long does twisted has this serial port support actually ? > Well over a decade: Mon Dec 16 04:04:25 2002 to be precise. See svn rev trunk at 4528. > > hth > lvh > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From werner at thieprojects.ch Mon Sep 22 15:25:43 2014 From: werner at thieprojects.ch (Werner Thie) Date: Mon, 22 Sep 2014 11:25:43 -1000 Subject: [Twisted-Python] GPRS/Modem protocol through USB support In-Reply-To: <54205D8C.2070606@diomedia.be> References: <542038F3.5000400@diomedia.be> <176F1F5E-AA0D-447C-87E8-6C67E44069D8@lvh.io> <542040D5.9010404@diomedia.be> <54205D8C.2070606@diomedia.be> Message-ID: <542093D7.5020607@thieprojects.ch> Aloha maybe you can get some clues from here https://gist.github.com/claws/2464017 which is pretty close to what your looking for. HTH, Werner On 9/22/14 7:34 AM, Bram Van Steenlandt wrote: > > op 22-09-14 18:03, Laurens Van Houtven schreef: >> On 22 Sep 2014, at 17:31, Bram Van Steenlandt wrote: >> >>> No specific reason actually, I didn't even know twisted had support for a serial port. >>> >>> This being said: >>> ftdio_sio can be a bit buggy at times, esp when using multiple devices on the same pc, ftdi on freebsd & osx isn't very stable either. >>> I use this interface selector system where the serial connection can be setup with pyserial,libftdi (which works much better for ftdi adapters) or a tcp socket (which uses a so called "device server" with ethernet and rs232 onboard). >>> Doing it with twisted seems more complicated to me (but I could be wrong), in my case it makes a lot of sense to wait for the data, I poll plc's over rs485 and those kind of things. >> Use cases vary :-) In a wait-dosomething-wait scenario, I find @inlineCallbacks to typically yield virtually the same code you ended up with. >> >> When I was doing this, there was some pretty complex chatter back and forth, and I was multiplexing several connections over a single RS232 device, so Twisted certainly came in handy. In your case: don’t fix what ain’t broken, of course :) > Wouldn't it be difficult to combine the twisted method with external > libaries like MinimalModbus, libftdi , libartnet etc ? > Or is there some magic that I'm not aware of ? > I there a disadvantage to using a thread ? > > >>> Also as I've said, there are sometimes a lot of hardware issues (usb bus resets and stuff, users that disconnect usb-ports and plug them back in) which can be difficult to deal with. >>> >>> How long does twisted has this serial port support actually ? >> Well over a decade: Mon Dec 16 04:04:25 2002 to be precise. See svn rev trunk at 4528. >> >> hth >> lvh >> >> >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python at twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > From glyph at twistedmatrix.com Mon Sep 22 23:45:16 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 22 Sep 2014 22:45:16 -0700 Subject: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool) In-Reply-To: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> References: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> Message-ID: On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco wrote: > has anyone had a need like this before? if so, what did you do? Yes. I re-wrote twisted.enterprise entirely, with a better design. You can see the result here: And you can 'pip install' it from here: Hopefully it will see a non-alpha release at some point. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From twisted-python at 2xlp.com Tue Sep 23 11:23:49 2014 From: twisted-python at 2xlp.com (Jonathan Vanasco) Date: Tue, 23 Sep 2014 13:23:49 -0400 Subject: [Twisted-Python] advice sought - two database connections in a RunInteraction (adbapi.ConnectionPool) In-Reply-To: References: <44E88BC2-DCB6-46AE-8A73-805A3EFEE1FE@2xlp.com> Message-ID: oh great! I'll jump through the docs this weekend! thanks so much for the reply and for writing that! On Sep 23, 2014, at 1:45 AM, Glyph wrote: > > On Sep 22, 2014, at 9:00 AM, Jonathan Vanasco wrote: > >> has anyone had a need like this before? if so, what did you do? > > Yes. I re-wrote twisted.enterprise entirely, with a better design. > > You can see the result here: > > > > And you can 'pip install' it from here: > > > > Hopefully it will see a non-alpha release at some point. > > -glyph > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python // Jonathan Vanasco c. 646.729.6436 | 415.501.9815 e. jonathan at 2xlp.com w. http://findmeon.com/user/jvanasco linkedin. http://linkedin.com/in/jonathanvanasco blog. http://destructuring.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Wed Sep 24 20:26:17 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 25 Sep 2014 02:26:17 +0000 (UTC) Subject: [Twisted-Python] Security Advisory: bash remote code execution Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, Today a security release of Bash was issued, fixing a critical vulnerability. This vulnerability allows an attacker to inject and execute arbitrary code on many web servers and other applications. This issue does not affect Twisted directly, but will affect many users of Twisted. Any web server which is serving traffic over a CGI or CGI-like interface (including WSGI) should upgrade its version of Bash immediately. This issue has been assigned CVE-2014-6271. A complete description of the bug is also available: http://seclists.org/oss- sec/2014/q3/650 New packages have been issued for the following operating systems: * Debian: http://www.debian.org/security/2014/dsa-3032 * Ubuntu: http://www.ubuntu.com/usn/usn-2362-1/ * Red Hat Enterprise Linux: https://rhn.redhat.com/errata/RHSA-2014-1293.html Please be aware that there are reports that the current patches do not completely solve the issue, it is likely that users will need to perform updates a second time. Sincerely, Alex Gaynor -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJUI30jAAoJEBJfXGff6UCEQfkQAIi36kVAmGimqiKeqgpbTX5F Z9TRMIhwVtZBtohQXj34QB+y382ArzLXve6toaj0si7lbQN0KnqOqsCyhJ5magxp ZtX/6ZDYcrgRM07sP+JVS+OO5YZnrH/EjfcxKA6kyDk2Rbs7rq21EHuw13HaS1Xo cPiusCM8XIhWZ9Fv8H1fSF7VkXb791kQJGMgcivvww3Z9+WltNbIuDOc57iuXIQ8 AnOxGkGcJL0qQHYkBYqXQejFNBMy0JZNE15ORyep7rYjDvCVacCfOyfjyd9kqpTt QHomR1XdMhZBffpygKbHnJ7wIyFcG0eQMwJxsX8gelmevvVwMmGC+qbmN4R+h1F6 HoSe0y5JHRNFgk3ClKBFgJtD1PB4Jrxl2YqCr+EyzwaQERzSes0bCGy4D1/VVn/Z JWoedrFl7R/pf+DAD/krTA0fVbfDprhlvc4c72p94m0h2xzgcI3RCHT+ZopyIMW8 7Fw0hawx8366m1E0RZBJsi/zvq8Eq/WhVbeTl2C9Vqgg3yE+DUvg6a65y78yXUN4 wgOzI/34ZHQbTll4wRBabllR7yU/WFCLJE95TTS5QIXhikzkd+T7MjiqeSSfEIfZ 55XFvKy1Z2AM/USozvLfwM5siCn+r2o3BmQ97Wm5i0YTbrJqUW2KFSM0xZp3YBQc ryYFOFCmIfZpjPEIheqD =oBm7 -----END PGP SIGNATURE----- From glyph at twistedmatrix.com Wed Sep 24 23:44:06 2014 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 24 Sep 2014 22:44:06 -0700 Subject: [Twisted-Python] Security Advisory: bash remote code execution In-Reply-To: References: Message-ID: On Sep 24, 2014, at 7:26 PM, Alex Gaynor wrote: > Please be aware that there are reports that the current patches do not > completely solve the issue, it is likely that users will need to perform > updates a second time. These reports are being tracked as CVE-2014-7169. More information here: Good night and good luck, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From haggardii at gmail.com Thu Sep 25 09:09:20 2014 From: haggardii at gmail.com (Matt Haggard) Date: Thu, 25 Sep 2014 09:09:20 -0600 Subject: [Twisted-Python] Security Advisory: bash remote code execution In-Reply-To: References: Message-ID: > > Any web server which is serving traffic over a CGI or CGI-like interface > (including WSGI) should upgrade its version of Bash immediately. > I feel ignorant, but I'm confused about how WSGI is affected (and have failed to exploit my WSGI app). AFAICT from reading the code, Twisted's WSGIResource doesn't invoke a shell. I see that it has an `environ` attribute that gets filled with user-provided information, but I don't see how that makes it into a shell's environment. We'll patch bash anyway. Thanks, Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Sep 25 13:54:29 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 25 Sep 2014 12:54:29 -0700 Subject: [Twisted-Python] Security Advisory: bash remote code execution In-Reply-To: References: Message-ID: <3FF9F33B-AFC4-474C-9CC5-AAC62C8D1735@twistedmatrix.com> On Sep 25, 2014, at 8:09 AM, Matt Haggard wrote: > > > > Any web server which is serving traffic over a CGI or CGI-like interface > > (including WSGI) should upgrade its version of Bash immediately. > > > > I feel ignorant, but I'm confused about how WSGI is affected (and have failed to exploit my WSGI app). AFAICT from reading the code, Twisted's WSGIResource doesn't invoke a shell. I see that it has an `environ` attribute that gets filled with user-provided information, but I don't see how that makes it into a shell's environment. As Alex's post said, this vulnerability does not affect Twisted directly. The point is that most people deploying web services are doing so in a UNIX environment, and in so doing they are probably invoking scripts of various kinds, or executables which may have been replaced with wrapper shell-scripts. It's hard to audit for environment variables containing attacker-controlled data, and this is the sort of thing we've all been trained to expect is safe, if they're variables in our own "namespace", so it's possible that any number of 3rd-party tools you are using with Twisted are vulnerable in surprising ways. So everybody should just upgrade :). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Sep 25 16:31:27 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 25 Sep 2014 15:31:27 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool Message-ID: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Hi all, (For those of you wondering about the subject, note that this message is sent in compliance with This email sent in compliance with .) I've been trying to improve the reliability of the buildbots. (On that note, by the way, several builders are failing and more will be soon as security updates roll out, it would be great if someone could review before the whole fleet turns red.) In so doing I (re-)discovered this bug: . At first I just wanted to delete an intermittently failing test which appears to be nonsense. However, after some prodding from exarkun, I investigated and discovered that this very poorly-written test case does in fact illustrate a real problem with our current threadpool implementation which can result in deadlocks, hangs on exit, and other unpleasantness. In my use of Twisted I have encountered several "huh, this can't happen, maybe cosmic rays or something" bugs which might have been caused by this, so I would very much like to fix it. One reason that our threadpool implementation is problematic is that it has a somewhat complex internal design, lots of weird little features (context preservation, completion callbacks, workload estimation) all built at the same layer with no composition. I tried to figure out a way to salvage its current architecture and I could not reasonably do so. I have prototyped a completely new threadpool implementation ("twisted.threads") in a spike. It is definitely not ready for review, but I would definitely appreciate design-level feedback on the code right now. You can see the implementation here: (or here: if that's more your speed) It's less code than the existing implementation and provides more useful features at the same time. It might, in fact, provide some of the threading primitives that we would need for a reactor-per-thread implementation. However, there is a very ugly implementation detail that prevents us from marching onwards to a glorious multithreading future: twisted.internet.interfaces.IReactorThreads.getThreadPool. The nominally "public" interface provided by its documented return type, the concrete (old-style!) class twisted.python.threadpool.ThreadPool, is ridiculously over-broad. Here are the features (attributes and methods) in its "public" interface: callInThread callInThreadWithCallback adjustPoolsize start stop q min max joined started name waiters threads working workers start startAWorker stopAWorker dumpStats __getstate__ __setstate__ threadFactory currentThread Here's what I would like its public interface to be, more or less what it's intended to be, based on the ways it's documented and used in Twisted and in the various projects I can see using it: callInThread callInThreadWithCallback adjustPoolsize start stop It would not be too much effort to also preserve, for legacy purposes: min max joined started name dumpStats However, I would REALLY like to delete these implementation details: workers threads working startAWorker q __getstate__ __setstate__ threadFactory currentThread Deleting these implementation details straight up would also make it such that anyone who inherited from the public class ThreadPool would not be able to override any of these things (oh my goodness, why would you do that, I really hope nobody has done that). The three ways I could proceed, in order of my own preference, are: Put the new implementation with entirely new interfaces into twisted.threads, put a compatibility shim into twisted.python.threadpool.ThreadPool that wraps those objects, provides my more minimal interface proposed above, and deletes 90% of its tests. Change IReactorThreads.getThreadPool to return a documented interface considerably more restricted than Compatibility implications: twisted.internet.interfaces.IReactorThreads's contract would be relaxed. Implementors of IReactorThreads would not see an impact unless they were talking to deleted parts of ThreadPool internally. ThreadPool's clients would have several methods removed, and subclassing would effectively not be possible any more (a bunch of overridable hooks would have been removed, and no replacements would be provided). However, I prefer this option because there are an extremely restrictive set of use-cases (basically: only logging, any behavioral changes would have been broken) where clients could have made legitimate use of these attributes or overriding any of these functions. Delete all of twisted.python.threadpool's test coverage, deprecate the entire module, and delete it in the next release (14.1+1). This has the advantage of making the test suite reliable, and gets the benefits for any users of callInThread, but does not relay any of the benefits to users of ThreadPool. The only ones doing this directly in Twisted are deferToThreadPool (and by extension, deferToThread) and WSGIResource. I could update these in tandem, however; the code changes would be very small. I don't think it would be incompatible to make WSGIResource itself accept a different type of constructor argument (although if we don't, I wonder if subclassing anything with a constructor can be considered applicable to the compatibility policy in Twisted; hmm). Actually go through the trouble of emulating all the attributes on ThreadPool and emulating them as best I can. Delete the direct tests for ThreadPool itself and write a private subclass that will be returned from getThreadPool that returns a compatibility shim which isinstance(ThreadPool) for compatibility and still has all the gross attributes. This change would be technically compatible, but would be a huge amount of additional work and I am doubtful that it would provide any value. So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From weykent at weasyl.com Thu Sep 25 19:36:16 2014 From: weykent at weasyl.com (weykent) Date: Thu, 25 Sep 2014 18:36:16 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? Yes. Specifically, I am maintaining this AMP responder method: @FetchThreadPoolStats.responder def fetchThreadPoolStats(self): pool = self.factory.threadPool return dict( threadsWaiting=len(pool.waiters), threadsWorking=len(pool.working), workerQueueSize=pool.q.qsize(), ) These statistics are chunked into graphite and displayed as a nice little graph. They’ve been quite useful on some occasions: sometimes all of the threads in a thread pool for a WSGI application got blocked, and the symptoms were that all of the requests coming in were timing out at the proxy in front of twisted. We were able to quickly tell that the problem was the WSGI thread pool because the threadsWorking count was at its limit and the workerQueueSize was skyrocketing. > If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) I don’t really see this as an abuse of the public interface. If possible, I’d like to see this diagnostic information preserved in the new interface, as I consider it invaluable information as long as we’re using twisted as a WSGI runner. I will say that it is certainly possible for me to emulate these attributes, but that would require touching implementation details of WSGIResource. I’m not sure which is worse. ~weykent From glyph at twistedmatrix.com Fri Sep 26 01:10:01 2014 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 26 Sep 2014 00:10:01 -0700 Subject: [Twisted-Python] naming question before we end up committed... Message-ID: While working on the package I just brought up in another thread, I realize we have another big new package coming in 14.1, "twisted.python.logger", and I'm wondering what the ".python" part of that package name is doing in there, and what value it adds. It's where "twisted.python.log" used to be, of course, but now I'm just wondering if we're just making the same mistake twice. Does anyone have an opinion on renaming it to "twisted.logger"? Or have a better suggestion? -glyph From glyph at twistedmatrix.com Fri Sep 26 01:43:22 2014 From: glyph at twistedmatrix.com (Glyph) Date: Fri, 26 Sep 2014 00:43:22 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> Message-ID: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> On Sep 25, 2014, at 6:36 PM, weykent wrote: > On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz wrote: > >> So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? > > Yes. Specifically, I am maintaining this AMP responder method: Wow, cool. So, okay, while I am a little unhappy that you're using this API in a slightly unfortunate way, that feeling is eclipsed by the joy that The Process Works :-). Thank you very much for responding so promptly, so specifically, and including the exact code that we need to discuss. > @FetchThreadPoolStats.responder > def fetchThreadPoolStats(self): > pool = self.factory.threadPool > return dict( > threadsWaiting=len(pool.waiters), > threadsWorking=len(pool.working), > workerQueueSize=pool.q.qsize(), > ) > > These statistics are chunked into graphite and displayed as a nice little graph. They’ve been quite useful on some occasions: sometimes all of the threads in a thread pool for a WSGI application got blocked, and the symptoms were that all of the requests coming in were timing out at the proxy in front of twisted. We were able to quickly tell that the problem was the WSGI thread pool because the threadsWorking count was at its limit and the workerQueueSize was skyrocketing. Analytics, monitoring, logging, and statistics have historically been a blind spot for Twisted and I am really glad you're bringing this up. I was thinking about bringing it up in my original message, but as my colleagues and friends will tell you, I tend to put too many words into emails, so unless I'm sure I need those words, I will delete them. The new interface should very definitely have metrics-gathering functionality, and possibly even a logging component. >> If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) > > I don’t really see this as an abuse of the public interface. If possible, I’d like to see this diagnostic information preserved in the new interface, as I consider it invaluable information as long as we’re using twisted as a WSGI runner. It's not exactly an "abuse", because the public interface is there, and you have a use-case, and you're satisfying it with what's available. For the purposes that you are using these attributes, I actually don't have a problem reproducing them at all. It should be pretty straightforward. My concern with exposing them overall is that what you've got with "ThreadPool.q" is not a queue that has a qsize() method that you can inspect, it's that it's a Queue that you can put things into, get things out of, and generally muck about with. Similarly pool.waiters is not simply an object with a __len__ that tells you how active the threads are, it's a list of Thread objects which you could potentially change the names of, turn into daemon threads, join(), and so on. More importantly it's a list that you could remove Thread objects from and that would affect their interactions with the pool. How about this: would you mind if ThreadPool were modified to still populate these attributes for monitoring and debugging purposes, but completely stopped honoring any destructive operations on them? In the case of "pool.q", there is no single Queue responsible for the whole pool, so I would like to make "put", "get", and "join" actually raise exceptions; in the case of "waiters" and "working" I guess I could fish out some actual Thread objects, that just involves a little bit of sad abstraction violation internal to the twisted.threads.Team implementation. My inclination would be to expand the thread pool to accommodate these usages, but still deprecate these attributes in the next version; but add a better "statistics" method that returned an object with "active", "pending", and "idle" attributes (all integers). The most important thing that I want to make sure nobody wants is to keep overriding (or calling) startAWorker, stopAWorker, __getstate__, __setstate__. Supporting callers of threadFactory and currentThread would be easy, but the attributes aren't really there for calling, they're there to be stubbed, and stubbing them won't keep working without tons of extra work. So would that work for you? > I will say that it is certainly possible for me to emulate these attributes, but that would require touching implementation details of WSGIResource. I’m not sure which is worse. Touching private (underscore-prefixed) implementation details is explicitly not supported - if you did it that way, you're on your own :-). And, as the end-user in this discussion, you win the argument by default. (You can see how I feel about supporting users here: ) Plus, if we're going to work on the thread-pool and have newer, better interfaces, WSGIResource's implementation is likely to change. Thanks a bunch for helping us improve Twisted, -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Fri Sep 26 01:49:03 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 26 Sep 2014 08:49:03 +0100 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: On 26 September 2014 08:10, Glyph wrote: > While working on the package I just brought up in another thread, I realize we have another big new package coming in 14.1, "twisted.python.logger", and I'm wondering what the ".python" part of that package name is doing in there, and what value it adds. It's where "twisted.python.log" used to be, of course, but now I'm just wondering if we're just making the same mistake twice. > > Does anyone have an opinion on renaming it to "twisted.logger"? Or have a better suggestion? > > -glyph I think that twisted.logger is better. Description of twisted.python is: Twisted Python: Utilities and Enhancements for Python ... and for me the new logger is more than an utility or enhancement to python language or stdlib. -- Adi Roiban From _ at lvh.io Fri Sep 26 01:53:06 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Fri, 26 Sep 2014 09:53:06 +0200 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: <2C57F9FB-041F-462F-AE88-F07FAFC34BA1@lvh.io> +0 on “logger” (top-level names have a history of whimsy) +1 on making it top-level hth lvh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: Message signed with OpenPGP using GPGMail URL: From weykent at weasyl.com Fri Sep 26 02:12:53 2014 From: weykent at weasyl.com (weykent) Date: Fri, 26 Sep 2014 01:12:53 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> Message-ID: <02CD8913-6E7C-4D53-B3DE-91A8AED4B460@weasyl.com> On Sep 26, 2014, at 12:43 AM, Glyph wrote: > How about this: would you mind if ThreadPool were modified to still populate these attributes for monitoring and debugging purposes, but completely stopped honoring any destructive operations on them? In the case of "pool.q", there is no single Queue responsible for the whole pool, so I would like to make "put", "get", and "join" actually raise exceptions; in the case of "waiters" and "working" I guess I could fish out some actual Thread objects, that just involves a little bit of sad abstraction violation internal to the twisted.threads.Team implementation. Having objects with only nondestructive methods would be fine for me. I don’t even care about getting Thread objects out, and I certainly hope nobody else cares about getting Thread objects out either. > My inclination would be to expand the thread pool to accommodate these usages, but still deprecate these attributes in the next version; but add a better "statistics" method that returned an object with "active", "pending", and "idle" attributes (all integers). This sounds fantastic. I’ll graph whatever the statistics method emits, so feel free to include other useful or interesting metrics as well. (Though, I’m not sure what else there is that’s easy to keep track of and useful.) > So would that work for you? Yes, absolutely. > Touching private (underscore-prefixed) implementation details is explicitly not supported - if you did it that way, you're on your own :-). And, as the end-user in this discussion, you win the argument by default. (You can see how I feel about supporting users here: ) Plus, if we're going to work on the thread-pool and have newer, better interfaces, WSGIResource's implementation is likely to change. I haven’t touched private implementation details of WSGIResource, nor was I planning to. I’ve already had some code (in another project) break across versions of twisted because of that, and I don’t want it to happen again. > Thanks a bunch for helping us improve Twisted, Thank you for taking the time to write up your original post and this reply! ~weykent From adi at roiban.ro Fri Sep 26 02:20:30 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 26 Sep 2014 09:20:30 +0100 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> Message-ID: On 26 September 2014 08:43, Glyph wrote: [snip] > My inclination would be to expand the thread pool to accommodate these > usages, but still deprecate these attributes in the next version; but add a > better "statistics" method that returned an object with "active", "pending", > and "idle" attributes (all integers). +1 for statistics method I am using some of those member for my custom test case https://github.com/chevah/empirical/blob/master/chevah/empirical/testcase.py#L66 A method which return those integers would be enough for my use case. ---- +1 for reducing the public interface. With a public adjustPoolsize I prefer to also have read-only attributes for min / max pool size. With a public start/stop a public read-only started attribute might be useful.... for the case when you don't want to do a try/catch or just want to see the state of the pool. dumpStats could be replaced by a better method dedicated to statistics, which will return integer instead of using the logger. `joined` looks like the opposite of started/running... ie, it is redundant. Regarding `name` attribute, I guess that it might make sense in the case an application uses multiple thread pools... I have never used multiple pools so I can not comment on that. Thanks! -- Adi Roiban From adi at roiban.ro Fri Sep 26 02:26:46 2014 From: adi at roiban.ro (Adi Roiban) Date: Fri, 26 Sep 2014 09:26:46 +0100 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: On 15 September 2014 07:38, Glyph wrote: [snip] > Special privileges are required because one of the aspects of buildslave maintenance is cloud control panel access to the cloud providers that maintain this hardware, which, if abused, could end up billing the non-profit which manages Twisted's funds an unlimited amount of money. The buildbots are also managed via shell accounts, which could of course be used to deploy malware. > > If you are interested in helping to perform maintenance on the buildslaves, access to those permissions could be arranged for you. It had not occurred to me to ask you before, because if I recall correctly, you'd refused commit privileges when they were offered in the past. > > -glyph Sorry for the delay. I would like to help the Twisted project in any way... including with buildslave maintenance. I think that at this level I only need ssh access. -------- I refused commit privileges as I wanted to contribute more patches and do more reviews as a read-only contributors, before starting to mess with Twisted repo. I think that it would be nice to have some process to initiate new contributors to the whole review process and once one or more senior-reviewers consider that the new contributors is good to do reviews by itself, to give him commit privileges. Cheers, -- Adi Roiban From exarkun at twistedmatrix.com Fri Sep 26 07:16:01 2014 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 26 Sep 2014 13:16:01 -0000 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: <20140926131601.17800.326361613.divmod.xquotient.0@top> On 08:26 am, adi at roiban.ro wrote: > >I think that it would be nice to have some process to initiate new >contributors to the whole review process and once one or more >senior-reviewers consider that the new contributors is good to do >reviews by itself, to give him commit privileges. +1 Jean-Paul From radix at twistedmatrix.com Fri Sep 26 09:53:20 2014 From: radix at twistedmatrix.com (Christopher Armstrong) Date: Fri, 26 Sep 2014 10:53:20 -0500 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> Message-ID: On September 25, 2014 at 8:39:54 PM, weykent (weykent at weasyl.com) wrote: On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? Yes. Specifically, I am maintaining this AMP responder method: @FetchThreadPoolStats.responder def fetchThreadPoolStats(self): pool = self.factory.threadPool return dict( threadsWaiting=len(pool.waiters), threadsWorking=len(pool.working), workerQueueSize=pool.q.qsize(), ) These statistics are chunked into graphite and displayed as a nice little graph. They’ve been quite useful on some occasions: sometimes all of the threads in a thread pool for a WSGI application got blocked, and the symptoms were that all of the requests coming in were timing out at the proxy in front of twisted. We were able to quickly tell that the problem was the WSGI thread pool because the threadsWorking count was at its limit and the workerQueueSize was skyrocketing. We did very similar stuff (logging thread pool size to graphite) at id, so they're probably still using those bits too. > If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-)  I don’t really see this as an abuse of the public interface. If possible, I’d like to see this diagnostic information preserved in the new interface, as I consider it invaluable information as long as we’re using twisted as a WSGI runner.  I will say that it is certainly possible for me to emulate these attributes, but that would require touching implementation details of WSGIResource. I’m not sure which is worse.  ~weykent  --  Christopher Armstrong http://twitter.com/radix http://wordeology.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jml at mumak.net Sat Sep 27 00:47:52 2014 From: jml at mumak.net (Jonathan Lange) Date: Sat, 27 Sep 2014 07:47:52 +0100 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> <8D3BAEA0-630B-4D02-A0F4-63E4C1FB60C2@weasyl.com> <35944278-5967-49EE-B317-3D77C9169A37@twistedmatrix.com> Message-ID: On Fri, Sep 26, 2014 at 8:43 AM, Glyph wrote: > > On Sep 25, 2014, at 6:36 PM, weykent wrote: > > On Sep 25, 2014, at 3:31 PM, Glyph Lefkowitz > wrote: > > So, does anyone out there have any code which makes use of the > aforementioned bad attributes of ThreadPool, whose applications would break > if I removed them? > > > Yes. Specifically, I am maintaining this AMP responder method: > > > Wow, cool. > > So, okay, while I am a *little* unhappy that you're using this API in a > slightly unfortunate way, that feeling is eclipsed by the joy that The > Process Works :-). > Yes! I saw this thread and immediately grabbed a colleague, shoved my phone under his face and said *this* is what I love about Twisted. Carry on. jml -------------- next part -------------- An HTML attachment was scrubbed... URL: From trac at twistedmatrix.com Sun Sep 28 00:05:18 2014 From: trac at twistedmatrix.com (trac at twistedmatrix.com) Date: Sun, 28 Sep 2014 06:05:18 -0000 Subject: [Twisted-Python] Weekly Bug Summary Message-ID: 1411884318.24.6180239602225432630@twistedmatrix.com An HTML attachment was scrubbed... URL: From hs at ox.cx Sun Sep 28 00:19:43 2014 From: hs at ox.cx (Hynek Schlawack) Date: Sun, 28 Sep 2014 08:19:43 +0200 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: > Does anyone have an opinion on renaming it to "twisted.logger"? Or have a better suggestion? +1 on dropping `python` here. That's a weird namespace anyway something as central as logging shouldn't be there. From wolfgang.kde at rohdewald.de Sun Sep 28 00:52:11 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Sun, 28 Sep 2014 08:52:11 +0200 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) Message-ID: <2329405.l76rWJfZKg@i5> I am getting this traceback on about every source file like: i5:~/ssdsrc/alt/Twisted/twisted/test (pb3) twistedchecker reflect_helper_ZDE.py ************* Module reflect_helper_ZDE W9001: 1,0: Missing copyright header W9002: 1,0: Missing a reference to test module in header C0103: 1,0: Invalid name "reflect_helper_ZDE" for type module (should match (([a-z_][a-z0-9_]*))$) W9208: 1,0: Missing docstring Traceback (most recent call last): File "/usr/local/bin/twistedchecker", line 5, in pkg_resources.run_script('TwistedChecker==0.2.0', 'twistedchecker') File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script self.require(requires)[0].run_script(script_name, ns) File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script execfile(script_filename, namespace, namespace) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/EGG-INFO/scripts/twistedchecker", line 10, in Runner().run(sys.argv[1:]) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/core/runner.py", line 325, in run self.linter.check(args) File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 542, in check self.check_astng_module(astng, walker, rawcheckers) File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 615, in check_astng_module walker.walk(astng) File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/utils.py", line 553, in walk cb(astng) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 179, in visit_module self._runPEP8Checker(node.file) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 188, in _runPEP8Checker recorder = PEP8WarningRecorder(file) File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 49, in __init__ self.run() File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 78, in run pep8.Checker.check_all(self) File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1445, in check_all self.check_logical() File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1338, in check_logical for offset, text in self.run_check(check, argument_names) or (): File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1278, in run_check return check(*arguments) TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) -- Wolfgang From chriswwolfe at gmail.com Sun Sep 28 09:36:46 2014 From: chriswwolfe at gmail.com (Chris) Date: Sun, 28 Sep 2014 10:36:46 -0500 Subject: [Twisted-Python] naming question before we end up committed... In-Reply-To: References: Message-ID: +1 for twisted.logger. From glyph at twistedmatrix.com Sun Sep 28 18:15:37 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sun, 28 Sep 2014 17:15:37 -0700 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <2329405.l76rWJfZKg@i5> References: <2329405.l76rWJfZKg@i5> Message-ID: How did you install twistedchecker? -glyph On Sep 27, 2014, at 11:52 PM, Wolfgang Rohdewald wrote: > I am getting this traceback on about every source file like: > > > i5:~/ssdsrc/alt/Twisted/twisted/test (pb3) twistedchecker reflect_helper_ZDE.py > ************* Module reflect_helper_ZDE > W9001: 1,0: Missing copyright header > W9002: 1,0: Missing a reference to test module in header > C0103: 1,0: Invalid name "reflect_helper_ZDE" for type module (should match (([a-z_][a-z0-9_]*))$) > W9208: 1,0: Missing docstring > Traceback (most recent call last): > File "/usr/local/bin/twistedchecker", line 5, in > pkg_resources.run_script('TwistedChecker==0.2.0', 'twistedchecker') > File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script > self.require(requires)[0].run_script(script_name, ns) > File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script > execfile(script_filename, namespace, namespace) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/EGG-INFO/scripts/twistedchecker", line 10, in > Runner().run(sys.argv[1:]) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/core/runner.py", line 325, in run > self.linter.check(args) > File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 542, in check > self.check_astng_module(astng, walker, rawcheckers) > File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/lint.py", line 615, in check_astng_module > walker.walk(astng) > File "/usr/lib/python2.7/dist-packages/pylint-0.26.0-py2.7.egg/pylint/utils.py", line 553, in walk > cb(astng) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 179, in visit_module > self._runPEP8Checker(node.file) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 188, in _runPEP8Checker > recorder = PEP8WarningRecorder(file) > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 49, in __init__ > self.run() > File "/usr/lib/python2.7/dist-packages/TwistedChecker-0.2.0-py2.7.egg/twistedchecker/checkers/pep8format.py", line 78, in run > pep8.Checker.check_all(self) > File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1445, in check_all > self.check_logical() > File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1338, in check_logical > for offset, text in self.run_check(check, argument_names) or (): > File "/usr/lib/python2.7/dist-packages/pep8-1.5.7-py2.7.egg/pep8.py", line 1278, in run_check > return check(*arguments) > TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) > > > -- > Wolfgang > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python at twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python From josh at bartletts.id.au Sun Sep 28 19:56:04 2014 From: josh at bartletts.id.au (Joshua Bartlett) Date: Mon, 29 Sep 2014 11:56:04 +1000 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: On 26 September 2014 08:31, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the > aforementioned bad attributes of ThreadPool, whose applications would break > if I removed them? If so, how can I make you feel as bad about yourselves > for using it as I feel bad about myself for writing it? ;-) > There are a few times in a codebase I help maintain where we want a thread pool of all daemon threads. The code we have for this overrides threadFactory() like so: class DaemonThreadPool(ThreadPool): def threadFactory(self, *args, **kwargs): t = threading.Thread(*args, **kwargs) t.setDaemon(True) return t How would I do this with your proposed new API? Josh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 29 00:16:00 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 29 Sep 2014 08:16 +0200 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: References: <2329405.l76rWJfZKg@i5> Message-ID: <4230856.dFXEK2FZA3@i5> Am Sonntag, 28. September 2014, 17:15:37 schrieb Glyph: > How did you install twistedchecker? installing is not even needed: git clone https://github.com/twisted/twistedchecker cd twistedchecker ~/src/Twisted/bin/trial twistedchecker trial being the latest from twisted svn trunk, I get the same error pylint --version No config file found, using default configuration pylint 0.26.0, astng 0.24.3, common 0.61.0 Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] -- Wolfgang From glyph at twistedmatrix.com Mon Sep 29 01:07:33 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Sep 2014 00:07:33 -0700 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <4230856.dFXEK2FZA3@i5> References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> Message-ID: <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> On Sep 28, 2014, at 11:16 PM, Wolfgang Rohdewald wrote: > Am Sonntag, 28. September 2014, 17:15:37 schrieb Glyph: >> How did you install twistedchecker? > > installing is not even needed: > > git clone https://github.com/twisted/twistedchecker > cd twistedchecker > ~/src/Twisted/bin/trial twistedchecker > > trial being the latest from twisted svn trunk, I get the same error > > pylint --version > No config file found, using default configuration > pylint 0.26.0, > astng 0.24.3, common 0.61.0 > Python 2.7.6 (default, Mar 22 2014, 22:59:56) > [GCC 4.8.2] Hmm. It seems as though twistedchecker does not itself have continuous integration set up, so I don't have any test results to look at. It seems to be working on our buildbots (for now, at least) but I'm not sure why. This week I will try to get merged, among other things. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.kde at rohdewald.de Mon Sep 29 01:31:07 2014 From: wolfgang.kde at rohdewald.de (Wolfgang Rohdewald) Date: Mon, 29 Sep 2014 09:31:07 +0200 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> Message-ID: <2623018.bnDGBy8Oq5@i5> Am Montag, 29. September 2014, 00:07:33 schrieb Glyph: > This week I will try to get merged, which requires exactly pep8 1.5.1 Ubuntu 14.04 installs 1.4.6 I have the current 1.5.7 After downgrading to 1.4.6, twistedchecker works correctly. -- Wolfgang From glyph at twistedmatrix.com Mon Sep 29 02:05:47 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Sep 2014 01:05:47 -0700 Subject: [Twisted-Python] Python 3.3 buildslaves In-Reply-To: <20140915000917.31272.1552860934.divmod.xquotient.92@top> References: <20140915000917.31272.1552860934.divmod.xquotient.92@top> Message-ID: <508ABC57-1F2C-4CFE-A49C-8594183E0AD6@twistedmatrix.com> On Sep 14, 2014, at 5:09 PM, exarkun at twistedmatrix.com wrote: > Hello, > > I noticed that https://twistedmatrix.com/trac/ticket/7355 is a blocker > for the release of Twisted 14.1 and #7355 is blocked Python 3.3 buildbot > configuration/availability issues. > > Who can look at why one of the Python 3.3 slaves is offline and why > the two slaves are configured differently? The configuration of the two slaves has been made consistent, and I have reviewed and landed the relevant ticket. At long last, that particular square on the build matrix is green. -glyph From glyph at twistedmatrix.com Mon Sep 29 04:11:20 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 29 Sep 2014 03:11:20 -0700 Subject: [Twisted-Python] glorious green Message-ID: <3B0EFF3D-F676-4CDE-BDE5-C02D34F26402@twistedmatrix.com> I will probably be going to sleep before py-select-gc gets around to finishing, but other than that, is completely green. If were reviewed, we might be able to make look a bit more like the solid green rectangle it really ought to be. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Sep 29 13:11:55 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 29 Sep 2014 12:11:55 -0700 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: <2623018.bnDGBy8Oq5@i5> References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> <2623018.bnDGBy8Oq5@i5> Message-ID: On Sep 29, 2014, at 12:31 AM, Wolfgang Rohdewald wrote: > Am Montag, 29. September 2014, 00:07:33 schrieb Glyph: >> This week I will try to get merged, > > which requires exactly pep8 1.5.1 > > Ubuntu 14.04 installs 1.4.6 > I have the current 1.5.7 > > After downgrading to 1.4.6, twistedchecker works correctly. The issue is actually logilab-common, not pep8 itself; twistedchecker can deal with more recent pep8 versions. I fixed this on twistedchecker master: More exciting, though, is Adi Roiban's contribution of Travis-CI continuous integration: so we can pick up the pace on twistedchecker improvements a bit. (The buildbot already runs straight out of Git, so improvements go live almost immediately.) -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From adi at roiban.ro Tue Sep 30 00:36:20 2014 From: adi at roiban.ro (Adi Roiban) Date: Tue, 30 Sep 2014 07:36:20 +0100 Subject: [Twisted-Python] twistedchecker TypeError: modifiedBlankLines() takes exactly 6 arguments (7 given) In-Reply-To: References: <2329405.l76rWJfZKg@i5> <4230856.dFXEK2FZA3@i5> <10DDD935-E53F-4EC2-8847-B153A7CBF807@twistedmatrix.com> <2623018.bnDGBy8Oq5@i5> Message-ID: On 29 September 2014 20:11, Glyph Lefkowitz wrote: > On Sep 29, 2014, at 12:31 AM, Wolfgang Rohdewald > wrote: > > Am Montag, 29. September 2014, 00:07:33 schrieb Glyph: > > This week I will try to get > merged, > > > which requires exactly pep8 1.5.1 > > Ubuntu 14.04 installs 1.4.6 > I have the current 1.5.7 > > After downgrading to 1.4.6, twistedchecker works correctly. > > > The issue is actually logilab-common, not pep8 itself; twistedchecker can > deal with more recent pep8 versions. > I got this similar error when I was using pep8 1.5.1 ... after upgrade to 1.5.6 all was fine. Please check latest setup.py from master... and the install line from travis.yml Cheers! -- Adi Roiban From glyph at twistedmatrix.com Tue Sep 30 22:09:12 2014 From: glyph at twistedmatrix.com (Glyph) Date: Tue, 30 Sep 2014 21:09:12 -0700 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: On Sep 28, 2014, at 6:56 PM, Joshua Bartlett wrote: > On 26 September 2014 08:31, Glyph Lefkowitz wrote: > So, does anyone out there have any code which makes use of the aforementioned bad attributes of ThreadPool, whose applications would break if I removed them? If so, how can I make you feel as bad about yourselves for using it as I feel bad about myself for writing it? ;-) > > > There are a few times in a codebase I help maintain where we want a thread pool of all daemon threads. The code we have for this overrides threadFactory() like so: > > > class DaemonThreadPool(ThreadPool): > def threadFactory(self, *args, **kwargs): > t = threading.Thread(*args, **kwargs) > t.setDaemon(True) > return t > > > How would I do this with your proposed new API? I suppose honoring the threadFactory attribute is possible. In the new thread pool prototype I've created so far, "Team", provides pluggability for creating different kinds of "Worker": and the thread "Worker" provides pluggability for creating different kinds of thread: so I'll make sure that this keeps working. That said: why did you need a threadpool of daemon threads? -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh at bartletts.id.au Tue Sep 30 23:12:36 2014 From: josh at bartletts.id.au (Joshua Bartlett) Date: Wed, 1 Oct 2014 15:12:36 +1000 Subject: [Twisted-Python] INCOMPATIBLE CHANGE: twisted.python.threadpool In-Reply-To: References: <8E367ECE-C3E2-4CEC-A5B6-5419B603D453@twistedmatrix.com> Message-ID: On 1 October 2014 14:09, Glyph wrote: > > That said: why did you need a threadpool of daemon threads? > That's a very good question. One of my comaintainers thinks the answer is, "My codebase is old enough to vote and to answer your question I'd probably have to turn our SVN server back on. Please don't make me." :-) I think the real reason is that the third-party library that we're calling from the threads sometimes takes a really long time to return, which means that unless we use daemon threads the only timely way to shut down / restart the service is kill -9, which is not ideal. Josh. -------------- next part -------------- An HTML attachment was scrubbed... URL: