[Twisted-Python] the good, the log, and the ugly

[Glyph]

On Mar 31, 2012, at 6:10 PM, Brian Warner wrote:

> On 3/30/12 5:49 PM, Glyph wrote:
> 
>>> http://foolscap.lothar.com/docs/logging.html has details.
>> 
>> Indeed, Foolscap has informed many of my future wishes for Twisted
>> logging, especially its incident reporting and logport features. (I
>> wasn't aware of UMIDs but I have independently invented the same
>> thing.)
> 
> (actually, the "UMID" support consists of ignoring extra kwargs, and an
> emacs macro which inserts umid="RANDOM" at point. It works remarkably
> well, and is a lot cheaper than having log.msg() capture the call stack
> so you can figure out which copy of the otherwise-anonymous
> log.msg("done") calls was invoked this time)

I was thinking more along the lines of something that would insert the output of "python -c 'import uuid, sys; sys.stdout.write(str(uuid.uuid4()))'", but yeah, same general idea. (Why do you have to ignore extra kwargs?)

>> Could we perhaps get some of foolscap logging's core features into
>> Twisted? We could have an AMP interface for subscribing to log streams
>> that would be slightly more low-fidelity (but, I would guess,
>> higher-performance) than the Foolscap one, but work based on the same
>> core mechanism which foolscap could then subscribe to.
> 
> Yeah, that'd be great. An AMP interface sounds lovely. I'd have to think
> about potential gotchas, but I'm sure we can work them out. Foolscap's
> remote protocol relies upon all the log.msg() arguments being
> foolscap-serializable, which occasionally reveals places where we're
> accidentally passing object instances into the log. (it uses
> CopiedFailure a lot, so we'd need to come up with an AMP equivalent).
> The stored-on-disk format just uses pickle, which I kinda regret, but it
> should be pretty easy to replace that with something safer.

I was thinking that AMP would offer an AnyOf() argument type, which would be dynamic within a small set, with a fallback.  Within the context of logging, we'd support unicode/bytes/float/int (and maybe some more esoteric numeric tower stuff too, like decimals and fractions), and possibly lists and dicts of those same types (but not structures of structures, so we don't have to worry about backreferences).  Maybe tuples supported the same as lists.

The fallback would simply be "repr() this thing", so if you log an object you still get something somewhat useful without potentially arbitrary serialization.

Foolscap could take the same data and just make a 

> There's also the notion of a "log port" (a foolscap object with a
> remote_subscribe() method). Does AMP have any notion of object
> references?

No, but I don't think we need such a notion.  If we did need it for some reason, AMP does have a notion of establishing new message streams over the same connection.  This message finally prompted me to file the ticket that I always think was already filed: <http://twistedmatrix.com/trac/ticket/5587>.

> If not, I can see how you could call in and ask
> (synchronously) for the current event log, but not how you'd provide a
> "callback object" to which future events should be streamed. How does
> pubsub work in AMP?

You just know that your peer knows how to respond to certain commands, and you send those commands.  For example,

from twisted.protocols.amp import Command, String, AMP
class Pub(Command):
    arguments = [("channel", String()), ("event", String())]
class Sub(Command):
    arguments = [("channel", String())]
class Hub(AMP):
    @Sub.responder
    def sub(self, channel):
        self.factory.observers[channel].append(self)
    @Pub.responder
    def pub(self, channel, event):
        for observer in self.factory.observers[channel]:
            observer.callRemote(Pub, channel=channel, event=event)

Then your peer just has the same definition of Pub and has a @Pub.responder that does whatever they like.  (Eric Mangold was working on formalizing this process somewhat at the Twisted sprint.)

>> I suspect that this would be useful to foolscap as well, because then
>> the code that foolscap calls into (for example, the web stuff for its
>> UIs) would be using the same logging convention.
> 
> Yeah, I'd love to get that factored out and let other people hack on it.
> My web UI skills are laughable.

Actually I was saying that Twisted could provide the core functionality and Foolscap could still provide the web UI - but if you wanted to break things out even further, but I suppose this would also present an opportunity for some third party to plug in some web UI on the side of Twisted that was not necessarily related to Foolscap, which would be cool too.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120331/a548a7cf/attachment.htm