[Twisted-Python] UDP Logging Server

[Pandelis Theodosiou]

On Sat, Mar 12, 2011 at 3:32 AM, Tim Allen <screwtape at froup.com> wrote:

> On Fri, Mar 11, 2011 at 01:15:47PM -0600, SIC FS LIST wrote:
> > So far I have a "working" implementation ... but I'm noticing that if I
> do
> > the following:
> > -- log when a message is received
> > -- that for that message it "might" show up in the file a pretty lengthy
> > period of time later
>
> Assuming the objects stored in DISKINFO[1] etc. are file objects, you
> seem to be writing to the files but never calling flush(). If you don't
> call flush(), Python (well, the C standard library) won't send the data
> on to the OS until its buffer is full, or the file handle is closed. If
> you're not getting that many log lines, it can take a while for that to
> happen.
>
> Of course, if you flush after every disk read, your program will run
> a bit more slowly and with more I/O... for an application where
> reliability is more important than performance (like logging) that's
> probably acceptable.
>

You may also setup a timer that flushes files every, say, 5 minutes.

One other thing I've read in the Python.org site is that flush() is not 100%
sure to work immediately and should be used in combination with os.fsync().
Is there someone that can explain if that is correct?


>
> > The actual UDP protocol:
> >
> > class VocsLogger(DatagramProtocol):
> >     def datagramReceived(self, data, (host, port)):
> >         _proc_msg(self.transport, data, (host,
> > port))._new_msg().addCallback(handler)
>
> _proc_msg doesn't seem to be complicated enough to need its own class,
> why not just do what _proc_msg does in VocsLogger?
>
> > The _proc_msg class:
> >
> > class _proc_msg:
> >     def __init__(self, sck, data, (host, port)):
> >         self._sck = sck
> >         self._data = data
> >         self._host = host
> >         self._port = port
> >
> >     def _new_msg(self):
> >         d, _ = LogMsg().ParseSocketMsg(self._data)
> >         if d.type.upper() == DISKINFO[0]:
> >             DISKINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> >         elif d.type.upper() == LOADAVG[0]:
> >             LOADAVG[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> >         elif d.type.upper() == MEMINFO[0]:
> >             MEMINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> >         elif d.type.upper() == NETDEV[0]:
> >             NETDEV[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> >         elif d.type.upper() == PSAUX[0]:
> >             PSAUX[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> >         elif d.type.upper() == WHOINFO[0]:
> >             WHOINFO[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
> >         else:
> >             DEFAULT[1].write(d.ToString() + "\n%s\n" % (LOG_DELIM))
>
> It depends on what DISKINFO[0] and DISKINFO[1] actually are, but
> assuming they're a string and a file-handle, this code would look more
> Pythonic as something like this:
>
> LOGSINKS = {
>        "DISKINFO": open("/var/log/diskinfo", "w"),
>        "LOADAVG": open("/var/log/loadavg", "w"),
>        "MEMINFO": open("/var/log/meminfo", "w"),
>        "NETDEV": open("/var/log/netdev", "w"),
>        "PSAUX": open("/var/log/psaux", "w"),
>        "WHOINFO": open("/var/log/whoinfo", "w"),
>        "DEFAULT": open("/var/log/default", "w"),
>    }
>
> def _new_msg(self, data):
>    d, _ = LogMsg().ParseSocketMsg(data)
>    type = d.type.upper()
>    sink = LOGSINKS.get(type, LOGSINKS['DEFAULT'])
>    sink.write("%s\n%s\n" % (d.ToString(), LOG_DELIM))
>
> Hope that helps!
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110312/a23fa892/attachment.html>