[Twisted-Python] Implementing Postfix Inet Policy Check Client

[Tom Boland]

Hi Exvito,

Thanks very much for this. I think I might just give a bit of context
about using this.

I have an SMTP proxy using Nginx.  It uses queries an HTTP service to
establish whether mail should be forwarded, and where to (on to
postfix/dovecot/exchange backends).  It uses Twisted for the HTTP
authorisation service, and I've been using a similar model for years for
POP and IMAP in production, and for the last year on an outbound SMTP
relay.  The Twisted http service handles password authentication as well
lookups that point connections on to the correct backend servers.  All
of our custom business logic is in the python (database lookups and the
like).  Twisted has performed brilliantly as the HTTP service for years.

I'm now writing an inbound SMTP proxy service, and initially, it just
does DNSBL checks, which will actually stop around 12 million emails a
day from reaching postfix and tying up smtpd processes!  I fire off a
number of DNSBL checks in a deferred chain, and just come back to it
once the chain has completed.  The postfix policy check was going to be
added to this chain of events and initially I was going to use it to
query the dovecot quota daemon running on the appropriate backend for a
given mailbox, therefore I'd have a few tens of connections open, one to
each of the policy daemons on the dovecot backend servers.  This is just
to avoid queuing and trying to deliver mail to a mailbox that's over-quota.

I think what you've provided me with is useful for me, but I think it's
backwards for my purposes, as I need to be connecting to the policy
daemon rather than being the policy daemon!

I wanted to do this with deferred calls in case one of the policy
daemons becomes unreachable and blocks my application.  Do you think I
should do something differently in that regard?  My SQL lookups are done
synchronously.  If the database server goes away, I've got bigger
problems anyway!

Many thanks for your help.  I'll work a bit on this this morning and
come back!

Thanks again.  Tom.



On 18/11/15 02:14, Ex Vito wrote:
> Tom,
>
> I guess LineReceiver can help you with that, it's a matter of using a
> different delimiter. Here is a rough sketch of something you can start
> exploring:
>
> from __future__ import print_function
>
> from twisted.internet import reactor, protocol, endpoints, defer
> from twisted.protocols import basic
>
>
> class PostfixProtocol(basic.LineReceiver):
>
>     # Assuming Postfix sends '\r\n' line breaks (does it?)
>     delimiter = '\r\n\r\n'
>
>     def lineReceived(self, multi_line):
>         input_dict = {
>             k: v for k, v in (
>                 line.split('=') for line in multi_line.split('\r\n')
>             )
>         }
>         self.gotPostfixRequest(input_dict)
>
>     def gotPostfixRequest(self, request_dict):
>         # Silly deferred-based implementation.
>         d = defer.Deferred()
>         d.addCallback(self.sendPostfixAction)
>         # Simulate a deferred being fired with success.
>         reactor.callLater(1, d.callback, 'OK')
>
>     def sendPostfixAction(self, response):
>         # NOTE: Sends self.delimiter after the payload.
>         #       Use self.tranport.write if you don't want it.
>         self.sendLine('action={}'.format(response))
>
>
> if __name__ == '__main__':
>
>     ep = endpoints.serverFromString(reactor, 'tcp:10000')
>     f = protocol.Factory()
>     f.protocol = PostfixProtocol
>     ep.listen(f)
>     reactor.run()
>
> Key ideas:
> - PostfixProtocol overrides LineReceiver's delimiter (setting it to
> '\r\n\r\n')
> - lineReceived parses the muti_line and calls gotPostfixRequest.
> - gotPostfixRequest should decide (or delegate, you choice) what kind
> of response to send back.
> - sendPostfixAction sends a response back to Postfix.
>
> Notes:
> - The multi_line parsing code is short(ish) but not very robust. It
> may fail in bad ways with invalid input.
> - The gotPostfixRequest implementation is written for the sake of a
> deferred-based example.
>
> Does this help you in any way?
> Cheers,
> --
> exvito
>
>
> On Tue, Nov 17, 2015 at 4:56 PM, Tom Boland <tom at t0mb.net
> <mailto:tom at t0mb.net>> wrote:
> > Greetings all.
> >
> > This may be a very basic question.  I'm hoping to implement a postfix
> > policy check client in twisted.  It's a simple protocol.  You send
> > newline separated key value pairs like:
> >
> > recipient=email at ddr.ess
> > sender=another at ddr.ess
> > size=1024
> > helo_name=mail.server
> > etc..
> >
> > you terminate the request with an additional newline.  The response
> > comes back like
> >
> > action=OK
> >
> > You can send mutliple requests in the same connection.  What I'm
> > envisaging is a module that can be used to provide a deferred
> > request/response pairing to my calling application.  The module class
> > will manage the single connection to the postfix policy daemon (I'm
> > actually going to have persistent connections to a few daemons), and
> > reconnect when necessary etc.  Any requests will return a deferred that
> > I can add callbacks to.  How would you design this with twisted?  I can
> > easily envisage a way of using a clientfactory to instantiate separate
> > connections for each request/response, but actually being able to simply
> > send a request and receive the single response for that request is
> > something I'm struggling to do within a LineReceiver instance (for
> > instance).  Would the twisted.protocols.amp module help given that I
> > can't change the server-side protocol?
> >
> > Any advice much appreciated!
> >
> > Thanks.  Tom.
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> <mailto:Twisted-Python at twistedmatrix.com>
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
> -- 
> --
>   exvito
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20151118/9211d411/attachment-0002.html>