[Twisted-Python] Specialized mail server

Rob Hoadley hoadley at gmail.com
Fri Apr 20 12:54:19 MDT 2007


Hello,

I'm building a new specialized mail server using twisted.mail and wanted to
get some feedback from the twisted community.

Some background:

The goal of the mail server is to answer and categorize all our bounces.
This includes bounces from our outbound mail servers that never make it to a
remote server or queue timeouts (i call these conversational bounces) and
bounces via the return-path (non-conversational bounces) that made it to the
remote mailserver but get bounced after remote mailserver acceptance.

If you are familiar with non-conversational bounces you'll know that you are
not guaranteed to get a bounce email in a standard format.  It all depends
on the foreign mailserver software and policies.

To handle the variations in bounce messages, I'm going to make a bounce
server that is configurable via addition of new rules similar to the
ipchains.  Each rule is a regex and the rules will look similar to the
Mailman bounce code regex's.  These rules are stored in a db with a whole
test system written around the rules to protect against bad rule entry and
fallback points to previous rule sets.  For the purposes of the server, the
idea is to start at the top of the rule chain and work your way down.  So
really, I'm just talking about a bunch of regex's w/processing if a regex is
hit or a final action if the none of the regex's are hit.

I've already built a specialized twisted mail server with a db lookup on
incoming mail that handles some of our other email (roughly 7-10K emails per
day) so I'm pretty familiar with the way twisted.mail works.  Where I need
some help is in the area of higher-volume matters.  I anticipate this
mailserver could get between 40-80K emails per day.  ( I work for a Email
Service Provider, our bounce percentage rates are low ).

My questions are as follows:

1) I've used qmail in the past.  If you are familiar with qmail, you
configure a locals number which tells you the number of concurrent emails
qmail can process locally.  This is different from the number of smtp
connections ( I'm not so much worried about that ).  qmail just forks a new
process up to the locals number and queues up other mail.

in eomRecieved what is the best manner to safeguard that I don't knock over
the server by processing 100 emails in short succession with the regex's?

2) Are there any examples of a high volume email server that somewhat
follows my processing requirements? I've googled around and not found any
examples.  It feels like I should be looking for a queue implementation
within a twisted.mail server.  I've looked at some of the Queue classes
within the twisted.mail code and I wanted to know if this is the right
direction?

3) If you have experience using twisted.mail have you had any problems with
this volume of mail with some sort of minor/major processing on each
incoming mail.  If you've just saved to disk have you had problems with a
load of say 200-300K emails per day?

I appreciate any help or comments.

thanks

-rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070420/b77beaef/attachment.html>


More information about the Twisted-Python mailing list