1
Identifying bounced emails
Question asked by Brian - 2/20/2023 at 11:51 AM
Answered
I'm looking to write some C# code to parse <some thing> to identify sent emails that have bounced.  I have automated emails that go out for transaction information.  I also have notification emails that go out based on an event in my domain.  These emails are sent from my software, and use an email address that is specific to the purpose of the email.

Emails can bounce for a variety of reasons.  I'm wanting to identify stale email addresses that are no longer in use (someone signed up for notifications, but have since quit using that email address).  I'm also interested in identifying bad email addresses, usually caused by the user mistyping their email address in to my system.  I see emails in there with incorrect domain names (e.g. @gmail instead of @gmail.com, or just @gmal.com) and I also see emails where the mailbox doesn't exist.  I want to be able to identify these email addresses and the remove or otherwise deal with them in my database.

My first thought was to write a little IMAP tool that would check the inboxes of these email addresses, and then try to parse the emails (headers, subject, body) to determine if the message was indicating a bounce, and if so, what kind of bounce.  So I did that, and quickly realized it got pretty complicated, because not all of the information was in one place, and the format varied widely.

After some web sleuthing, I saw that some people parse the mail logs on the server to do the job.  So I looked at the delivery.log, and it looks like it has exactly what I need:

22:06:18.727 [19020746] Delivery for xxx@yyy.com to aaa@bbb.net has bounced. Reason: Remote host said: 550 5.1.1 Not our Customer
This one line pretty much gives me everything I need.  I can get the from address, the to address, the "bounced" part, and the error number which I can parse and act accordingly.  But one thing gives me pause.  Is this log file likely to change format?  Am I going to have to modify my code regularly because the log format changed?  If its pretty steady, then I'm OK with this option, and if I have to occasionally update my code when updating SmarterMail, then I guess its not that big of a deal.

I see there is a SmarterMail API.  I didn't see anything right away that looked like it would provide what I'm after.  Am I missing something?  Can I get this info from the API?  It might be a little more stable and easier to use than a brute force parsing of a text line in a log file!

Thanks!

7 Replies

Reply to Thread
1
Kyle Kerst Replied
Employee Post
I wasn't able to find any API methods that relate specifically to logging or SMTP statistics unfortunately. I can say that the log format is not likely to change drastically, though we do add more information to it from time to time. The basic structure of the logging generally stays the same though, with the date/time coming first, followed by a session ID and then the log entry itself. 

Something that might help limit how much you have to parse is first identifying the session you're looking for, then returning only the log entries that pertain to that session ID. Perhaps others can chime in with additional suggestions on this though!
Kyle Kerst
System/Network Administrator
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
2
Douglas Foster Replied
My thoughts:
1) List clean-up is critically important.   I had conversations with an organization which sends a daily new summary to a million subscribers.  No ads, no trash content, just news summaries.   Spamhaus blacklisted them and their delivery rate fell 10% overnight.   The explanation:   too many bad addresses in their list.   To get off the list, they were required to implement dual-verification opt-in.   If your list is not opt-in, you are also at risk of being blacklisted when recipients don't want the message and flag your messages as spam.

2) Non-Delivery Types
You have four types of non-delivery
- Message cannot be delivered because domain name is wrong
- Message rejected during SMTP session
- Message rejected via non-delivery report
- Message rejected and silently discarded.
Your analysis options are affected by whether you use an outgoing gateway, and what brand of gateway is used.

Undeliverable Messages
The bad domains can be partially filtered by static analysis of your list.   Typos will appear as domains with few subscribers, and can be checked in advance with DNS lookups to see if MX, A, or AAAA queries return data.  (A and AAAA can be used as substitutes for a missing MX.)   If you don't catch invalid domains in advance, they can take days to return a non-delivery report to the sending account, depending on your retry settings.  So your result analysis needs to allow for the delay.   The failure message will be from your own infrastructure, so parsing the error message should be a one-time effort.   Log analysis may prove more useful than message parsing.

Rejected during SMTP session.
These failures should be relatively easy to parse out of the relevant log file.   The rejection event must include an SMTP Response Code and may include an SMTP Extended Status Codes.
and
If your logs do not contain both types of codes, you will want to consider an outbound gateway that does capture both.

Non-Delivery Report (NDR) validation
One distinguishing feature of NDRs is that the SMTP MailFrom address is (almost always) empty.   Some inbound gateways will add a message header to capture the MailFrom address.   So parsing the message headers might be an option, but might be a bit difficult.   Alternatively, you can use available log data to capture MailFrom addresses and status codes.   

One problem with NDRs is knowing whether they are real responses to your messages or fraud trying to confuse your email filter.  The most effective tool for this is BATV:
and
Barracuda uses a proprietary version of this idea in their products.
If you have an outbound gateway that applies the encoding, and an inbound gateway that can validate it, you can have certainty that the allowed bounces are really from you.   This typically involves using a commercial product from a single vendor for both roles.

NDR parsing
As you have noted, parsing NDR message content is very difficult because the format is inconsistent.   Manual processes are likely to be needed for awhile.  But the number of filtering products is drastically less than the number of recipient domains, so you will probably be able to move from manual to automated parsing over a period of time.

Silent Discards
You will never know about the silent discards.

I hope this helps with your planning,

Doug Foster
0
Brian Replied
Marked As Answer
Nice info Doug!

I'm just a small time operation, with NO marketing emails.  Only transactional emails and opt-in notification emails for events that occur in my domain.  And its small numbers too.  Only a couple of hundred outbound emails on a normal day, maybe up to a few thousand on a busy (notification) day.  My email lists only source is from my client's fingertips.

I like your idea about validating the supplied domain name vs DNS lookup of MX records.  While a bad domain can't really affect your "spam rating", it does result in a user not getting an email they are expecting.  Checking it as part of the email validation process (I don't usually do a formal email verification) just makes sense.  That way you can give them an opportunity to resolve the error while you are still able to communicate with them.  If they just enter a bad email address and walk away, then I have no way to contact them to let them know the email address was bad!

I think this idea to parse the delivery.log file is going to work just fine for me.  I threw some code together and did some testing yesterday.  Here is a little snippet of C# code that does the trick for me, with the current (Feb 2023) version of the log file format:

public class DeliveryLogReader
    {
        private readonly string _logPath;
        private readonly Regex _regexDelivered;
        private readonly Regex _regexBounced;
        private readonly Regex _regexReason;

        public DeliveryLogReader(string logPath)
        {
            _logPath = logPath;
            _regexDelivered = new Regex(@"^(?<timeStamp>\d{2}:\d{2}:\d{2}\.\d{3}) \[(?<sessionId>\d+)\] Delivery for (?<from>\S+) to (?<to>\S+) has completed \(Delivered\)", RegexOptions.Compiled);
            _regexBounced = new Regex(@"^(?<timeStamp>\d{2}:\d{2}:\d{2}\.\d{3}) \[(?<sessionId>\d+)\] Delivery for (?<from>\S+) to (?<to>\S+) has bounced\. Reason: (Remote host said: )?(?<reason>.+)$", RegexOptions.Compiled);
            _regexReason = new Regex(@"^(?<code>\d{3}) (?<subCode>[\d\.]+)? ?(?<reasonDetail>.+)$", RegexOptions.Compiled);
        }

        public void ReadDeliveryLogs()
        {
            var files = Directory.GetFiles(_logPath);
            foreach (var file in files)
            {
                var fileName = Path.GetFileName(file);
                if (fileName.EndsWith("-delivery.log"))
                {
                    Log.Information("Parsing delivery log file: {fileName}", fileName);

                    var results = new DeliveryLogReaderResults();

                    var date = fileName.Substring(0, 10);

                    foreach (var line in File.ReadLines(file))
                    {
                        ProcessDelivered(date, line, results);
                        ProcessBounced(date, line, results);
                    }

                    ReportResults(results);
                }
            }
        }

        private void ProcessDelivered(string date, string line, DeliveryLogReaderResults results)
        {
            var match = _regexDelivered.Match(line);
            if (match.Success)
            {
                var groups = match.Groups;
                results.AddDeliveredResult(date, groups["timeStamp"].Value, groups["sessionId"].Value, groups["from"].Value, groups["to"].Value);
            }
        }

        private void ProcessBounced(string date, string line, DeliveryLogReaderResults results)
        {
            var match = _regexBounced.Match(line);
            if (match.Success)
            {
                var groups = match.Groups;
                var reasonMatch = _regexReason.Match(groups["reason"].Value);

                if (reasonMatch.Success)
                {
                    var reasonGroups = reasonMatch.Groups;
                    results.AddBouncedResult(date, groups["timeStamp"].Value, groups["sessionId"].Value, groups["from"].Value, groups["to"].Value, reasonGroups["code"].Value, reasonGroups["subCode"].Value, reasonGroups["reasonDetail"].Value);
                }
                else
                {
                    results.AddBouncedResult(date, groups["timeStamp"].Value, groups["sessionId"].Value, groups["from"].Value, groups["to"].Value, null, null, groups["reason"].Value);

                }
            }
        }

        private void ReportResults(DeliveryLogReaderResults results)
        {
            foreach (var r in results.BouncedMessages)
            {
                Log.Information("{from} -> {to} bounced: {code} {subCode} {reason}", r.From, r.To, r.Code, r.ExtendedCode, r.Reason);
            }
            var countOfCode = results.BouncedMessages
                .GroupBy(r => r.Code)
                .Select(g => new 
                    { Code = g.Key, Count = g.Count() })
                .OrderByDescending(g => g.Count);

            foreach (var code in countOfCode)
            {
                Log.Information("Code: {code} seen {count} times", code.Code, code.Count);
            }
        }
With this code, I've got all the data I need.  Now I just need to make decisions on what to do about it!  I'm thinking pretty much all 5xx codes can be counted as bad email addresses that can be purged from the email list.  

Maybe this will help someone in the future.
0
Douglas Foster Replied
Glad you have a solution.

One topic to consider is "mailbox full".   Does this mean the user never reads his mail and is a lost cause, or does it mean he messed up and the mailbox will be available again in a day or two?  All other 5xx codes should be fatal.

Glad you found working code.   I have tools to parse my Delivery, SMTP, and Declude logs into a Microsoft SQL database.   They are implemented as SQL stored procedures.   I can make the code available to anyone on request.

0
Brian Replied
Mailbox full.  Hmmm.  If the goal is to get rid of BAD email addresses.  Then I think you don't remove for the case of mailbox full.  If the goal is to get rid of UNUSED email address, then dump it.  However, seems like you would want to see mailbox full, multiple times, over a period of time, before you did anything hasty.

I found a bad one that is a recurring theme:

06:10:55.928 [19023793] Delivery for xxx@yyy.com to abcxyz@bellsouth.net has bounced. Reason: Remote host said: 553 5.3.0 alph747 DNSBL:RBL 521< my_ip_address >_is_blocked.For assistance forward this error to abuse_rbl@abuse-att.net
Its a 553 / 5.3.0 that I am seeing ONLY from att.net.  I'm not sure if this is a result of my recent migration to a new server and a day of sending emails without a Reverse DNS PTR record (so easy to overlook, I always forget that and how important it is).  OR if its a result of years worth of me sending out emails to old/stale/bad email addresses.  I probably have more email addresses for att.net, bellsouth.net, and sbcglobal.net than for any other "domains".  Bet its going to be REAL fun to try and get off their naughty list.  I need to be able to DELIVER the message to know if its good or bad.  If you block me and don't let me have a look now, then how am I to know?
 
Designing the database tables and writing the code to get the parsed log data in there right now!
1
Douglas Foster Replied
Add DKIM signing if you do not have it already, then add a DMARC policy so that recipients will use the DKIM signature for verification.   Stronger authentication makes it easier to get off the naughty list.
0
Brian Replied
Douglas, I really appreciate your input.  SPF, DKIM, and DMARC were all in place and verified, I just forgot about the Reverse DNS PTR record.  It took me almost a day to realize that I forgot it.  And another day for my (now former) web/server hosting company to add one (AFTER I was already live).

Had no idea that there are email validation services out there (but I should have), I guess mainly because it just hasn't been much of an issue for me.  But with a few mis-steps on configuring a new server and combined years of not properly maintaining my email list, its now become clear to me that I need to "pay attention" and be responsible.  Just got set up with zero bounce and I'm going to do some cleaning on my email lists, then try to maintain it properly with my delivery.log parser!

Reply to Thread