I have sputtered about the limitations of spam filtering several times recently.    Blocking bad actors and blocking bad content is a losing process because the problem is infinite.   The only way to block spam is to focus on the wanted mail, which is a finite question.

We started building a dataset of known senders awhile back, using a nightly batch script that parses yesterday's delivery log of accepted messages.   That allowed us to run daily reports of messages from unknown senders, and it showed that malicious traffic was getting through every day.   It also showed that our incoming volume was 98% known senders, 2% unknown senders.   Essentially all of our risk is in that 2%.   Then, we were able to classify the unknown sender traffic into three groups, based on spam risk:

- Group 1:  Messages from mailbox providers like Gmail and Yahoo.   Currently, these have a relatively low spam rate, spam tends to be phishing content rather than direct attacks, and existing tools are catching the spam pretty well.   This group also includes a volume of important messages from clients, which are incorrectly quarantined because our commercial spam filter does not like photos sent from cell phones.  

- Group 2: Messages from email service providers (ESPs) like SendGrid.net and ConstantContact.com, where the SMTP From domain is the ESP and the message From address indicates the client.   Some of these sources are wholly unwanted advertising and can be blocked based on the SMTP From address.   Others have a mix of important messages and unimportant advertising.

- Group 3:  Messages sent directly, so that the SMTP domain and the message domain are the same.    This includes most of our current spam, including the attacks using free gifts from major brands and confirmations of fake payment transactions.

After contemplating this problem for a long time, I have begun moving to a security model that quarantines messages from unknown senders, starting with the last group.   I am already seeing the benefit -- attacks that were getting through are now getting quarantined, and the volume of incorrect quarantine has increased very little.