2
Purge bad emails - scope management
Idea shared by Douglas Foster - 7/9/2025 at 5:12 PM
Proposed
This is a follow-up to this discussion:

As mentioned at the end of that chain, I have begun working on building blocks for this problem.   I am new to REST programming and the SmarterMail API, so it is proceeding in fits and starts.    

Right now, my focus is on using the message retrieval calls to find messages within a single folder of a single account.   In hopes that I can get that working, I have begun thinking about the larger problem of looping through all of the accounts and all of the folders where the message might exists.   This part of the problem exposes a massive performance risk:   If you look in every folder of every user, the payload will be deployed hours before your search finishes, and performance will be clopbbered for all users while it runs.   To limit scope, you need to capture message metadata into a database as the message is being processed.   This is the approach:

Use Declude, Declude Reboot, or you own code that monitors the PROC folder.   When each message comes through, SmarterMail drops an EML file containing the message itself, and a .HDR file containing message metadata.  For this project, the most content is:
  • SMTP Mail From address, 
  • Message From Address, 
  • SMTP Recipient Addresses, 
  • and a timestamp.
Parsing the EML file to extract the message subject will make the captured data even more vaiuable.
Both files disappear as the message is processed, which is why the data has to be captured and stored on the fly.

There still seem bot be some users who do not use Declude or Declude Reboot, and honesly, I cannot understand why.   It is a powerful supplement to any other product you may want buy or build.  Those detractors can probably obtain a similar result, on a next-day basis, by parsing the SMTP and Delivery logs.

Once the database exists, you use SQL queries to find the problem messages and the exact list of recipients for those messages.   Instead of checking thousands of mailboxes, you check a handful.

1
Douglas,
Appreciate your effort to help us admins with this problem! Here's an idea for efficiency -- instead of searching through every folder of every user, can you search the log files to see what users received the phishing message?

Phishing keeps getting worse and we need a way to fight it. Not sure why SmarterTools doesn't jump on this enhancement as it's the #1 most requested item in this forum with 50 votes.  If you come up with a solution, I'm sure a lot of people would be interested in using it.

Thanks,
KevinD
0
I have parsed the log files, but it is more complicated to capture all of the data that you want from that source.  In a crisis, it is better to have the data already at your fingertips than to begin data collection after the crisis starts.

Having used a database of message metadata for years, it is a tremendously valuable tool that I use all the time.  My database holds more than just metadata.  Much of the local policy data that drives my filtering is based on database tables, using custom filters in Declude that invoke stored procedures.   Spam filtering results are also stored in the database, which includes both authentication results and disposition decisions.  All of that allows me to compute statistics, go threat hunting, and evaluate rule changes using historical data.   For space management, I have a script that runs once weekly to discard messages more than 90 days old. 

I believe strongly that if you can wait for someone else to solve your problem, then your problem is not very important.   If the problem is important and urgent, you need to solve it yourself.   

For this problem, the database of message metadata will allow you to immediately start remedying these problem immediately, with manual effort and impersonation of affected users.  My experience is that most such attacks have a very small number of recipients, so manual intervention is feasible once you know where to look.  Automation will be desirable for the long haul, but the security vulnerability gets fixed in the short term.

I don't know that SmarterTools is the right organization to build the database of metadata, so I don't know that this is their problem to solve.  From discussions with product vendors, my understanding of the Office365 remediation feature is that Outlook allows messages to be extracted based on a unique key, and the commercial spam vendors remediate bad messages by providing that key.   The SmarterMail API may not have a magic key, but they provide enough. 



Reply to Thread

Enter the verification text