archive and spool overview
Question asked by Sabatino - 5/9/2024 at 2:28 AM
I remain of the opinion that the log and/or log search system absolutely needs to be improved.

Let's go to the specific case anyway

In the spool overview I see the address bounce-ms.u74602@b74602.multibuy.org in Top Outbound Senders

I want to check.

I go to do an archive search and I can't find anything from this sender.

After some investigation here is a message received from that sender

So in the archive I have to look for info@multibuy.org
However, carrying out checks becomes difficult because first I have to go into the logs to look for the real sender.

Return-Path: <bounce-ms.u74602@b74602.multibuy.org>
Received: from smsh1.mailgenio.com (smsh1.mailgenio.com []) by smtp.mailplan.it with SMTP
    cipher=Aes256 bits=256);
   Thu, 9 May 2024 10:11:57 +0200
Received: (SwiftMTA); Thu, 09 May 2024 10:11:57 +0200
Received: from localhost (localhost [127.0.01])
    by smsh1.mailgenio.com (SwiftMTA/3.0.15) with ESMTPSA id 1EADCAB7-1FFB-496D-AB8F-21B0DBD819F5.122
    envelope-from <info@multibuy.org>
    tls TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (authenticated bits=0);
    Thu, 09 May 2024 10:11:57 +0200
X-MessageSniffer-ResultCode: 0
X-CTCH-RefID: str=0001.0A702F23.663C8553.0122:SCFSTAT110587460,ss=1,re=-4.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0
Authentication-Results: spool.smtp.mailplan.it; iprev=pass (; dkim=pass (rsa-SHA256) header.s=ms1 header.d=multibuy.org header.b="e//jsw+4"
Authentication-Results: smtp.smtp.mailplan.it; dmarc=passed (v=DMARC1 p=none adkim=r aspf=r); dkim=pass (rsa-SHA256) header.s=ms1 header.d=multibuy.org header.b="e//jsw+4"
X-SmarterMail-SpamAction: None | NoAction
X-SmarterMail-TotalSpamWeight: 1
Sabatino Traini
      Chief Information Officer
Genial s.r.l. 
Martinsicuro - Italy

5 Replies

Reply to Thread
Douglas Foster Replied
You are correct, of course.    Since the turnaround time on this request is unknown, here is my suggestion for how you can build it yourself, and hopefully do so pretty quickly.  This assumes Declude or something else that uses the Spool\Proc folder.

SmarterMail invokes Declude by creating two files in the Proc folder:  The .EML file has the message itself, the .HDR file has summary information, including the SMTP information that is not available in the message.   I am suggesting that parsing the HDR file is your path forward.

Declude is invoked before the other "Spool Filtering" or "Outbound SMTP" checks, so you will capture more than just the quarantine, but it should include everything quarantined by those two phases.   

"Inbound SMTP Blocking" tests run before Declude, but they never produce quarantine, so they are already out of scope.

Both files have the same prefix, differing only in the file extension   The format of the HDR files is officially undocumented, but it is pretty easy to reverse engineer.

After Declude processing completes, SmarterMail adds an "x-" prefix to the filename.

Here is a sanitized HDR file to explain its contents:
SmtpToUser@SmtpToDomain (multiple lines for multiple users)
retry: (count);(timestamp)
from: FromUser@FromDomain
spamcheck: (tests and results)
creationdate: (timestamp)
smarthost: SmtpFromUser@SmtpFromDomain=(Guid of outbound path)
notify: SmtpFromUser@SmtpFromDomain=
containsLocalDeliveries: (False for an incoming gateway, True for a mail store server)
connectedip: ipaddress
helo: HeloHostName
connectedhostname: (ReverseDnsHostName)
smtpSessionId: (integer)
dmarcResult: Skipped (DMARC Disabled)
authentication-results: (test results)


Your task:
(1) Write a Declude custom filter and a little code to parse this file and load the results into a SQL Database.  Include the filename as well, as it will be needed for linking to your quarantine folder.  If  you feel ambitious, parse the .EML file to extract the subject text as well.  If you want to have access to more than just the current quarantine, Declude can be configured to copy the files to a folder of your choosing.

(2) Perform SQL queries against your database information, or write a web page to provide an interface for people who don't know SQL syntax.   The other advantage of a web page is that you configure the web page to find and open the file on request.

(3) Use the SQL results or the fantasized web page to open the EML files from the Quarantine folder.

This is a simplified version of what I have built over the last few years, so I can provide assistance if  you run into obstacles.

Matt Petty Replied
Employee Post
Just to add onto Douglas' message. 
The second line is actually comma separated not line-per-recipient. 

The first 3 lines are always the same order (Status, SmtpFrom, SmptTo's)
Then the rest of the lines follow a "<name>: <data>" pattern and can be in any order.
Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
Douglas Foster Replied
Thanks Matt.  Follow-up questions:
(1) Are there any possible values for the first line other than "Written"?
(2) In the rare case of a message body with multiple From addresses, are they also comma separated?
Matt Petty Replied
Employee Post
2. The recipients/sender in the HDR come directly from the MAIL FROM and RCPT TO's in the SMTP session. So there will always been only 1 sender per the SMTP spec. From: and To: (in the MIME content) are mostly kinda visual from a delivery stand point.

1. Writing (while receiving message bytes via Smtp session), Failed (something during SMTP failed), Quarantined.
The spool will automatically remove spool messages marked as Failed but there is a moment where its still in the spool even though the delivery failed.

Matt Petty Software Developer SmarterTools Inc. (877) 357-6278 www.smartertools.com
Sabatino Replied
I don't totally agree. archive search should also work on the envelope sender without me having to write code myself. It is information that is present in archived messages.
Sabatino Traini Chief Information Officer Genial s.r.l. Martinsicuro - Italy

Reply to Thread