Missing Emails

Problem reported by Ng Cher Choon - 9/7/2018 at 3:18 AM

Resolved

We are running the latest SM v16.3.6821.

This morning, we received complaints from customers that emails were not sending and receiving from the email server. There are no delivery for entirely for the server between 7:55am to 9:25am when we decided to restart the server.

The server was up and running by 9:45am, after some Microsoft update. Those emails sent between 7:55am to 9:25am were missing. We checked the SMTP log and to give you an example:

[2018.09.07] 09:11:27 [209.85.221.58][60017447] rsp: 220 mail.sgcom.com.sg
[2018.09.07] 09:11:27 [209.85.221.58][60017447] connected at 7/9/2018 9:11:27 AM
[2018.09.07] 09:11:27 [209.85.221.58][60017447] cmd: EHLO mail-wr1-f58.google.com
[2018.09.07] 09:11:27 [209.85.221.58][60017447] rsp: 250-mail.sgcom.com.sg Hello [209.85.221.58]250-SIZE 40894464250-AUTH LOGIN CRAM-MD5250-8BITMIME250 OK
[2018.09.07] 09:11:27 [209.85.221.58][60017447] cmd: MAIL FROM:<chuaminglunp8+bncBDN4FMMOWAJRBPNAY7OAKGQEXCYXMPY@googlegroups.com> SIZE=859668
[2018.09.07] 09:11:27 [209.85.221.58][60017447] senderEmail(1): chuaminglunp8+bncBDN4FMMOWAJRBPNAY7OAKGQEXCYXMPY@googlegroups.com parsed using: <chuaminglunp8+bncBDN4FMMOWAJRBPNAY7OAKGQEXCYXMPY@googlegroups.com>
[2018.09.07] 09:11:27 [209.85.221.58][60017447] rsp: 250 OK <chuaminglunp8+bncBDN4FMMOWAJRBPNAY7OAKGQEXCYXMPY@googlegroups.com> Sender ok
[2018.09.07] 09:11:28 [209.85.221.58][60017447] cmd: RCPT TO:<lionel@ihub.com.sg>
[2018.09.07] 09:11:28 [209.85.221.58][60017447] rsp: 250 OK <lionel@ihub.com.sg> Recipient ok
[2018.09.07] 09:11:28 [209.85.221.58][60017447] cmd: DATA
[2018.09.07] 09:11:28 [209.85.221.58][60017447] Performing PTR host name lookup for 209.85.221.58
[2018.09.07] 09:11:28 [209.85.221.58][60017447] PTR host name for 209.85.221.58 resolved as mail-wr1-f58.google.com
[2018.09.07] 09:11:28 [209.85.221.58][60017447] rsp: 354 Start mail input; end with <CRLF>.<CRLF>
[2018.09.07] 09:11:28 [209.85.221.58][60017447] senderEmail(2): chuaminglunp8@gmail.com parsed using: Ming Lun Chua <chuaminglunp8@gmail.com>
[2018.09.07] 09:11:30 [209.85.221.58][60017447] rsp: 250 OK
[2018.09.07] 09:11:30 [209.85.221.58][60017447] Received message size: 859675 bytes
[2018.09.07] 09:11:30 [209.85.221.58][60017447] Successfully wrote to the HDR file. (d:\SmarterMail\Spool\proc\1664392092503.hdr)
[2018.09.07] 09:11:30 [209.85.221.58][60017447] Data transfer succeeded, writing mail to 1664392092503.eml (MessageID: <CAOGA3ph95Qa1HhzpkboQhFrdFkrSNytzZAuP_Js+z1mONippqw@mail.gmail.com>)
[2018.09.07] 09:11:30 [209.85.221.58][60017447] cmd: QUIT
[2018.09.07] 09:11:30 [209.85.221.58][60017447] rsp: 221 Service closing transmission channel
[2018.09.07] 09:11:30 [209.85.221.58][60017447] disconnected at 7/9/2018 9:11:30 AM

This particular email was not found in the user Inbox at all.This is only one of the case and there are many of them. Missing emails are serious problems to any email server. How can SM16 retrieve back all those emails between 7:55am to 9:45am?

THIS IS VERY FRIGHTENING!

32 Replies

Reply to Thread

Derek Curtis Replied

9/7/2018 at 7:47 AM

Employee Post

Was/is the email in the spool? That snippet shows both the .eml and .hdr files were written, so the message should have hit the spool. What do the delivery logs show for this message? Any chance it would have been caught by a spam rule or maybe a content filter? The delivery logs should give you an idea of what happened to the message.

Also, have you enabled the 22334 debug log? IF not, I'd recommend you set that up. Log in as a system admin and go to Manage -> Troubleshooting and on the Options page, add in "22334" to the text box under Debug Log IDs (one per line). That debug log is very verbose and will give a complete account of what happens to messages in the spool.

Derek Curtis COO SmarterTools Inc. www.smartertools.com

Manuel Martins Replied

9/7/2018 at 8:42 AM

Hi,

It has happened a situation like that on our server some weeks ago, after the restart the server was OK, but we missed emails from 1 a.m. to 9 a.m. (time the server was restarted), we never could find those missed emails, neither understood what "on earth" happened...

Ng Cher Choon Replied

9/7/2018 at 8:59 AM

After adding in "22334" to the textbox under the Debug Log IDs (one per line), how do we view the debug log? Can you provide a step-by-step instruction on how to view it?

Derek Curtis Replied

9/7/2018 at 9:29 AM

Employee Post

It will be in your default logs location (e.g., C:\SmarterMail\Logs) and be formatted like DebugLog-22334.log. You can then open it in a text editor and look for specific messages. That said, it will just capture data from when you activated the log, so moving forward it should start capturing data.

Derek Curtis COO SmarterTools Inc. www.smartertools.com

Paul Blank Replied

9/7/2018 at 12:30 PM

My two cents' worth:

If your version of SM has archiving (Enterprise or Free), and it's not activated, you should activate it to capture all emails, coming and going.

If you do have archiving running, check the archive for those missing emails.

ST, if this customer has SM Pro, perhaps you will at least temporarily upgrade them to Enterprise to see if the missing messages make it to the archive, the next time this happens.

BTW if this user has other email clients besides webmail, you might at least suspect those. I have seen smartphone clients and occasional desktop clients "lose" emails, and yes, even delete them from the server, although this is admittedly rare. In those cases, a functioning SM archive will definitely have the missing messages - given that there's no malicious mischief.

Ng Cher Choon Replied

9/7/2018 at 5:54 PM

We are running on Enterprise version. But for those who are on the Pro version, your suggestion will be good for ST to upgrade them to Enterprise version.

If we do the archiving of all messages, the storage will grow very large. Is there a way to auto-clean after some time?

The above example is only one case. It affects all users across all domains. We did a check on the logs with many users's inbox folders. It is unlikely all messages are deleted by users. There are no emails on any of the users within the time period. We also check on the Delivery logs, The delivery stops at 08:05am as shown here:

07:57:14 [92138] Delivery started for brownd07@credit-adjustments.com at 7:57:14 AM

07:57:23 [92138] Added to SpamCheckQueue (1 queued; 0/30 processing)

07:57:23 [92138] This message has been quarantined because a virus was found.

07:57:23 [92138] Removed from SpamCheckQueue (0 queued or processing)

07:57:26 [92138] .eml file not found. Removing .hdr file.

07:57:26 [92138] Delivery finished for brownd07@credit-adjustments.com at 7:57:26 AM [id:1664392092138]

08:05:14 [92109] Added to RemoteDeliveryQueue (1 queued; 0/50 processing)

08:05:14 [92109] Sending remote mail for hafiedah@bsts.org

08:05:14 [92109] Spam check results: [_REVERSEDNSLOOKUP: Failed], [BACKSCATTER: passed]

08:05:14 [92109] Bounce email written to 1664392092180.eml

08:05:14 [92109] Delivery for hafiedah@bsts.org to hamzah@steelhwak.com.my has completed (Bounced)

08:05:14 [92109] Removed from RemoteDeliveryQueue (0 queued or processing)

08:05:17 [92180] Delivery started for at 8:05:17 AM

08:05:17 [92180] Spool message was missing Return-Path; Also missing FROM header

08:05:17 [92109] Removing Spool message: Killed: False, Failed: False, Finished: True

08:05:17 [92109] Delivery finished for hafiedah@bsts.org at 8:05:17 AM [id:1664392092109]

08:05:20 [92180] Added to SpamCheckQueue (1 queued; 0/30 processing)

08:05:20 [92180] Launching 'D:\Program Files (x86)\Plesk\admin\bin\smmailfilter.exe' command line exe.

08:05:20 [92180] Command line exe finished.

08:05:20 [92180] Starting Spam Checks.

08:05:20 [92180] Skipping spam checks: Internally Generated Message

08:05:20 [92180] Spam Checks completed.

08:05:20 [92180] Removed from SpamCheckQueue (0 queued or processing)

08:05:23 [92180] Added to LocalDeliveryQueue (1 queued; 0/50 processing)

08:05:23 [92180] Starting local delivery to hafiedah@bsts.org

08:05:23 [92180] Delivery for to hafiedah@bsts.org has completed (Forwarded Delivered) Filter: None

08:05:23 [92180] End delivery to hafiedah@bsts.org (MessageID: <636719043147883937@mail.sgcom.com.sg>)

08:05:23 [92180] Removed from LocalDeliveryQueue (0 queued or processing)

08:05:26 [92180] Added to LocalDeliveryQueue (1 queued; 0/50 processing)

08:05:26 [92180] Starting local delivery to daniel@bsts.org

08:05:26 [92180] Delivery for to daniel@bsts.org has completed (Delivered) Filter: None

08:05:26 [92180] End delivery to daniel@bsts.org (MessageID: <636719043147883937@mail.sgcom.com.sg>)

08:05:26 [92180] Removed from LocalDeliveryQueue (0 queued or processing)

08:05:29 [92180] Removing Spool message: Killed: False, Failed: False, Finished: True

08:05:29 [92180] Delivery finished for at 8:05:29 AM [id:1664392092180]

09:36:27 Relay server stopped at 7/9/2018 9:36:27 AM

09:39:31 Delivery server started at 7/9/2018 9:39:31 AM

09:39:31 Updating ClamAV database...

09:40:10 Updating the ClamAV database has completed successfully

09:43:40 [66033] Delivery started for tj.tan@iso-team.com at 9:43:40 AM

09:43:40 [66032] Delivery started for maybank.equities@maybank-ib.com at 9:43:40 AM

09:43:43 [66032] Added to SpamCheckQueue (1 queued; 0/30 processing)

09:43:44 [66032] Launching 'D:\Program Files (x86)\Plesk\admin\bin\smmailfilter.exe' command line exe.

09:43:45 [66032] Command line exe finished.

Our point is this: Emails should never be lost by our email server. We have used MailEnable for 12 years and we have never had any missing emails under any circumstances. There should be never in any case of rarity that emails are ever lost as any of those emails may contain important and legal information to the users. One email lost is just too many.

We have just switched over to Smartermail for barely 1 month. We have faced many critical errors. One is the formatting error and the other are the TCP errors. And now we are facing the missing mails problem, we find that these are not acceptable bugs. All of us will lose customers due to the SM instability program. The customer can TOLERATE us for a while, but not all the times. If we lose customers, so will ST. In additional, This may involve legal action as well for all parties.

ST's reputation is now going down the drain, as we have been religiously following the community and reading the feedback of the users. If ST doesn't buck up, potential buyers will shun away from buying SM.

Therefore, I suggest that ST needs to look into its stability issues before looking at any of added features first. I guess their Quality Control must enhanced further. No customer will accept the reason that different users has different situation if it is a stability issues.

Paul Blank Replied

9/7/2018 at 6:46 PM

Manuel, which version of SM, please?

Paul Blank Replied

9/7/2018 at 6:52 PM

I don't think the archive has auto-clean, but I think you can simply delete older emails from the archive folder and then stop and restart the Windows SM service.

That process can probably be automated as well.

Someone kindly set me straight if I'm incorrect here

Ng Cher Choon Replied

9/7/2018 at 8:00 PM

We suspect we may find the cause of the above-mentioned problem.

If we login in as System Admin, we did a test as follows:

- if we click onto Setting->Antispam->Spam Check, the moment Declude is enabled for spooling, all messages cannot be sent and received. Thus all the emails went missing.

- We disable that option again, all emails can be received and sent.

Can the ST please verified this problem?

Thanks

Paul Blank Replied

9/7/2018 at 9:46 PM

Interesting. Do you actually have Declude instead? (edit: I meant "installed," not "instead.")

Ng Cher Choon Replied

9/7/2018 at 9:51 PM

Sorry, do you mean that do we have Declude enabled? If that is the question, we have disabled that option. Isn't Declude part of SM16?

Even, if it isn't included, mails should never go missing.

Paul Blank Replied

9/8/2018 at 2:53 AM

I don't run v16; I didn't know if/that Declude was included. But I agree, of course; emails should not go missing, should all be logged, and should be in the archive, if it's enabled.

Ng Cher Choon Replied

9/8/2018 at 6:00 PM

Are the ST looking into the Declude bug?

echoDreamz Replied

9/8/2018 at 9:02 PM

I'd love to hear what legal action ST will face because of email delivery issues...

echoDreamz Replied

9/8/2018 at 9:09 PM

Declude is not and as far as I know has never been part of SmarteMail. So this sounds more like a poor admin not doing his job than a glitch with SM, just going around flipping options without knowing what they do. Though, SmarterMail should verify that Declude is installed / available before allowing you to enable it.

When you enable Declude, if I remember correctly, this causes SM to leave emails in the proc folder for declude to pickup and process. You told SM to use Declude without Declude being installed and configured, so the emails just pile up waiting for declude to grab them.

The emails shouldnt be missing, they are just sitting waiting for declude to process, my bet is that you have a ton of emails sitting in the "proc" directory.

When you disable declude, it doesnt move the emails that were waiting for declude, you have to manually handle them.

I believe it was sometime in 2014, Declude was bought by Mails Best Friend, which you can find here http://mailsbestfriend.com/downloads/

Ng Cher Choon Replied

9/8/2018 at 9:22 PM

Thanks for the information. Will check out the proc directory. But this is very frightening as there is no documents on this Declude and how they work in Smartermail. I think ST needs to explain things on this matter in greater detail.

Paul Blank Replied

9/8/2018 at 10:37 PM

Thanks for the response, Christopher.

It would be nice, however, if the SM logs would show where these messages went, especially if they were redirected by a setting within SM.

Again, kindly correct me if I'm wrong.

Paul Blank Replied

9/8/2018 at 10:38 PM

Even if Declude is not installed, and the box for Declude is checked, SM should log it somewhere, and certainly should preserve the emails if it can't deliver them somewhere.

I'm not clear about what's actually happening in this event.

Ng Cher Choon Replied

9/8/2018 at 11:24 PM

I second that as well

echoDreamz Replied

9/9/2018 at 9:01 AM

Been a while since we used declude, so I cannot remember how the logging behaves.

Matthew Leyda Replied

9/9/2018 at 9:30 AM

It creates a log file "DECxxx.log". In the declude directory the log file location is specified in the global.cnf file.

Kendra Support http://www.kendra.com support@kendra.com 425-397-7911 Junk Email filtered ISP

echoDreamz Replied

9/9/2018 at 10:25 AM

Well, they did not have declude installed, so there is no way for it to create the log files :)

echoDreamz Replied

9/9/2018 at 6:21 PM

The issue is that SM drops them into the proc folder, expecting another app to grab them and process them, them move them to the "drop" directory for processing. That is what the proc folder is for.

SM should do a reg check or at least a path check to see if declude is installed and then disable / enable allowing you to enable it.

Paul Blank Replied

9/9/2018 at 9:46 PM

But the email shouldn't just vanish. I'm still unclear about if it's in any folder, and why it's not noted in the logs.

Linda Pagillo Replied

9/10/2018 at 1:43 PM

Declude is NOT a part of SmarterMail. It is a program you can use which easily integrates into SM. When it is installed, messages coming into the server will go to Smartermail\Spool\Proc when they first come in and then they are moved to Smartermail\Spool\Proc\Work to be scanned by Declude then handed back to SM for delivery to the inbox or spam folder. If you do not have Declude installed and you check the box in SM and try to save it, SM should tell you that it can't enable it because Declude is not running on your server. I am very surprised that it let you enable it if you are not using Declude.

Linda Pagillo Mail's Best Friend Email: linda.pagillo@mailsbestfriend.com Web: www.mailsbestfriend.com Authorized SmarterTools Reseller Authorized Message Sniffer Reseller

echoDreamz Replied

9/10/2018 at 2:05 PM

Linda, ditto. I figured SM would look for the registry keys / normal install path for declude and check for it.

Or at least say hey... emails have been sitting in the proc directory for over x hours, something is wrong, please check... To at least alert you to something not processing emails smoothly.

We have PRTG set to monitor our proc / drop directories and alert us when they file count grows too high.

Employee Replied

9/10/2018 at 3:25 PM

Employee Post

Thank you for bringing this issue to our attention. There are two scenarios in which mail will be placed in a proc folder instead of the normal spool folder: either Declude is enabled (Sys admin > Settings > Antispam > Spam Checks) or "Enable spool proc folder" (Sys admin > Settings > Antispam > Options) is enabled. If you accidentally, or even on purpose, enable either Declude or "Enable spool proc folder", emails DO NOT go missing. The messages will go to the proc folder and await a third-party app to process them. It is the responsibility of the third-party app to then move the messages into the spool folder. If nothing is installed or there is NO third-party app, you can manually move any messages from the proc folder over to the spool to resume normal delivery. Please remember to turn either, or both, off.

That said, when looking at this issue we decided to make a change to the Declude Antispam option: we will simply disable the Declude option if SM detects that it is not running.

As an aside, under the Options tab, if a system admin enables the "Enable spool proc folder" option, there is already a yellow warning banner that states "When configuring 'Enable spool proc folder', a 3rd party application must move messages into the spool after processing."

I hope this clears up some frustrations or misunderstandings.

Ng Cher Choon, can you verify whether or not your "missing" messages are in the proc folder?

Paul Blank Replied

9/10/2018 at 3:37 PM

And perhaps any redirection of messages should be noted in the SM logs.

Ng Cher Choon Replied

9/10/2018 at 4:26 PM

Dear Robert Emmett,

Thank you for your update. The approach that you mentioned should be the way to go because as a new system administrator of SM, we do not know what Declude is. In additional, SM documentation does not mention much about the Declude and what the mechanism is being run when Declude is enabled. At least, there was a warning about 'Enable spool proc folder' option.

We did see all the emails being moved to the Proc folder and we move them back to the SubSpool folder. The mails eventually were delivered after we restart the SM services.

Thanks

Ng Cher Choon

Paul Blank Replied

9/10/2018 at 4:50 PM

Thanks for the follow-up, Ng Cher Choon. Nice to hear that the emails were not totally lost!

echoDreamz Replied

9/13/2018 at 1:45 PM

This has been fixed in the latest SM 16 release https://www.smartertools.com/smartermail/release-notes/16

Employee Replied

9/27/2018 at 10:57 AM

Employee Post Marked As Resolution

In SmarterMail 16.3.6830 we addressed this issue:

Changed: If Declude is detected as either not installed or not running, the Spam Checks grid will disable Declude settings and operations to prevent messages from moving to proc folder for Declude processing.

Back to Community Threads

Please leave this box unchecked