SmarterMail 14.4.5784 SpamClient.exe at 100% CPU Mail stuck in queue
Problem reported by Bruce - 11/5/2015 at 3:51 AM
Known
After upgrading to SmarterMail 14.4.5784 are now getting problem with the new SpamClient.exe used by Cyren.
 
This new SpamClient.exe is using 100% CPU usage.
 
The other issue is that some emails are now getting stuck in the spam check for over 20 hours and there is no way to get them to be processed by SmarterMail without restarting the server.
 
Also having issues with large bandwidth usage from IMAP and IMAP folders not syncing.
 
Lots of issues with SmarterMail 14.4.5784 is there a way to roll back?

29 Replies

Reply to Thread
1
Christian Schmit Replied
We have the same issue. You can simply kill the Spamclient.exe process in the task manager to solve the problem, Smartermail itself does not need to be restarted. Spamclient.exe will restart itself automatically afterwards.
 
0
Bruce Replied
Tried that but it starts up again in a few minutes at 100% and does not clear emails stuck in the queue marked as 'spam check'.

The only way to fix the issue is to restart the server to get the Spamclient.exe running at low CPU usage again and clear the queue.

SmarterTools appear to have removed this new version 14.4.5784 from their download page so looks like there are other issues with this new release.

Have now installed the previous 14.3.5752 version as the server has over 11,000 users need to ensure that it is running a stable release.
1
Bruce Replied
Also had a 3rd issue with 14.3.5752 that had not seen before whereby emails would get stuck in the spool and would be partially delivered to customers inboxes.
 
We had customers receiving the same email 1,400 times per hour and the only way to stop it was to delete the emails from the mail spool.
1
Matt Petty Replied
Employee Post
Thank you for reporting this, I'm going to start looking into this.
 
Might I ask what kind of load is on your server? Users / Inbound Emails per hour
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Bruce Replied
It is still very early in the morning here so stats for the last hour would be low, but stats for the last 24 hours are;

Users: 11,021
Inbound email; 51,304
Local deliveries; 32,541
Remote Deliveries; 18,760
Blocked as Spam 39,830
0
Bruce Replied
Additional info on this loop issue;

When it occurs you only see one email in the spool for the email which is being delivered multiple times and the delivery log is full of the following;

12:52:33 [39616] Exception: rawText does not contain a valid rfc2822 header field
12:52:36 [39616] Starting local delivery to xxx@xxxxxxxx.co.uk
12:52:36 [39616] Exception: rawText does not contain a valid rfc2822 header field
12:52:39 [39616] Starting local delivery to xxx@xxxxxxxx.co.uk
12:52:39 [39616] Exception: rawText does not contain a valid rfc2822 header field
12:52:43 [39616] Starting local delivery to xxx@xxxxxxxx.co.uk
12:52:43 [39616] Exception: rawText does not contain a valid rfc2822 header field
12:52:46 [39616] Starting local delivery to xxx@xxxxxxxx.co.uk
12:52:46 [39616] Exception: rawText does not contain a valid rfc2822 header field
12:52:50 [39616] Starting local delivery to xxx@xxxxxxxx.co.uk
12:52:50 [39616] Exception: rawText does not contain a valid rfc2822 header field
12:52:53 [39616] Starting local delivery to xxx@xxxxxxxx.co.uk
0
Bruce Replied
I have also added above additional info above on the email delivery loop that occurred. It happened to 3 customers who received emails from the same company.
1
Chris Daley Replied

We had 30+ phone calls this morning from users who had received no email on 14.4.5787, the spool was showing only a handful of messages however on looking at the spool folder there was 12,000+ files so 6,000+ emails not delivered. Delivery log was nearly empty. Had to run the installer a couple of times over the top (which failed) however it appears to have corrected the issue for now.

 

Did anyone at ST test the new version before release?

 

I know I said in my other thread the spool was poorly designed however I wasn't exactly expecting for it to be broken in the latest release!

 

1
Bruce Replied
Have tried version 14.4.5787 which worked for about 7 hours before 1,000's of emails got stuck in the spool marked as 'spam check'.
 
Terminating the MailService_SubProcess did not cause a new MailService_SubProcess to start and had to restart the SmarterMail service in order to get the spool queue cleared.
 
The queue started to clear after restating the SmarterMail service then it looks like MailService_SubProcess crashed and disappeared from the task manager and did not restart.
 
With 1,000's of emails building up in the mail spool had no choice but to go back to the last stable release 4.3.
0
Christian Schmit Replied
We had the same problem this morning with 14.4.5787. We had to restart the smartermail service to clear the issue. For now I have disabled Cyren as we still have messagesniffer for spam filtering.
0
Paul Blank Replied
Another argument in favor of using a cloud-based email filtering service. Add that to the fact that doing that keeps the majority of spam and malware - and its attendant traffic - from ever crossing your "border".
0
Matt Petty Replied
Employee Post
If your having these issues and want to help.
1. Open CMD Prompt
2. Type or copy "C:\Program Files (x86)\SmarterTools\SmarterMail\Service\MailService_SubProcess.exe debug" and hit enter
 
in that window that pops up can you type. "loggingon" and then "init".
Once any issue occurs can you send sales@smartertools.com the  "C:\Program Files (x86)\SmarterTools\SmarterMail\Service\SP_log.txt" file.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Lucidio Arruda Neto Replied
I had the exactly same issue with one customer and 14.4.5784. Needed to restart SM Service to get over. We have an open ticket with Rod Lasky.

[2015.11.04] 11:24:41 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:24:44 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:24:44 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:24:47 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:24:47 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:24:50 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:24:50 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:24:53 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:24:53 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:24:56 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:24:56 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:24:59 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:24:59 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:25:02 [21547] Starting local delivery to **********@**************
[2015.11.04] 11:25:02 [21547] Exception: rawText does not contain a valid rfc2822 header field
[2015.11.04] 11:25:05 [21547] Starting local delivery to **********@**************

Running with very low load (24 hours):
Users: 45
Inbound email; 2165
Remote Deliveries; 81
Spam: 1838
0
Bruce Replied
Unfortunately as it is a production mail server service for over 11,000 users have had to roll back to 14.3 until all the service affecting bugs in 14.4 can be fixed as each of these issues reported are causing outages and delays for mail services.
0
David Sovereen Replied
We're seeing similar problems. My MailService_SubProcess.exe was using all available CPU. I ended the process and it did not restart. When I manually run it in a Command Prompt, I get the following:

Received request to scan a message...
Scanning Message: 31231
Received request to scan a message...
Received request to scan a message...
Scanning Message: 31223
Received request to scan a message...
Scanning Message: 31225
Received request to scan a message...
Scanning Message: 31226
Received request to scan a message...
Scanning Message: 31232
Received request to scan a message...
Scanning Message: 31220
Received request to scan a message...
Scanning Message: 31219
Received request to scan a message...
Scanning Message: 31218
Received request to scan a message...
Scanning Message: 31236
Received request to scan a message...
Scanning Message: 31229
Received request to scan a message...
Scanning Message: 31227
Scanning Message: 31224
Received request to scan a message...
Scanning Message: 31228
Received request to scan a message...
Scanning Message: 31233
Received request to scan a message...
Scanning Message: 31230
Received request to scan a message...
Scanning Message: 31222
Received request to scan a message...
Scanning Message: 31235
Sending SmarterMail error code: e10
Error Message: 31231~%!#~Exception while processing message: 31231, Exception: C
ommtouch license details are not set
Sending SmarterMail error code: e10
Error Message: 31219~%!#~Exception while processing message: 31219, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31220~%!#~Exception while processing message: 31220, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31218~%!#~Exception while processing message: 31218, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31227~%!#~Exception while processing message: 31227, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31223~%!#~Exception while processing message: 31223, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31225~%!#~Exception while processing message: 31225, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31228~%!#~Exception while processing message: 31228, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31235~%!#~Exception while processing message: 31235, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31236~%!#~Exception while processing message: 31236, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31232~%!#~Exception while processing message: 31232, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31230~%!#~Exception while processing message: 31230, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31226~%!#~Exception while processing message: 31226, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31233~%!#~Exception while processing message: 31233, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31222~%!#~Exception while processing message: 31222, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31224~%!#~Exception while processing message: 31224, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31229~%!#~Exception while processing message: 31229, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 31234
Sending SmarterMail error code: e10
Error Message: 31234~%!#~Exception while processing message: 31234, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 31241
Received request to scan a message...
Scanning Message: 31243
Received request to scan a message...
Scanning Message: 31244
Received request to scan a message...
Scanning Message: 31238
Sending SmarterMail error code: e10
Error Message: 31243~%!#~Exception while processing message: 31243, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31238~%!#~Exception while processing message: 31238, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31244~%!#~Exception while processing message: 31244, Exception: O
bject reference not set to an instance of an object.
Sending SmarterMail error code: e10
Error Message: 31241~%!#~Exception while processing message: 31241, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 31242
Sending SmarterMail error code: e10
Error Message: 31242~%!#~Exception while processing message: 31242, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 31239
Sending SmarterMail error code: e10
Error Message: 31239~%!#~Exception while processing message: 31239,

..etc, etc.

I have disabled the Cyren AntiSpam for the time being (we have Message Sniffer and others still working) until a fix is issued. Hopefully that is soon. I really don't want to restart MailService.exe or downgrade if I can avoid it.

Dave
0
David Sovereen Replied
I see that it says Commtouch license details are not set. We have it licensed. Under license details, it says:

Feature Cyren Premium Antispam
Status Active Subscription
Expiration 12/22/2016
Limits 4397 of Unlimited mailboxes
Trial Available No
0
David Sovereen Replied
I upgraded to the Custom Build posted overnight.  The upgrade was smooth and MailService.exe and MailService_SubProcess.exe was running for a while.  When I checked on the server this afternoon, the CPU was super high.  MailService.exe was using everything and MailService_SubProcess.exe was not running.
 
The spool was backed up with about 1500 messages.
 
I disabled Cyren Anti-spam checking and the CPU gradually went down and the spool cleared through.
 
I then ran MailService_SubProcess.exe in a Command Prompt and re-enabled Cyren Anti-spam checking.  It outputted the following:
 
C:\Program Files (x86)\SmarterTools\SmarterMail\Service>MailService_SubProcess.e
xe
Received request to scan a message...
Scanning Message: 54976
Sending SmarterMail error code: e10
Error Message: 54976~%!#~Exception while processing message: 54976, Exception: C
ommtouch license details are not set
Received request to scan a message...
Scanning Message: 54974
Sending SmarterMail error code: e10
Error Message: 54974~%!#~Exception while processing message: 54974, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 54975
Sending SmarterMail error code: e10
Error Message: 54975~%!#~Exception while processing message: 54975, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 54977
Sending SmarterMail error code: e10
Error Message: 54977~%!#~Exception while processing message: 54977, Exception: O
bject reference not set to an instance of an object.
Received request to scan a message...
Scanning Message: 54978
Sending SmarterMail error code: e10
Error Message: 54978~%!#~Exception while processing message: 54978, Exception: O
bject reference not set to an instance of an object.
 
I have re-disabled the Cyren Anti-spam checking until this issue can be resolved.
 
Dave
0
Matt Petty Replied
Employee Post
When you run Cyren through command line make sure you do it as an admin, and that you type "init" and hit enter also make sure you type "MailService_SubProcess.exe debug" when booting it up.
That issue you posted is because you ran it without saying init.
Could you send me the SP_log.txt in your Service folder?

You can send it to sales@smartertools.com and mention that you want the logs to go to me.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Matt Petty Replied
Employee Post
Thanks for the logs, in a reply you should see a new custom build linked which fixed some issues with Cyren. I appears the issues you had seemed to be related to Cyren not being able to make a connection to their servers.
This new build will fix issues with Cyren restarting and being pummeled with message process requests before it's even fully initialized.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Jeffrey Lemire Replied
We're running into the same issue. Processor cycles maxed out when Cyren is enable and drops off when it's not. Just upgraded to 14.4.5787.

Is the custom build a solution to this issue and if so, is it possible to provide this?

TIA :-)
0
David Sovereen Replied
The custom build that Matt sent me the link to privately definitely helped. No crashes or CPU issues to report yet after a couple days. Hopefully they post the link here in the forum for others to try who are having similar problems.
0
Matt Petty Replied
Employee Post
(11-13-2015) Custom Build
http://www.smartertools.com/downloads/SmarterMail/CustomBuilds/14.4.5795.18448/SmarterMail14_Setup.exe
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
David Sovereen Replied
Hi Matt, this is even newer than the build link you sent to me. Should I upgrade as well as a precautionary measure, or hold off until a new full release is made? Thx
0
Matt Petty Replied
Employee Post
I would suggest being on the latest. If you are able to.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
1
kevind Replied
Hello, just wondering when we can expect a public release to resolve this issue. Hesitate to install custom builds as there are no release notes to see what other changes might affect users.
 
FWIW, nice job splitting off Cyren. I suggest you call this new service MailService_Cyren instead of MailService_SubProcess. This would open the door to splitting off other processes:
  • MailService_SpamAssassin
  • MailService_SMTP
  • MailService_IMAP
  • MailService_Web
Which would make SM more powerful and easier to manage. You could start/stop services as needed (e.g. when the Spool is backed up) and open Task Manager to see how much RAM and CPU each process is using (instead of just seeing MailService at 98%).
 
Thanks,
Kevin
0
Matt Petty Replied
Employee Post
We have a minor today that adds in a lot of fixes to the Cyren process. I will post an update in this thread when we release it.
I do believe it is on the map to add Message Sniffer into this same SubProcess.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
kevind Replied
OK, thanks for quick reply.
Suggest 2 processes: MailService_Cyren and MailService_MsgSniffer so you can see bad behavior (high CPU/RAM) in Task Manager and stop/restart it.
0
Matt Petty Replied
Employee Post
That is a good idea, I'm sure we will come to a decision once we start working on it. Keeping things apart from each other is nice for when you are trying to debug or when you need to, like you said, restart or stop a part.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Matt Petty Replied
Employee Post
The new minor has been released. This contains many fixes for issues that Cyren was having.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com

Reply to Thread