Messages stuck at "Spam Check"
Problem reported by Benjamin Tompkins - April 18, 2017 at 2:27 AM
Known
We have a recent rash of messages that are stuck in the spool at the "Spam Check" phase. They have to be manually removed or they will bring all mail delivery to a halt. CPU skyrockets, spool fills up, everything slows down while Smartermail tries to deal with a handful of these. The message content looks to be some spam image links, then an incomplete <style> directive with a bunch of filler to try to get past size limit scanning. They look to be delivered by a botnet so source blocking isn't feasible. Here is a sample. 
 
Return-Path: <contact@qq.com>
Received: from qq.com (server.ch-logicslegit.net [209.236.117.240]) by XXXX with SMTP;
   Tue, 18 Apr 2017 04:38:44 -0400
Received: from localhost (127.0.0.1) by qq.com id hun88c16lt0r for <XXXX>; Mon, 17 Apr 2017 20:46:51 -0400 (envelope-from <contact@qq.com>)
MIME-Version: 1.0
Precedence: Normal
From: Dan Felbern <contact@tecbile.com>
To: XXXX
Subject: Government to Ban This Natural Pain Reliever?
Content-Type: text/html
Date: Mon, 17 Apr 2017 16:07:19 -0400

<CENTER>
<CENTER><a href="hxxp://198.50.137.131//ql.html?r=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"><br><img src="hxxp://198.50.137.131//4001/tb20069green123987.jpg"></a></CENTER>
<CENTER><a href="hxxp://198.50.137.131//ql.html?o=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"><br><img src="hxxp://198.50.137.131//4001/tb20069green1_uns.jpg"><h3></a></CENTER>
<CENTER><a href="hxxp://198.50.137.131//ql.html?u=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"><img src="hxxp://198.50.137.131//cn71-1.png"></a></CENTER>
<CENTER><img src="hxxp://198.50.137.131//ql.html?i=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"width=1/> </CENTER>
</CENTER>



<style>
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Anni///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Antal///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Anticipation///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Anton///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
<snip>
 
on til it reaches 957k.
 
Any idea on how to block these?
 

11 Replies

Reply to Thread
0
Dan Slage Replied
I am having the exact same problem with the exact same messages. I had to do a EHLO block of qq.com to stop these messages. I also had to turn off the indexing service because once two of these messages get stuck in the processing queue the server slows to a crawl. 
My guess is Smartermail is having problems parsing these messages. 
The URI blacklist checks have huge delays with these messages but for unknown reasons turning off URIBL checks still does not prevent the messages from getting stuck in the "Spam Check" phase.

Any help on this would be appreciated. 
 
0
Jose Gomez Replied
We've been dealing with a backed up queue since yesterday. Is deleting these messages helping to get the queue moving for you guys?
0
Benjamin Tompkins Replied
Yes. After I got them all out everything started flowing normally. Luckily no new ones have came in since yesterday afternoon.
0
Dan Slage Replied
I am not sure. If it does help their is a delay involved. Just one of these messages will cause huge CPU usage. CPU will go down after deleting the message but only after a delay.
These messages will also get stuck in your indexing queue so it is worthwhile to delete them anyway you can.
0
Jose Gomez Replied
We woke up to 20k+ emails and zero deliverability today. ST working on the issue, currently trying to determine the root cause. Thanks for sharing this as it provides more clues as to why this might be happening.
0
Employee Replied
Employee Post
Benjamin, likely something in the content of these messages is hanging up the spam check process, and leading to a stalled spool. I've sent you a PM with the custom build download link so you can give this a try - as it should resolve those issues. This fix will also be included in our next minor release.
 
That aside, stalled spool issues involving "Spam Check" status can vary in cause, and in the behavior they present. As such, we recommend submitting a support ticket to allow us to investigate further. If its determined the issue is related to a bug, we'll refund the support ticket back to your account. Thanks for reading! Have a great day!
0
Scarab Replied
Count another that is having the same problem (we've been having it since the 14th). Posted in https://portal.smartertools.com/community/a88872/malformed-email-causes-spool-to-hang.aspx#97662 before I found this thread. I did find a lot more as to what is causing the high CPU than is listed here if you need specific details as to what Strings are causing the problem and that it is generating massive Windows Heap Dumps (.HDMP files) every time one of these emails is touched by Smartermail. 
2
Scarab Replied
If you need immediate relief we were able to map out the spamnet that was sending these malformed messages. Until Smartools gets around to fixing the issue blocking these IP Ranges in SECURITY > BLACKLIST has helped us without a single re-occurrence for the past 4 days.
 
IP Block Range
Description (Network Owner, Country)
208.110.93.56 - 208.110.93.63
Kaam Badwan (WholeSale Internet, Inc.)
192.99.131.132 - 192.99.131.135
HOTServers LLC (OVH Hosting, Inc.)
51.254.91.28 - 51.254.91.31
OVH Hosting, Inc. (UK)
209.134.22.0 - 209.134.23.255
Inter Net Bilgisayar Ltd. Sti
136.243.198.192 - 136.243.198.199
Hetzner Online GmbH (DE)
173.254.192.0 - 173.254.255.255
QuadraNet, Inc
209.236.117.239 - 209.236.117.254
Datis Media Ltd (DFW Datacenter)
195.154.252.248
Iliad Entreprises (FR)
173.45.128.0 - 173.45.159.255
Innovative Scaling Technologies
5.226.171.0 - 5.226.171.255
Foroquimica SL
64.251.27.0 - 64.251.27.255
ServerPronto (Infolink Global)
192.151.144.112 - 192.151.144.119
Ecaterina Varga (DataShack, LC)
91.211.244.0 - 91.211.247.255
VPSnet.lt (LT)
77.123.120.0 - 77.123.127.255
Galaxy Traiding Ltd (BG)
84.234.96.0 - 84.234.111.255
OBUKHOVNET (UA)
185.118.240.0 - 185.118.243.255
ITNSGLOBAL (BG)
0
Paul Blank Replied
Nice work, Scarab. I am using a 3rd party filter and have not seen this issue, but kudos!
 
1
Joe Mocerino Replied
Anyone know if there is a fix in for this, add another to the list of people having the issue.  I'll give a try to adding those blacklist.
0
Scarab Replied
This had been fixed in SM sometime last year for the malformed Base64 string that was being used causing RegEx filters to hang (not sure off-hand what sub-version of v15 but it was definitely after 15.5.6222 and pre-v16...it should be in the Release Notes...v15.7.6390 or thereabouts?). The IPs for the Blacklist probably don't pertain anymore as that was for one specific bot-net that has probably long ago rotated out of those addresses (most cycle through IPs daily or weekly with re-occurrences no less than every 90 - 180 days before they get back to those original IPs).

If you are getting 100% CPU and have messages stuck in the Spool you may want to start saving them in a subdirectory (when this occurred we had to stop the Spool, remove the malformed message, and restart the Spool) so that you can either find the common denominator or submit them to SmarterTools.

Reply to Thread