Messages stuck at "Spam Check"

Problem reported by Benjamin Tompkins - 4/18/2017 at 2:27 AM

Resolved

We have a recent rash of messages that are stuck in the spool at the "Spam Check" phase. They have to be manually removed or they will bring all mail delivery to a halt. CPU skyrockets, spool fills up, everything slows down while Smartermail tries to deal with a handful of these. The message content looks to be some spam image links, then an incomplete <style> directive with a bunch of filler to try to get past size limit scanning. They look to be delivered by a botnet so source blocking isn't feasible. Here is a sample.

MIME-Version: 1.0
Precedence: Normal
From: Dan Felbern <contact@tecbile.com>
To: XXXX
Subject: Government to Ban This Natural Pain Reliever?
Content-Type: text/html
Date: Mon, 17 Apr 2017 16:07:19 -0400

<CENTER>
<CENTER><a href="hxxp://198.50.137.131//ql.html?r=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"><br><img src="hxxp://198.50.137.131//4001/tb20069green123987.jpg"></a></CENTER>
<CENTER><a href="hxxp://198.50.137.131//ql.html?o=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"><br><img src="hxxp://198.50.137.131//4001/tb20069green1_uns.jpg"><h3></a></CENTER>
<CENTER><a href="hxxp://198.50.137.131//ql.html?u=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"><img src="hxxp://198.50.137.131//cn71-1.png"></a></CENTER>
<CENTER><img src="hxxp://198.50.137.131//ql.html?i=clk01*dtcfgdvbvbvcb=oth.2odu.1m8vab4.4hc6e.c0bld__3fgu2qVfn/00034n"width=1/> </CENTER>
</CENTER>



<style>
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Anni///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Antal///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Anticipation///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////
||<<<!//!<<||{}{&^$$#{&*}{{}{*&&$^^^^%$#{}{&*&$$#&*||<<<!//!///////////Anton///////////&*#(((((*#$!~~!!~~!!#&))))||!!!!***)))))***||<<<!//!<<||//?||$*$$!!///&^$$////

<snip>

on til it reaches 957k.

Any idea on how to block these?

11 Replies

Reply to Thread

Dan Slage Replied

4/18/2017 at 7:09 AM

I am having the exact same problem with the exact same messages. I had to do a EHLO block of qq.com to stop these messages. I also had to turn off the indexing service because once two of these messages get stuck in the processing queue the server slows to a crawl.
My guess is Smartermail is having problems parsing these messages.
The URI blacklist checks have huge delays with these messages but for unknown reasons turning off URIBL checks still does not prevent the messages from getting stuck in the "Spam Check" phase.

Any help on this would be appreciated.

Jose Gomez Replied

4/18/2017 at 7:33 AM

We've been dealing with a backed up queue since yesterday. Is deleting these messages helping to get the queue moving for you guys?

Benjamin Tompkins Replied

4/18/2017 at 8:17 AM

Yes. After I got them all out everything started flowing normally. Luckily no new ones have came in since yesterday afternoon.

Dan Slage Replied

4/18/2017 at 8:18 AM

I am not sure. If it does help their is a delay involved. Just one of these messages will cause huge CPU usage. CPU will go down after deleting the message but only after a delay.
These messages will also get stuck in your indexing queue so it is worthwhile to delete them anyway you can.

Jose Gomez Replied

4/18/2017 at 8:33 AM

We woke up to 20k+ emails and zero deliverability today. ST working on the issue, currently trying to determine the root cause. Thanks for sharing this as it provides more clues as to why this might be happening.

Employee Replied

4/18/2017 at 1:32 PM

Employee Post

Benjamin, likely something in the content of these messages is hanging up the spam check process, and leading to a stalled spool. I've sent you a PM with the custom build download link so you can give this a try - as it should resolve those issues. This fix will also be included in our next minor release.

That aside, stalled spool issues involving "Spam Check" status can vary in cause, and in the behavior they present. As such, we recommend submitting a support ticket to allow us to investigate further. If its determined the issue is related to a bug, we'll refund the support ticket back to your account. Thanks for reading! Have a great day!

Scarab Replied

4/21/2017 at 4:20 PM

Count another that is having the same problem (we've been having it since the 14th). Posted in https://portal.smartertools.com/community/a88872/malformed-email-causes-spool-to-hang.aspx#97662 before I found this thread. I did find a lot more as to what is causing the high CPU than is listed here if you need specific details as to what Strings are causing the problem and that it is generating massive Windows Heap Dumps (.HDMP files) every time one of these emails is touched by Smartermail.

Scarab Replied

4/24/2017 at 2:05 PM

If you need immediate relief we were able to map out the spamnet that was sending these malformed messages. Until Smartools gets around to fixing the issue blocking these IP Ranges in SECURITY > BLACKLIST has helped us without a single re-occurrence for the past 4 days.

IP Block Range	Description (Network Owner, Country)
208.110.93.56 - 208.110.93.63	Kaam Badwan (WholeSale Internet, Inc.)
192.99.131.132 - 192.99.131.135	HOTServers LLC (OVH Hosting, Inc.)
51.254.91.28 - 51.254.91.31	OVH Hosting, Inc. (UK)
209.134.22.0 - 209.134.23.255	Inter Net Bilgisayar Ltd. Sti
136.243.198.192 - 136.243.198.199	Hetzner Online GmbH (DE)
173.254.192.0 - 173.254.255.255	QuadraNet, Inc
209.236.117.239 - 209.236.117.254	Datis Media Ltd (DFW Datacenter)
195.154.252.248	Iliad Entreprises (FR)
173.45.128.0 - 173.45.159.255	Innovative Scaling Technologies
5.226.171.0 - 5.226.171.255	Foroquimica SL
64.251.27.0 - 64.251.27.255	ServerPronto (Infolink Global)
192.151.144.112 - 192.151.144.119	Ecaterina Varga (DataShack, LC)
91.211.244.0 - 91.211.247.255	VPSnet.lt (LT)
77.123.120.0 - 77.123.127.255	Galaxy Traiding Ltd (BG)
84.234.96.0 - 84.234.111.255	OBUKHOVNET (UA)
185.118.240.0 - 185.118.243.255	ITNSGLOBAL (BG)

Paul Blank Replied

4/24/2017 at 9:53 PM

Nice work, Scarab. I am using a 3rd party filter and have not seen this issue, but kudos!

Joe Mocerino Replied

4/24/2018 at 8:49 AM

Anyone know if there is a fix in for this, add another to the list of people having the issue. I'll give a try to adding those blacklist.

Scarab Replied

4/24/2018 at 11:33 AM

Marked As Resolution

This had been fixed in SM sometime last year for the malformed Base64 string that was being used causing RegEx filters to hang (not sure off-hand what sub-version of v15 but it was definitely after 15.5.6222 and pre-v16...it should be in the Release Notes...v15.7.6390 or thereabouts?). The IPs for the Blacklist probably don't pertain anymore as that was for one specific bot-net that has probably long ago rotated out of those addresses (most cycle through IPs daily or weekly with re-occurrences no less than every 90 - 180 days before they get back to those original IPs).

If you are getting 100% CPU and have messages stuck in the Spool you may want to start saving them in a subdirectory (when this occurred we had to stop the Spool, remove the malformed message, and restart the Spool) so that you can either find the common denominator or submit them to SmarterTools.

Back to Community Threads

Please leave this box unchecked

11 Replies

Reply to Thread

Tags

Related Knowledge Base Articles