SmarterMail Bayesian Filter Flags 99% of All Messages, Even Ham Messages.
Problem reported by Colton Morrison - February 28 at 7:51 AM
Resolved
We've upgraded our SmarterMail server to the latest public version and now Bayesian filtering is wayyyyy too aggressive. Some customers use the webmail client and some use their own devices (POP/IMAP). I've been forced to basically undermine the whole Bayesian filter system because it would send 99% of messages to the spam filter. This is based on rules which were working well before we upgraded from v15.4.6151. What do I do to fix this? Please help!
 
About SmarterMail:
  • SmarterMail Enterprise Edition
  • Version 15.7.6614
  •  

39 Replies

Reply to Thread
0
Scarab Replied
If you have a backup of your Bayesian filter dat files from 15.4.6151 I would try restoring those and restarting SM.
 
You can find these files in \Smartermail\Service as:
 
    bayes.dat
    bayes_default.dat
    bayes1.dat
    bayes2.dat
 
If that still doesn't resolve the issue you may have to disable Bayesian Filters until you get a response from SmarterTools themselves.
1
Hemen Shah Replied
Have you noticed anything specific to gmail, wherein more number of mails to gmail users are getting delivered to spam folder, post last couple of SM 15.x updated i have been noticing that many customers are complaining about mails getting delivered to spam folder and more of them are gmail recipients, emails with just test word and clean smtp IP even getting delivered to spam.
0
Hemen Shah Replied
Hi, Have you raised this to SM team and if so then any response received,
off lately after last couple of 15.x fixes i too am getting complaints to legit mails getting delivered to user spam box and they are as good as just test word and even from clean IPs
1
Hemen Shah Replied
Are we getting any update on this from SM Team on this, it seems to be some scoring issue with Bayesian filtering, emails with simple text is getting scores 10 by bayesian and this is post last two SM 15.x last 2 builds.
 
Thanks
0
Colton Morrison Replied
I've opened a ticket about this bug today since the SM Team has not responded to this thread. Usually I've seen pretty quick response on these forums, but not this time...

I will let you know what they say.
0
Hemen Shah Replied
Ok thanks, i have noticed one more thing that REVERSEDNS is getting triggered for internal mails which again is adding score leading to mails getting flagged as spam
1
Hemen Shah Replied
Can we get some update on this thread from SM team ?
 
 
2
Ronald Raley Replied
Set the Bayesian Filtering score to something minimal like 3.
 
In our years of using SmarterMail, Bayesian Filtering is the most likely filter to score false-positives.  This seemed to happen all time time.  Even after resetting it and increasing memory for it.
 
It is unreliable in my opinion.
0
Colton Morrison Replied
I'm inclined to agree with you.
0
Hemen Shah Replied
I had raised the issue to SM and they have confirmed rdns triggering for internal mails has bug and fix is expected soon
1
Hemen Shah Replied
I have been facing bayesian issue off lately post last couple of 15.x builds and before that was ok, off lately lot of complaints of false positives.
0
Colton Morrison Replied
That's good information. Thanks for following up!
No fix for my ticket from SM yet.
1
Linda Pagillo Replied
Hey guys. I have an email in to SM support about this, as you guys are not the only people seeing this issue. I also had a customer of ours report it to me the other day. As soon as I receive a response, I will post an update here.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
1
Colton Morrison Replied
So, I did receive an update from SM Support, but they only said that the Bayesian filter needs retrained: 

I think you finally just reached a point where enough users selected "Mark as spam" to start skewing the results.  

Sometimes the Bayesian filtering does need to get retrained. Typically we see this when the end-users are marking messages as spam, but frequently not marking messages as not-spam. I would recommend resetting this by the following steps:
1. Stop the SmarterMail service and ensure mailservice.exe exits completely.
2. Navigate to C:\Program Files (x86)\SmarterTools\SmarterMail\Service
3. Delete the Ham and Spam folders, and also the bayes.dat file. If you see any Bayes1.dat or Bayes2.bat, etc please clear these as well.
4. Start the SmarterMail service
5. The bayesian filter will now start fresh. Based on your settings, the filter should kick in after 500 classifications of spam\not spam by your end users.

So, it appears that if there are not enough users marking messages as "Not-Spam" then your Bayesian filter can be skewed and send everything into spam. 

1
Linda Pagillo Replied
Hi Colton. Thanks for the info. I really find it hard to believe that this is what is really happening. You are the 3rd person that has reported this in the past week or so. I'm thinking that there may really be a problem with the Bayes. It just seems strange that it's happening to you guys all of a sudden. I'm hoping that SM support gets back with me soon. If I receive the same answer that you received, I will question it further because again, I really feel that there is more going on. Thanks.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
3
kevind Replied
As Ron mentioned above, we've seen this behavior for many years, nothing new here.  You can reset (retrain) it as listed above, but that only lasts for a few months.
 
Idea for Software Enhancement:  Users shouldn't have to mark legitimate messages as Not Spam. Instead, all messages delivered to the Inbox with a spam score of 0 or less should be automatically fed into Bayes to keep it trained.
3
Linda Pagillo Replied
I agree with your suggestion, Kevin. I'm still waiting on SM support to respond to my inquiry. I just want to make 100% sure that there is not an actual issue with Bayes.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
0
Emmet McGovern Replied
I'd rather see messages that are replied to are marked as ham. We see plenty of zero day spam with a score of 0.
0
kevind Replied
That's a good idea, but maybe change the criteria to auto feed messages that are from contacts and trusted senders. Then it will include newsletters and marketing emails which you don't reply to.

Zero day spam should get 'marked as spam' and fed back to Bayes.
0
Colton Morrison Replied
SM Support just emailed to say they are looking more into the Bay filter.
Wanted to let you know we're going to review this with the developers and see what can be done here. I'll let you know as soon as I have any more information.
2
Linda Pagillo Replied
Hey guys. SM support emailed me and said that they are working on this further with their dev team. I'm hoping to hear back with more info as soon as possible.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
0
Hemen Shah Replied
Hi Linda / Colton,
 
Any one of you have received any further update from SM on bayes.
i  have just sent them reminder on my open ticket.
 
Tks
0
Colton Morrison Replied
Nothing yet on Bayes filtering.
0
Linda Pagillo Replied
I haven't heard anything back as of yet.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
0
Hemen Shah Replied
I have received an email from SM stating, they were able to test the bayes behaviour and have identified major flaw for which it is only possible when they rewrite the whole thing which is not possible now and as a result bayes would be removed starting from SM 17 and will not be offering support for bayes on older versions also they have asked to disable the bayes filter strongly
0
Colton Morrison Replied
Yes, I received the same message recommending we disable Bayes. What do I replace it with? Cyren doesn't 'suspect' enough emails which Bayes would normally flag.
0
Colton Morrison Replied
See below for SM Support's response...
2
Linda Pagillo Replied
Guys, here is the exact message I received from SM support about this...
 
Linda,
 
I've had an opportunity to sit down with the SmarterMail team and demonstrate the behavior. Through our testing, we've identified a major flaw with the Bayesian filter implementation, as a result of our investigation, Bayesian filtering is actually going to be removed from SmarterMail, starting with SmarterMail17 and dropping support in all older versions as well. 
 
We can no longer offer support for the Bayesian filtering spam check since it is fundamentally flawed and cannot be corrected without a complete re-write. We strongly advise disabling the Bayesian filter check indefinitely. Please note that the performance of the Bayesian filtering within SmarterMail has been surpassed by other methods of spam detection such as Message Sniffer and Cyren. These two platforms leverage hashing functionality along with heuristically based analytics to more accurately detect and stop spam messages from being delivered.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
0
Linda Pagillo Replied
I would give Message Sniffer a try.
Linda Pagillo
Mail's Best Friend
Email: linda.pagillo@mailsbestfriend.com
Web: www.mailsbestfriend.com
Authorized SmarterTools Reseller
Authorized Message Sniffer Reseller
 
0
Joe Wolf Replied
I figured that out several years ago and quit using the SM Bayesian filtering. Don't use the SM DNS Cache either unless you want RBL/URIBL false results.

-Joe
0
Robert Emmett Replied
Employee Post
@Joe, in SmarterMail 17 we have replaced our internal DNS lookup tool with a third-party library. After testing several different DNS libraries--some suggested by community members--we've decided to use DnsClient. This will greatly help with DNS caching and delays with DNS lookups in general.
Robert Emmett
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
4
Colton Morrison Replied
Someone suggested using Message Sniffer. So since we used Cyren already, I turned on the trial and ran an analysis.
 
This scorecard only includes messages which were scored by both Cyren and Message Sniffer (aka I excluded the messages from today when I did not have Message Sniffer trial enabled.) This pie also has nothing to do with changing settings or scored values away from the defaults. 
 
The two little slices are the key parts for your consideration. 
Cyren flagged 7785 messages on its own.
Message Sniffer caught 6251 messages on its own.
 
Hope this helps!
5
kevind Replied
Official Word from SmarterTools?
 
Linda and others have done a great job of researching this topic and I was hoping to get the official word from SmarterTools.  Should we disable Bayes on all production SmarterMail installations effective immediately?
 
Since this is pretty significant, I'm sure all customers would be interested. Also, please include info on the 'fundamental flaw' so that we can determine how it affected our servers & messages.
 
Thanks in advance!
0
Ionel Aurelian Rau Replied
+1
1
Paul Blank Replied
The chart is nice. I'm curious to know if the messages caught / passed are all known spam or all messages combined. Would also be nice to know how many false positives were caught, etc.
0
Joe Wolf Replied
For those of you that feel the need for Bayesian filtering and don't want the expense of Cyren or MessageSniffer (which do much of the same thing on their own systems) there is an alternative that is much lower cost.  You can use SpamAssassin In A Box by JAM Software to configure a fully operational version of SpamAssassin that works great with SmarterMail.  It does take some time to get the configuration right, but once that's done it can take over most or all of the spam filtering for you (and includes Bayesian filtering).  It only cost about $50 per year.
 
I used SpamAssassin In A Box for several years and had good luck with it.  I personally dropped it when I switched to MessageSniffer.  I had our inbound spam filtering as good with SpamAssassin In A Box as we get with MessageSniffer, but SpamAssassin is not the way to go for outbound filtering.  MessageSniffer is excellent for outbound filtering as well.
 
The ONE good tool that SmarterTools gives us for outbound protection (Abuse Protection - Bounces Indicate Spammer) hasn't worked for years.  If that one tool worked it would essentially solve a huge problem, but I have to face reality that they'll never fix it.
 
The point is that there is a choice.
 
I don't want to go off topic, but there was a comment that SM17 is going to use a different setup for DNS. I have no idea why SM is in the DNS caching or any third party service for any reason.  We ALL run SM on a Windows server and every Windows server has a very fast an accurate DNS server built in... free of charge. Most URIBL's and some RBL rate limit or block public or high volume DNS query servers.  I don't know of any reason any SM installation should use anything but the DNS server on the local machine.  There is no value of any kind for using a public or third party DNS resolver.
 
-Joe
0
Colton Morrison Replied
No, sorry, I don't have that kind of telemetry.
1
Andrea Rogers Replied
Employee Post
Hi all,
 
As Linda mentioned above, we have removed the option for Bayesian filtering from SmarterMail 17.x (currently in BETA), 16.x and 15.x, due to its fundamentally flawed implementation. Please note that the performance of Bayesian filtering within SmarterMail has been surpassed by other methods of spam detection, such as MessageSniffer and Cyren Antispam. These two platforms leverage hashing functionality along with heuristically based analytics to more accurately detect and stop spam messages from being delivered. In short, Bayesian filtering is a relic of older days. You could consider it the horse and carriage compared to the automobile that is modern spam protection. 
 
That said, those who still wish to implement Bayesian filtering may benefit from reviewing eFa project, a third-party, open-source filter that includes Bayesian and dedicated Spamassassin.

Andrea Rogers
Communications Specialist
SmarterTools Inc.
(877) 357-6278

www.smartertools.com

0
Joe Wolf Replied
Another option would be SpamAssassin In A Box by JAM Software. Includes full SpamAssassin with the Bayesian filtering. Low cost and can run on the same server as SmarterMail. To integrate you just add SpamAssassin In A Box as an external SpamAssassin server in SM. There are a couple of third party rulesets you can add that are very good.

-Joe

Reply to Thread