Back to Community Threads
7
Message Sniffer Support?
Idea shared by Emmet McGovern - 11/5/2014 at 7:48 AM
Completed
I thought you were adding message sniffer support in SM13?  What happened?

119 Replies

Reply to Thread
4
Employee Replied
11/5/2014 at 11:21 AM
Employee Post
Hello Emmet,
That functionality is still planned for SmarterMail 13, but will be coming in a minor release.
4
SmarterMail already supports Message Sniffer.  Here are the installation instructions:
http://armresearch.com/Documentation/Papers/InstallGuides/Smartermail.jsp
 
Just to throw in my opinion... Message Sniffer is probably the best single anti-spam tool available... period.
 
-Joe
Thanks, -Joe
0
That's fairly expensive...
0
Also that method will not work with smtp blocking will it?
0
Well depending on the license it's cheaper (or the same) than Cyren and in my opinion MUCH better than Cyren. I don't know how SmarterTools would implement Message Sniffer, but I suspect it would be the same as Cyren so the answer would be NO you couldn't use it for SMTP Blocking. But there are much easier and better ways to accomplish that if desired.

I'm not trying to sound like a commercial for Message Sniffer, but it's one of the few products out there that does exactly what they claim. When I tried Cyren it was hit and miss. Message Sniffer is rock solid at blocking even the very newest spam storms.

-Joe
Thanks, -Joe
1
Can I remove any of my other checks once message sniffer is installed?
0
While you can use MS through command line or other tools, its not as ideal as direct integration assuming SM is going to use the SDK method for integration. That said... no matter how you use it, it is the single best spam prevention we use. We don't even weight on it, we just delete.
0
No, you should still use your other checks. We just delete on positive which eliminates a majority of the spam, then let the rbls and uri's clean up the rest.
0
I don't know your configuration, but I would keep a few RBL's like zen.spamhaus.org, and b.barracudacentral.org and bl.spamcop.net. I would also keep the Reverse DNS test. Enable SMTP Blocking and set your Incoming Threshold to a value that would require at least 2 of the above tests to FAIL. So if all of the above were weighted at 10 then set the SMTP Blocking Threshold to 20. That way you never block a valid message, but you'll reduce the load on your server considerably since about 2/3 of the spam will simply be blocked by using these very reliable RBL's and the rDNS. Never block on any one test.

Additionally I'd add three URIBLS... dbl.spamhaus.org, multi.uribl.com, and multi.suribl.com and keep a weight of half an RBL... so in my above example they would keep a weight of 5.

You can keep other tests like SPF, DKIM, but never give them a negative weight for passing, and only a slight weight for failure.

Then if you add Message Sniffer to your SpamAssassin-Based Pattern Matching and you will virtually ELIMINATE all spam and have a near zero false positive rate.

One other suggestion... never set a spam weight action that would Delete a message. That leaves messages unaccounted for (neither the sender nor receiver know about the message) and could put you in a very bad legal liability position.

Best of luck,
-Joe
Thanks, -Joe
0
My setup now is basically 90% exactly what Bruce recommends in his spam document. I am also currently running a full blown SpamAssassin that has helped immensely with reducing spam. I was able to tweak the default scores to get the best results.

I want to get away from that though as we are getting false positives, not many but still.

I am goign to remove my spam assassin setup and substitute it with MS.

I'll test for a month and see if we want to continue.
0
Emmet, I don't know your reason for running SmarterMail. If you're running simply for yourself, or your company, and the company has made the decision to delete messages then I think that's fine. But if you provide email services for others then it's very dangerous to set an action to Delete.

Everyone is free to run their server how they see fit, and I'm not preaching here, but I would just like to point out the danger of setting spam weight action to Delete.

First, if you use good SMTP Blocking practices (use only highly accurate tests and make sure the message fails at least two tests). This will eliminate most spam, and the sender of the message is notified that their message wasn't delivered. You're covered because they were notified the message was blocked.

Once a message passes SMTP Blocking for further testing then you should really only use up to three actions. Either do nothing and deliver to the user, prefix the subject as suspected spam and deliver to the user, or hold the message in their Junk E-Mail folder. In the case of the latter we send out a report every night to each user informing them of messages held in their Junk E-Mail folder but will be automatically deleted in 14 days (via Folder Auto-Clean). But in any case the message is accounted for... it was either delivered or held. and the recipient was notified.

If the user decides to set a filter to Delete a message that's their choice and you have no liability.

If you set an action to Delete then neither the sender nor the receiver know anything about the message. There are 100% valid messages sent thru email servers that are improperly configured. It happens. If you take upon your own authority to Delete a message without notifying either the sender or recipient of the message 100% of the liability is on your shoulders.

Just my opinion,
-Joe
Thanks, -Joe
0
We use Smartermail's message archiving system wide. So all messages are backed up.

Your users must have huge junk folders!
1
What about ClamAV, is it safe to disable once MS is in place?
0
No, not really, but it really doesn't matter since it's Auto-Cleaned. Our script that sends out our nightly Quarantine Report gives users instructions on how to add any valid messages held in their Junk E-Mail folder to their Trusted Senders list which improves overall accuracy. For example, there are a few newsletters like "This is True" and "Swanni's TV Predictions" that are always listed on an RBL. Those are just a couple of examples, but many newsletters end up on RBL's. The way we have things set-up the users that want to receive those types of newsletters are notified if their held in Junk E-Mail and only have to add them to Trusted Senders one time. Other users consider them spam.

The bottom line is that most of our users really like the system. We also allow them to opt-out of receiving the report, but very few do.

We also archive ALL messages for 3 days but we don't let our customers know that. We do sell per domain Archiving service (1, 3, or 5 years) for those needing HIPAA compliance. Keep in mind if you set any spam weight action to Delete your server cannot be considered HIPAA compliant.

But once again, everyone is free to run their server the way they want to do so.

-Joe
Thanks, -Joe
0
Thanks Joe!
0
If you're using a product like SpamAssassin In A Box then you could simply add Message Sniffer to that instead of directly to SmarterMail.
Thanks, -Joe
0
Yeah that's what we were using... I will maybe attempt that later, but I was suspecting it for false positives. Doesn't MS do all I need? I am going to use the command line just till it's added natively in smartermail.
0
I know Message Sniffer claims to do virus scanning, but since it's really just a rule-set I don't have full confidence in it catching all the viruses. I'd keep ClamAV enabled. There will be no conflict between the two, and there's nothing wrong with staying safe and leaving ClamAV running.

-Joe
Thanks, -Joe
0
Message Sniffer is great, but SMTP Blocking (and Greylisting) eliminate the most spam. If you don't SMTP Block you'll be overwhelmed with spam. Take a look at your Reports, Dashboards, System Statistics, Security tab and you'll see how important it is to not accept all those messages. Just make sure to use reasonable SMTP blocking policies.

If you have good SMTP Blocking policies in place, enable Greylisting with reasonable settings, then you might be able to get good results by using Message Sniffer as the only test. I've never tried it. Give it a shot and see what happens.

-Joe
Thanks, -Joe
0
We've been using Message Sniffer since it was released back in the Sort Monster days. I haven't seen a false positive in years. Occasionally we quarantine it for review but most of the time it goes straight to the trash.
0
Well for now I am simply leaving my setup alone except for swapping SA for MS... I'll see how we do.
0
The ONE thing I'm not sure about if you use that setup is that I THINK you'll have to update your local.cf file every time SmarterMail is updated. I believe it overwrites the existing local.cf, but I am NOT sure of this.

Technically SmarterMail should NOT overwrite the local.cf (and maybe they don't, but I think they do).

-Joe
Thanks, -Joe
0
Well hopefully the next minor has the MS integration
0
Hope to have this feature implemented in SM 13 as it will help us.
0
I got a few spam over the weekend that I then used to tweak the Smartermail scoring with. So far so good though. I am seeing less spam overall I believe.
1
WARNING:  If you're running Message Sniffer via the command line and added Message Sniffer to your SpamAssassin-Based Pattern Matching local.cf file you will have to add the rules back to the local.cf file any time you update or re-install SmarterMail.
 
I set up a test, and SmarterMail overwrites the local.cf file.  I tried to make the local.cf file read-only and the upgrade threw an error.
 
So just remember to add the Message Sniffer rules back into the local.cf after any re-install or upgrade.
 
SmarterTools:  I think this is a bug.  The local.cf file should be a persistent file in any SpamAssassin installation and an sa-update, etc. should not touch that file.  I think your installer should check to see if a local.cf exists and if so do not overwrite it.
 
-Joe
 
Thanks, -Joe
0
Rather than have them worry about that, I would say it's best they focus on actual core integration in Smartermail. My hope is to see it in the next minor... Which with any luck may be sooner than later. Fingers crossed.
0
I agree that native integration will be an improvement, but the local.cf being overwritten is a bug. SmarterMail's SpamAssassin-Based Pattern Matching is just a subset of the SpamAssassin content filtering rules. System Admins may wish to add or remove weight from some of those rules or even create their own rules. The local.cf is the place to make such changes and should NOT be overwritten.
Thanks, -Joe
0
I was able to tweak SpamAssassin in a Box to increase the score for tests that I wanted.
 
How am I able to tweak the scores in Message Sniffer?
 
Also it looks a lot like Message Sniffer is using smap assassin.
0
Why do you say it uses spam assassin?
0
Just because the scoring lines added to the header looks identical to what my SpamAssassin in a box looked like.
0
How else would it be compatible with systems designed to filter on SA like headers?
0
OK but my point is how is this any better than a normal SpamAssassin install that I can tweak and fine tune.

I have not gotten DIET emails since tweaking SpamAssassin in a box to score them higher to 5.0. Now with Message Sniffer I am getting DIET emails again and they are scored 0.7 (which was the default SA score BTW)

Being that I can't adjust Message Sniffer scoring then all of a sudden it's not serving me as well as SAiab
0
You can just add the Message Sniffer rules to SpamAssassin In A Box and the weight will transfer just fine.

Take a look at the Message Sniffer lines and see what it's saying on those messages. It takes a while for Message Sniffer to adapt to your system.

You should be seeing lines like the following in the message header:
X-MessageSniffer-Identifier: C:\SmarterMail\Spool\SubSpool3\261079409881.eml
X-GBUdb-Analysis: 0, 198.57.220.30, Ugly c=0 p=0 Source New
X-MessageSniffer-Scan-Result: 55
X-MessageSniffer-Rules: 55-6399744-1835-1906-m
55-6399744-3239-3310-m
55-6399744-4726-4797-m
55-6399744-0-4899-f

Then the appropriate line passing the weight to SmarterMail:
X-SmarterMail-SpamDetail: 5.0 SNF_MALWARE Virus, worm, and exploit patterns

-Joe
Thanks, -Joe
0
How are you implementing MS?
0
I use it inside SpamAssassin In A Box, but I've modified the normal install of SpamAssassin In A Box quite a bit.

-Joe
Thanks, -Joe
0
Sorry... I was asking Steve
0
I used Joe's initial link that impliments it via command line.
0
How do I add Message Sniffer to my spam assassin in a box install? I read the instructions but they are not clear enough
0
I got it configured and running, however I am seeing this error:

ERROR_MSG_XHDRi: Drop Msg XHDR injector can't remove original!

Even though the temp folder has everyone permissions
0
Looking forward to this...
0
Turns out somehow the plugin I downloaded was old. Got the latest version from Linda Pagillo, thanks so much!! Issue resolved.
0
Just FYI, if you're using SpamAssassin In A Box and want to use Message Sniffer the easiest way to do it is to follow the SmarterMail instructions for Message Sniffer, but simply add the rules to your SpamAssassin In A Box local.cf instead of the SmarterMail SpamAssassin-Based Pattern Matching.

You do NOT need to add the SpamAssassin plugin for Message Sniffer to SpamAssassin In A Box.

-Joe
Thanks, -Joe
0
Yeah but using the plugin means no command line is required.

Also nothing needs to be added to local.cf
0
Where did you download the outdated version Steve? Im curious about trying the direct integration into SpamAssassin In A Box. I haven't used the box version in a while but I see the CPU and stop issues are supposed to be resolved in the last update.
0
OK, I've installed it both ways. I'll try and give the pluses and minuses of both methods.

First, if you install Message Sniffer via the SmarterMail command line and add the rules to your SpamAssassin local.cf (either version) you get more information in the message header. For example:

X-MessageSniffer-Identifier: C:\SmarterMail\Spool\SubSpool7\261079438354.eml
X-GBUdb-Analysis: 0, 65.55.116.23, Ugly c=0 p=0 Source New
X-MessageSniffer-Scan-Result: 53
X-MessageSniffer-Rules: 53-6565193-2022-2076-m
58-6560660-2022-2076-m
53-6565193-0-32495-f
X-SmarterMail-SpamDetail: 5.0 SNF_SCAM Phishing, 419, and other scam patterns

So you can see what rules the message failed and the weight it transferred to SmarterMail and listed in the X-SmarterMail-SpamDetail header.

The minuses to this method are that Message Sniffer runs on all messages (both inbound and outbound) but of course doesn't apply any weight to outbound messages unless you select that option in your Antispam tests. Also, Message Sniffer always passes a weight of 5 to SpamAssassin and as a result it's a go or no go test.

Secondly, if you have SpamAssassin In A Box or other Remote SpamAssassin server and implement Message Sniffer via the snf4sa.cf and snf4sa.pm files you get the benefit of variable weights assigned by Message Sniffer (it can even apply a negative weight which I don't really like, but I'm sure I can configure it to not do so). Here are a few examples:

X-SmarterMail-SpamDetail: -1.2 SNF4SA Message Sniffer
X-SmarterMail-SpamDetail: 5.0 SNF4SA Message Sniffer
X-SmarterMail-SpamDetail: 4.2 SNF4SA Message Sniffer

Another benefit is that it doesn't run on outbound email messages unless you enable Remote SpamAssassin for outbound messages (bad idea).

The minuses of installing this way is that the above line is the only information you get. You can't tell what Message Sniffer rules were tripped... all you get is "Message Sniffer" and a weight.

My summary: I like the additional message header information provided by installing the Message Sniffer rules to the local.cf and that it is a static weight if the message fails Message Sniffer. Message Sniffer seems to use very little CPU and RAM so the fact that it checks the outbound messages (even if you don't use it to apply any weight) the additional load is of no concern to me. Also this method works for any SmarterMail installation and doesn't require SpamAssassin In A Box or other Remote SpamAssassin installation.

Just my opinion... I like the additional info.

-Joe
Thanks, -Joe
0
Steve Reid, could you please describe your install procedure, and post a few examples of the headers from Message Sniffer / SmarterMail spam weighting.

I'm very interested in your results.

Thanks,
-Joe
Thanks, -Joe
0
Emmet McGovern, I downloaded the old non working version right off armresearch.com

I have informed them that the download is incorrect.
1
For anyone with SpamAssassin In A Box or Remote SpamAssassin and needs the updated files to integrate Message Sniffer you can grab them here:
 
-Joe
Thanks, -Joe
0
Once I got the most current version of the Message Sniffer Plugin SNF4SA it was straight forward. I just extracted the snf4sa.pm and snf4sa.cf into the C:\ProgramData\JAM Software\spamdService\sa-config folder and it began working.
 
I don't have good examples of headers though, since I enabled it I haven't had even one spam get through.
0
This message was spammy but not actual spam:

X-SmarterMail-Spam: SPF_Pass, Backscatterer, HostKarma - Yellowlist, SpamAssassin 0 [raw: 0], DK_None, DKIM_Pass
X-SmarterMail-SpamDetail: Content analysis details: (0.0 points, 5.0 required)
X-SmarterMail-SpamDetail: pts rule name description
X-SmarterMail-SpamDetail: ---- ---------------------- --------------------------------------------------
X-SmarterMail-SpamDetail: -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2)
X-SmarterMail-SpamDetail: [209.85.220.66 listed in wl.mailspike.net]
X-SmarterMail-SpamDetail: 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
X-SmarterMail-SpamDetail: (colorway12[at]gmail.com)
X-SmarterMail-SpamDetail: 0.0 T_KAM_HTML_FONT_INVALID BODY: Test for Invalidly Named or Formatted
X-SmarterMail-SpamDetail: Colors in HTML
X-SmarterMail-SpamDetail: 0.0 HTML_MESSAGE BODY: HTML included in message
X-SmarterMail-SpamDetail: 0.0 HTML_FONT_LOW_CONTRAST BODY: HTML font color similar or identical to
X-SmarterMail-SpamDetail: background
X-SmarterMail-SpamDetail: 0.0 T_OBFU_PDF_ATTACH BODY: PDF attachment with generic MIME type
X-SmarterMail-SpamDetail: 0.0 XPRIO Has X-Priority header
X-SmarterMail-SpamDetail: 0.0 T_FREEMAIL_DOC_PDF MS document or PDF attachment, from freemail
0
Joe important note that Message Sniffer only passes a 5 if you are using an old version of SpamAssassin. On new versions the weighting is dynamic.
0
They have updated the plugin download on their website. New version installed and working.
0
Steve, I think you misunderstood what I was saying. If you use the local.cf method it will score per the rules in the local.cf. All the rules (except 3) score 5. So there is no way to get a dynamic score using the local.cf method regardless of the version of SpamAssassin. Using the local.cf method also allows you to see the Message Sniffer rule results in the message header.

If you use snf4sa.cf then you get dynamic scoring as long as you're running SpamAssassin 3.2.0 or later. I'm running 3.4.0.30. The problem with this mode is that you don't get any of the Message Sniffer rule details in the message header. Just the one line with the dynamic score and "SNF4A Message Sniffer". No other info.

-Joe
Thanks, -Joe
0
In the above example Message Sniffer didn't trip any rules. You can send yourself some spam test messages from this site:
http://domino-118a.maysoft.com/selfservespam2.nsf/dl
Thanks, -Joe
0
I'm afraid everything is working so well now that these emails do not make it through. I still have not seen a single spam. Don't forget in our setup we delete.
0
I have pointed Arm Research support to this thread. I asked for them to include the same amount of detail in the headers for the plugin.
1
Would love to see Message Snifter built in!
4
After giving Message Sniffer a go for the 30 day trial period we have decided not to purchase a subscription. I honestly fail to see how it is any better than SpamAssassin.
 
We were still getting spam through and also I was getting complaints about false positives.
 
Message Sniffer is by no means a solution to Smartermails current spam issues.
0
Thanks for the information. We keep looking to find the "right" solution and haven't had one solution that our clients all universally accept as "good for them".
0
Hi Steve, Are you sure you were running it correctly, what is hard is you have to edit the local.cf file each time you install a new SmarterMail version, than you have to stop and start MailService after adjusting the local.cf file in SAData folder. Can't wait until SmarterTools integrates this into a minor update of v13. -dave
0
I actually am already running SpamAssassin in a box. So I chose to integrate it via the plugin for SA. It was all setup properly however there was only a negligible reduction in spam. Also there were more false positives than without MS. Either way we are sick of paying for so called spam solutions only to be left wanting more every time. My SpamAssassin in a box license was $77 for three years. Whereas Message Sniffer wanted $99 per year, which is insane considering how light our mail volume truly is.