Oh wow, very cool of you to put it in Github. I'll have to experiment with these rules at some point, maybe we can foster our own set of rules here on this community. Good to hear the integration is working out for you. Have you setup the SPAM/HAM stuff as well?

Hi Matt

Thanks for the feedback. The rules will grow day by day. The potential is really high to create cool stuff. 

The SPAM/HAM stuff is actual a mystery to me. I see in the WebUI, that the system has some "learns" and in the log i see strange entries, that Redis needs more input to learn. I have to learn this stuff.

Rspamd is a beast :-)

Really glad to hear things are working out, it is a great tool!

We are not familiar with Ubuntu and would like to see a SmarterMail step-by-step.

Thank you,

Ron

Ron,
Maybe my installation guide will help you?

"The SPAM/HAM stuff is actual a mystery to me"

I've tried to find a way to have full confirmation that it works but I didn't succeded here. 

Maybe I ask different question: does scan/learn (HAM and SPAM) from rspamd GUI from Scan/Learn section is the same what is SmarterMail is doing when using learn SPAM API endpoint? I'm asking because there was some message which I've from webmail marked multiple times as SPAM which caused webmail to move this  message to junk mail. Few moments ago I've found this message in spam and used its source to check message in rspamd gui. Then I used this spam message source and used Upload SPAM button from rspamd gui and scanned again. Take a look at difference between scans:

as you can see there is huge difference before and after using button Upload SPAM from rspamd gui interface and I've observed this behavior multiple times in past when I had few free moments to take a look at this. So conclusion:

Is there any chance to have some kind of debug log activated by some ID number to enable rspamd communication logging to find out how this communication looks?

Great comment and interesting questions! But they will tricky to answer without help of the programmer.

Maybe it makes sense to switch the log of Rspamd to debug mode. Link: 

Docs Logger

Ron,

There are a few ways to set up Rspamd. Martin has a good guide, and we have a KB on how we set it up for our environment: https://portal.smartertools.com/kb/a3595/deploying-rspamd-for-use-with-smartermail.aspx

Ok but how about situation where I can see rspamd in header but still I'm not sure about HAM/SPAM learning? Troubleshooting section from mentioned KB article is related to SPAM classification (which works fine) but there is no mentioning how to troubleshoot learning SPAM/HAM (at least from SmarterMail side - no log entries related to success or failed operation).

EDIT:

Maybe I will add some clarification about environment on my end. I use incoming gateways for spam checking where rspamd is configured to perform spam checks (so /checkv2 command from rspamd) BUT I also have configured rspamd on main SmarterMail instance where no spam checks are enabled BUT rspamd is added and Antispam -> Send user spam feedback to antispam providers is enabled (so we have only usage of /learnspam and /learnham rspamd endpoints from main SmarterMail instance). Looking into reports and rspamd processing on incoming gateway we can see:

where on main SmarterMail instance:

(0 for each day). I'm even not sure if connecting to /learnham and /learnspam enpoints is also being counted for Rspamd processing report and thats why I think there should be some kind of place for troubleshooting /learnspam and /learnham connectivity.

I've reopened ticket 0FF-2C54BEB7-0B4F which was related to HAM checks for rspamd with my additional comment about it. Basically using command:

tail -f -n 30  /var/log/rspamd/rspamd.log | grep trusted

you can see all connections from secure_ip addresses which are not using any form of authentication since they are whitelisted and I see only IPs from my incoming gateway. Only scenario where I can find there IP of main SmarterMail instance is when I open RSPAMDURL:11334/learnham URL in browser where marking as SPAM using webmail didn't ended with main SmarterMail IP appeared in filtered logs.

Ok. I've done some testing. You can find in rspamd.log entries from learning spam/ham by using:

tail -f -n 30  /var/log/rspamd/rspamd.log | grep learned

command. It does not report SmarterMail move to junk folder button actions right away. On my end it took about 3 minutes between moving message to junk folder and showing some info about it in rspamd.log. I'm not sure if all messages moved to junk folder appear in rspamd.log. Maybe some of them don't since they are already on bayes db. So sending spam to provider works but still I think it would be nice to have some logging on this to have possibility to compare messages marked as spam with messages parsed by rspamd.

@Webio

We have a couple minute delay before we commit the files as SPAM/HAM incase users accidentally move items into their Junk or Inbox that they didn't intend to and undo their action.
Also, each message will only report as HAM or SPAM once and then never again for that message.

Also also, the move to junk/inbox button simple moves the message. If you manually move from Inbox -> Junk or Junk -> Inbox (only those folders) we will also trigger the spam/ham functionality, this accommodates clients that move mail. 

@Matt: This is an interesting answer and explains a lot :-) 

How much CPU/RAM does the Ubuntu server require for rspamd?

Ron,

It depends probably how many mails Rspamd processes per day. My server runs with 4 GB Ram, one CPU 2GHz processing between 3 to 5K Mails per day. The CPU Load is average at 7 to 9%. So Rspamd would run on a toaster.

Ron, our rSpamD instance (also runs VadeSecure) we have at 16GB of RAM, uses around 10GB for the Redis backend, we process about half-a-million or so emails per day. CPU is 8 cores total, running at 25%.

I'm sorry if this has been explained elsewhere but I looked in the SmarterMail Help, the community thread asking for the report spam button, and searched the community for info and couldn't find the answer:

Does the SmarterMail SPAM/HAM learn functionality teach SmarterMail itself anything or does it only provide the messages to Rspamd?

In my rspamd build moving to junk did not result in 'learned as spam' in rspamd.log.  Tried both in client (outlook 2016) and webmail   Waited an hour but no result.

There are plenty of entries 'learned as ham'.  Unsure if it is my rspamd build or something else.

In my opinion would be nice to have some sort of /learnham and /learnspam troubleshooting or test baked in the SM as confirmation everything is working.  Maybe just some type of log in Troubleshooting so you can trace the information being passed to rspamd.

As i said, the HAM/SPAM thing is kinda a mystery.

My Rspamd-Console shows 283 learns for HAM and 74 for Spam. This is to low to take action. 200 are needed. I asked for more background information. I didn't receive a helpful answer.

Actually i have to say, that the HAM/SPAM module is kinda not satisfying. 

I recalled seeing somewhere in the rspamd documentation that you can modify that 200 count threshold.

Yes i know. I asked how the default values stands in relation to the number of incoming mails. Just a little background would be great. But the universe answered with silence.

I think it is here:

Rspamd trains a neural network when (ham_samples + spam_samples) >= max_trains

@Webio: Thanks! I have read this before, but i'm not really sure, which setting needs to be changed. But it's okay for the moment. I don't have the time and nerves to do basic research work. 

Alright guys. So let's talk about the elephant in the room.

We are running Cyren, Message Sniiffer, RBLs, and all the obvious stuff like Dmarc, Reverse DNS checks, etc.

No spam complaints in a long time. Thank God. Our spam settings are "tuned."

What advantage is rspamd to setup a new VM, configure, run, pay for.

Hypothetically, would this cost... move our 98% spam catch rate to 98.2%. Who cares at that point?

What's the bottom line?

Bottom line: it does matter IMHO only for people who can run rspamd on their own (so no additional cost of installation and running it) so they don't need to use paid solutions (for Message Sniiffer and Cyren unlimited license cost is about 1450 eur) which you are using.

Got it.  We had not considered a complete replacement with rspamd.  Thank you.

I think rspamd makes sense if you really like to invest time into your own ruleset. A lot of logic that rspamd uses is already in smartermail like: Greylisting, RBL, URIBL, DKIM, SPF, DMARC, Spamassasin Patterns, antivirus. 

Rspamd makes sense, if you don't want to invest monthly or yearly fees for a antispam software or a SAAS thing. As i wrote before, Rspamd will be my replacement for Declude.

My Rspamd server cost a few dollars a month for the cloud server. Greylisting DKIM, SPF DMARC will probably will stay at SmarterMail. 

I created now tons of simple regex stuff like i did it with Declude. Later on, the "dumb" regex stuff can be extended or replaced by scripts. So you need to learn LUA.

A challenge will be, if you are stuck and you need help. Forget it. I'm sure, that other people made some cool scripts or shiny rules. Nobody shares something and nobody shares knowledge. 

Actually i let declude run side by side with Rspamd as long as my Rspamd stuff will be super cool, or at least cool :-) I'm not in hurry.

I think, in terms of money, Rspamd isn't a cheap solution if you need t count your time to create rules and scripts. The lack of community support may also count to make a decision.



. 
 

I set up my rspamd server just to explore a new thing, learn and see what it can do.  Wasn't purpose driven by cost or anything else.