AI-Based Spam Protection
Idea shared by Ron Raley - 3/24/2024 at 5:08 PM
Can we look at competitive spam detection partnerships for SmarterMail? I read Google's spam system is AI-based now. Someone somewhere has to be doing similar things...

7 Replies

Reply to Thread
Most of the cloud-bssed services will claim an A.I. component, an assertion which I am inclined to take at face value    Getting details about what their A.I. does is another matter.  But you could subscribe to one of them if you think they are worth the high cost.  Of course, A I. is still guesswork, even if it is sophisticated guesswork.

A I. requires a large database of sample mail, which is why these solutions are cloud-based.   SmarterTools is obviously not privy to that trove of data, so they are never going to be a player in that space.  

The ONLY thing it would take for Smartertools to be competitive in that field would be to impelement a centralized spam reporting service where all the users of smartermail would send the reports to.

That way the list of spammers/emails and domains would grow rapidly making the list better an better.

Right now everyone is figtning spam on their own and it would add incredible value to SM to do that.
HOWEVER, even a centralized spam reporting service is subjective to the market/industry/customer being served. And that is a big stumbling block.

The war on spam is not a one-size-fits-all situation. The potential spam/malicious emails seen/experienced by a clothing outlet chain will be very different from that seen by a law office which will be very different than a health care provider which will be very different from an international supply chain logistics provider which will be very different from a global electronic components supplier.
I have made good progress, at relatively low cost, by combining using a heavily customized version of Declude for sender filtering, and a relatively inexpensive commercial appliance to do content filtering and message review.

Sender filtering blocks known-bad mail, validates mail that needs whitelisting, and flags unauthenticated mail.  Unauthenticated mail is reviewed and categorized:  unwanted senders get a block rule, while wanted senders get an allow rule which includes underlying identifiers that can be authenticated.   This exposes a lot of spam and unwanted advertising.

I use a commercial appliance for content filtering because I don't have the email data set or the parsing sophistication to filter content effectively on my own.   My commercial appliance does pretty well, although I do supplement it some with local rules.

For message review, I use the message log in my commercial appliance as the primary tool, as well as SQL data logged by Declude as a secondary tool.   Both SmarterMail and Declude lack an adequate tool for efficiently reviewing message disposition, so solving that obstacle is a first step toward an adequate spam filtering solution. 

Douglas, I agree that Declude, and even better, DR (the rebuilt version), does a great job. And in my case, is the only real option as it seems most off-the-shelf email filtering solutions are mostly centered around the one-size-fits-all category for easy administration.
I manage email for a single business.  My users expect me to deliver all of the wanted mail as well as block the dangerous mail.   So guesswork has little appeal, whether the guess is implemented using A.I. or SpamAssassin, or something else.  I need control.

I have shopped, and looked at products with pricing up to $36 per user per year, but been consistently disappointed.  I kept asking vendors, "How do I override a sender's SPF error or omission using a combination of HELO name (verified by forward-confirmed DNS) and SMTP domain?   This was driven by experience:  multiple attempts to enforce SPF had to be rolled back quickly because of SPF failures.   It seemed like a basic feature requirement that would be apparent to anyone involved in email filtering.  It was not my only expectation, just the easiest one.

I got a steady stream of the same answers:  "you can't do that", often mixed with "just trust us, you will never need to do that".  Eventually, I stumbled on Declude and quit looking.  Declude needed customization to do SPF checking correctly, and to evaluate forward-confirmed DNS, but unlike everything else, it could be customized.   

For what I wanted Declude to do, the number and size of filter files quickly became unmanageable and inefficient, since Declude rereads every file for every messages.   That's why I moved most of the processing into SQL, where lookups are indexed.  I currently have just under 20,000 rules in 6 SQL tables, supplemented by a manageable number of Declude custom filters for situations that do not fit my database design.

When my labor cost is factored in, our solution may or may not be cheaper than the cloud solutions, but it definitely gives us better control.

Have begun playing with Declude Reboot, but have not completed the process.   The priority has been a rewrite of my custom scripts using Python email module and related modules.
IMHO rspamd is best answer but you need to dig in into documentation and know a little linux administration (there is a docker version but I didn't used it).

Best of all here I think is that SmarterMail can push spam/ham messages to rspamd API and learn its filters based on users markings.

Reply to Thread