Overview
SpamFoo has updated their documentation quite significantly from the first time that I checked their site. Integrating SpamFoo with Declude is pretty straightforward.
The primary function looks like this:
httpresponse = requests.post(
url='http://localhost:16253/classify-stream'
, headers={"Content-Type" : "application/json"}
, json=jsonquery
, data=filebytes
)
Why Integrate with Declude (or other product)
The default SmarterMail configuration of SpamFoo has these characteristics:
- SpamFoo runs on the post office server.
- SpamFoo checks every message in the delivery queue.
- SpamFoo cannot be influenced using message headers from upstream filtering processes.
- SpamFoo runs after other filtering processes, immediately before delivery to the user, to redirect some messages to the Junk Email folder.
- SpamFoo is able to override whitelisting dispositions assigned earlier in the filtering process, so whitelist rules must be configured in SpamFoo also.
- Learning is a delayed process. User feedback is passed to the vendor, the model is updated for all clients, then the rules in the new model apply for the all users on the server. This also introduces some uncertainty into the process.
- SpamFoo has only two options: deliver normally, if not spam, or delivery to Junk Email folder if spam. The “Should Escalate” result is not used because there is no quarantine option.
An alternate configuration is needed when:
- SpamFoo should cooperate with the primary filtering process, rather than second-guessing it.
- SpamFoo should run earlier in the filtering process, often on an incoming gateway, so that dangerous and unwanted messages can be kept out of the user’s mailbox.
- SpamFoo should run conditionally, when a final disposition cannot be determined by prior checks. This eliminates unwanted overrides while minimizing processing effort.
- SpamFoo returns results indicating whether the message should be blocked (“is spam”), quarantined (“should escalate”) or allowed (anything else). Integrating with Declude allows the “Should Escalate” result to be utilized.
- Spam-getting-through needs to be blocked for all users, quickly and with certainty.
- Spam filtering remains firmly in control of the system administrator.
Calling SpamFoo from within Declude permits the alternate configuration. Declude can be used to call a script which first decides whether to invoke SpamFoo at all, then return SpamFoo results as a list of tests triggered and test weights. This permits SpamFoo results to be integrated with other processing steps in any manner chosen by the administrator.
Configuring SmarterMail
These System Administration options should be chosen:
- Setttings… Anitspam… Spam Checks… SpamFoo… Disabled
- Manage… Services… SpamFoo… Started
Testing confirms that the SpamFoo service will be restarted after system restart, even though the SpamFoo test is not enabled.
Configuring Declude’s Global.cfg file
Configure a bitmask test in Declude. Because of problems when attempting to call Python directly, I always call a VBscript, then use that script to call Python. This sample bitmask is based on my sample code, which appears later. Test weights are at the discretion of the system administrator, so I simply show all weights as zero.
SPAMFOO bitmask 0 "cscript /nologo C:\SmarterMail\Scripts\spamfoo.vbs " 0 0
# Operating system or SpamFoo unexpected errors
SPAMFOO-ERROR bitmask 1 "SPAMFOO" 0 0
SPAMFOO-BADHTTP bitmask 2 "SPAMFOO" 0 0
# Exempted by Python or Bypassed by SpamFoo explicit rule
SPAMFOO-NONEED bitmask 4 "SPAMFOO" 0 0
SPAMFOO-BYPASS bitmask 8 "SPAMFOO" 0 0
# Spam Test Results
SPAMFOO-ISSPAM bitmask 16 "SPAMFOO" 0 0
SPAMFOO-ESCALATE bitmask 32 "SPAMFOO" 0 0
# Classification Results
SPAMFOO-PRIMARY bitmask 64 "SPAMFOO" 0 0
SPAMFOO-ADS bitmask 128 "SPAMFOO" 0 0
SPAMFOO-UPDATES bitmask 256 "SPAMFOO" 0 0
SPAMFOO-TRX bitmask 512 "SPAMFOO" 0 0
SPAMFOO-UNKNOWN bitmask 1024 "SPAMFOO" 0 0
Administrators may wish to use the Declude COPYFILE action to save specific messages in designated folders for administrative review. After review, spam corrections and classification corrections can be sent to SpamFoo using the /feedback-stream call. This would typically involve placing messages in specific feedback folders, and then processing the folder to transmit corrections. The review folders will need a process to purge old files, which is easily implemented on Windows using ForFiles.exe.
Calling Python from VBScript
This snippet of VBScript will receive the filename passed in by Declude, send it to the Python script, and return results:
Set WshShell = CreateObject("WScript.Shell")
set ObjArgs = Wscript.Arguments
ArgCount = ObjArgs.Count
if ArgCount <> 1 then
WScript.Quit(1)
end if
Set fso2 = CreateObject("Scripting.FileSystemObject")
filename = "" + objArgs(0)
runcommand = "c:\progra~1\python312\python.exe c:\smartermail\scripts\spamfoo.py " & filename
runstatus = WshShell.Run(runcommand,7,-1)
WScript.Quit(runstatus)
Calling SpamFoo from Python
This sample code has been tested successfully.
import re
import sys
import requests
#
# Note: Beta license for SpamFoo requries using API function /classify-stream.
# API call to /classify will cause SpamFoo to throw an error 500 for being unlicensed.
#
def ParseHeader():
emlfile = sys.argv[1]
Match1 = re.findall('([A-Za-z0-9\\-\\_]+).eml',emlfile)
TrxID = Match1[0]
hdrfile = emlfile[:-4]+'.hdr'
file1 = open(hdrfile, 'r')
Lines = file1.readlines()
# Strips the newline character
hdrdict = {}
hdrdict['EnvFrom'] = None
hdrdict['EnvTo' ] = None
hdrdict['helo' ] = ''
hdrdict['connectedip'] = ''
hdrdict['connectedhostname' ] = ''
hdrdict['EnvFrom']=''
sent_to = []
for itm in range(len(Lines)):
thisline = Lines[itm].strip()
tmppos = thisline.find(":")
if tmppos > 0:
keyword = thisline[:tmppos].lower()
keyvalu = thisline[tmppos+1:].strip()
hdrdict[ keyword ] = keyvalu
if keyword == 'smarthost':
tmppos = keyvalu.find("@")
tmppos = keyvalu.find("=",tmppos + 1)
if tmppos > 0:
keyvalu = keyvalu[:tmppos]
sent_to.append(keyvalu)
else:
if itm == 0:
hdrdict[ 'Flag' ] = thisline.strip()
if itm == 1:
hdrdict[ 'EnvFrom' ] = thisline.strip()
elif itm == 2:
hdrdict[ 'EnvTo' ] = thisline.strip()
file1.close()
return( emlfile, TrxID, hdrdict , sent_to )
# End of ParseHeader function
def AskSpamFoo(serveruri,jsonquery,filebytes):
httpresponse = None
httpresponse = requests.post(
url='http://localhost:16253/classify-stream'
, headers={"Content-Type" : "application/json"}
, json=jsonquery
, data=filebytes
)
if httpresponse.status_code >= 200 and httpresponse.status_code <=299 :
results = httpresponse.json()
results['httpstatus']=httpresponse.status_code
results['httperror'] = False
else:
results = {}
results['httpstatus']=httpresponse.status_code
results['httperror'] = True
return results
# end: domain list fetch block
# ******* Edit these parameters *******************************
servername = 'http://localhost:16253' # Server to query
WindowsErrorFlag = 1
HttpErrorFlag = 2
SpamFooNotNeeded = 4
SpamFooBypassFlag = 8
IsSpamFlag = 16
ShouldEscalateFlag = 32
ClassPrimaryFlag = 64
ClassPromotionsFlag = 128
ClasUpdatesFlag = 256
ClassTransactionsFlag = 512
ClassUnknownFlag = 1024
MaxSizeToScan = 10000000
# *************************************************************
hdrdict = {}
sent_to = []
emlfile, TrxID, hdrdict, sent_to = ParseHeader()
# ****************************************
# BEGIN: Decide whether to call SpamFoo
# ****************************************
CheckSpamFoo = False
if 0 > 1:
exit(SpamFooNotNeeded)
fileobj = open(emlfile,"rb")
filebytes = fileobj.read()
if len(filebytes) > MaxSizeToScan:
exit(SpamFooNotNeeded)
# ****************************************
# END: Decide whether to call SpamFoo
# ****************************************
# ****************************************
# Invoke SpamFoo
# ****************************************
jsonquery={
"path": emlfile,
"ipAddress": hdrdict['connectedip'],
"senderEmail": hdrdict['EnvFrom']
}
results = AskSpamFoo(servername,jsonquery,filebytes)
# ****************************************
# Process SpamFoo results into a bitmap
# ****************************************
rsltflag = 0
if 'httperror' in results:
if results['httperror'] == True:
rsltflag = rsltflag | HttpErrorFlag
if 'isSpam' in results:
if results['isSpam'] == True:
rsltflag = rsltflag | IsSpamFlag
if 'bypassReason' in results:
if isinstance(results['bypassReason'],str) == True:
if len(results['bypassReason']) > 0:
rsltflag = rsltflag | SpamFooBypassFlag
if 'shouldEscalate' in results:
if results['shouldEscalate'] == True:
rsltflag = rsltflag | ShouldEscalateFlag
if 'classification' in results:
if results['classification'] == 'primary':
rsltflag = rsltflag | ClassPrimaryFlag
elif results['classification'] == 'promotions':
rsltflag = rsltflag | ClassPromotionsFlag
elif results['classification'] == 'updates':
rsltflag = rsltflag | ClasUpdatesFlag
elif results['classification'] == 'transactions':
rsltflag = rsltflag | ClassTransactionsFlag
else:
rsltflag = rsltflag | ClassUnknownFlag
# End of SpamFoo Checks
exit(rsltflag)