RBL lookup fails but mail delivered anyway

Question asked by Dave Lerner - 10/9/2014 at 5:32 PM

Unanswered

Hi,

I've got the spam settings on 12.4.5364 set up pretty much like how Bruce recommends. However, I see the following in my logs:

[2014.10.09] 18:05:58 [19283] Delivery started for xxxxx at 6:05:58 PM
[2014.10.09] 18:06:01 [19283] Spam check results: [BARRACUDA - BRBL: passed], [CBL - ABUSE SEAT: passed], [HOSTKARMA - BLACKLIST: passed], [HOSTKARMA - BROWNLIST: passed], [HOSTKARMA - YELLOWLIST: failed], [SORBS - ABUSE: passed], [SORBS - DYNAMIC IP: passed], [SORBS - PROXY: passed], [SORBS - SMTP: passed], [SORBS - SOCKS: passed], [SPAMHAUS - CBL: passed], [TRUNCATE: passed], [UCEPROTECT LEVEL 1: passed], [UCEPROTECT LEVEL 2: passed], [UCEPROTECT LEVEL 3: passed], [VIRUS RBL - MSRBL: passed], [_REVERSEDNSLOOKUP: passed], [_SPF: Pass], [_DK: None], [_DKIM: Pass], [SURBL - ABUSE BUSTER: passed], [SURBL - JWSPAMSPY: passed], [SURBL - MALWARE: passed], [SURBL - PHISHING: passed], [SURBL - SPAMASSASSIN: passed], [SURBL - SPAMCOP: passed]
[2014.10.09] 18:06:04 [19283] Starting local delivery to xxx@somelocaldomain
[2014.10.09] 18:06:04 [19283] Delivery for xxx to xxx@somelocaldomain has completed (Delivered) Filter: None
[2014.10.09] 18:06:04 [19283] End delivery to xxx@somelocaldomain
[2014.10.09] 18:06:04 [19283] Delivery finished for xxx at 6:06:04 PM [id:76219283]

In the log excerpt above, the mail failed the hostkarma yellowlist, but was still showt as "delivered" to the user's mailbox. If I search the word "failed" in the delivery log (I have detailed logging enabled), I get a ton of entries similar to this...where the inbound message "fails" an RBL, but then the log entry for that message shows it as delivered.

I must be either misunderstanding what the log entries mean, or something is wrong with my config. I have virtuall all the RBLs (as you can see from the log entries) enabled, and configured for SMTP blocking, almost exactly as shown in Bruce's document.

Any ideas what is going on?

Thanks!

steve

18 Replies

Reply to Thread

Bruce Barnes Replied

10/9/2014 at 7:21 PM

If you look at the message header, you should see a spam score, placed in there by SmarterMail.

What is the spam score and what is your delete set to?

Do you have hostkarma yellow list set to weight or SMTP delete?

Bruce Barnes ChicagoNetTech Inc brucecnt@comcast.net Phonr: (773) 491-9019 Phone: (224) 444-0169 E-Mail and DNS Security Specialist Network Security Specialist Customer Service Portal: https://portal.chicagonettech.com Website: https://www.ChicagoNetTech.com Security Blog: http://networkbastion.blogspot.com/ Web and E-Mail Hosting, E-Mail Security and Consulting

Dave Lerner Replied

10/10/2014 at 11:46 AM

I have more intel for you, but first, I thought that if it got a hit on an RBL, the spam score is irrelevant, it should just be immediately rejected, correct? I have those RBLs set to Enable for SMTP blocking, and the blocking set up with an incoming weight of 15, enabled, and that's the only one checked. See my attached images of my config.

I have a full cicruit one where the mail got through this morning as an example. This particular example is one where I actually want the email, ironically. The spam score was 10, however, it also got a positive hit on HostKarma yellow, and at the time, I had that set for enable for SMTP blocking, so it *should* have rejected it, shouldn't it?

Here are images of my setup:

The header is below, and images are attached. And here is the corresponding log entry:

[2014.10.10] 08:13:23 [12721] Delivery started for faithfromthekitchn-dlirill1utfituiu1j@cmail2.com at 8:13:23 AM
[2014.10.10] 08:13:31 [12721] Spam check results: [BARRACUDA - BRBL: passed], [CBL - ABUSE SEAT: passed], [HOSTKARMA - BLACKLIST: passed], [HOSTKARMA - BROWNLIST: passed], [HOSTKARMA - YELLOWLIST: failed], [SORBS - ABUSE: passed], [SORBS - DYNAMIC IP: passed], [SORBS - PROXY: passed], [SORBS - SMTP: passed], [SORBS - SOCKS: passed], [SPAMHAUS - CBL: passed], [TRUNCATE: passed], [UCEPROTECT LEVEL 1: passed], [UCEPROTECT LEVEL 2: passed], [UCEPROTECT LEVEL 3: passed], [VIRUS RBL - MSRBL: passed], [_REVERSEDNSLOOKUP: passed], [_SPF: Pass], [_DK: None], [_DKIM: Pass], [SURBL - ABUSE BUSTER: passed], [SURBL - JWSPAMSPY: passed], [SURBL - MALWARE: passed], [SURBL - PHISHING: passed], [SURBL - SPAMASSASSIN: passed], [SURBL - SPAMCOP: passed]
[2014.10.10] 08:13:32 [12721] Starting local delivery to myaddress@domain.com
[2014.10.10] 08:13:32 [12721] Delivery for faithfromthekitchn-dlirill1utfituiu1j@cmail2.com to myaddress@domain.com has completed (Delivered) Filter: None
[2014.10.10] 08:13:32 [12721] End delivery to myaddress@domain.com
[2014.10.10] 08:13:32 [12721] Delivery finished for faithfromthekitchn-dlirill1utfituiu1j@cmail2.com at 8:13:32 AM [id:839903212721]

Received: from CO2PR04MB650.namprd04.prod.outlook.com (10.141.198.145) by
CO2PR04MB652.namprd04.prod.outlook.com (10.141.198.149) with Microsoft SMTP
Server (TLS) id 15.0.1044.10 via Mailbox Transport; Fri, 10 Oct 2014 14:26:47
+0000
Received: from CO2PR04MB652.namprd04.prod.outlook.com (10.141.198.149) by
CO2PR04MB650.namprd04.prod.outlook.com (10.141.198.145) with Microsoft SMTP
Server (TLS) id 15.0.1044.10; Fri, 10 Oct 2014 14:26:46 +0000
Received: from POP by CO2PR04MB652.namprd04.prod.outlook.com with Microsoft
Exchange Server; Fri, 10 Oct 2014 14:26:42 +0000
Return-Path: <faithfromthekitchn-dlirill1utfituiu1j@cmail2.com>
Received: from mx29.h.outbound.createsend.com (mx29.h.outbound.createsend.com [204.75.142.29]) by mail.f18.net with SMTP;
Fri, 10 Oct 2014 08:13:22 -0600
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=cm; d=thekitchn.com;
h=From:To:Reply-To:Date:Subject:MIME-Version:Content-Type:List-Unsubscribe:Message-ID; i=kitchnupdate@thekitchn.com;
bh=os0XSlzpdW0KSqmS8ULE7YlzsLI=;
b=b32TrGufVAfcKPF9yb9i7mTin1EvJ8XS+1bd+r/0y9hiL/NZi/eDdS6i78KIfADvlWz/gkLyse5m
KmAcNObeaesiOuwkLZESoJW2HiT3yW1IZDx3+qe3enSJevBOQfMw
From: "Faith from The Kitchn" <kitchnupdate@thekitchn.com>
To: "myaddress@domain.com" <myaddress@domain.com>
Reply-To: kitchnupdate@thekitchn.com
Date: Sat, 11 Oct 2014 01:06:22 +1100
Subject: =?utf-8?Q?The_10_Best_Trader_Joe?=
=?utf-8?Q?=e2=80=99s_Pumpkin_Goodies_|_Ghost_Pears_|_Baked_Potato?=
=?utf-8?Q?_Casserole?=
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="_=aspNetEmail=_3b12f615d8aa47da8406575d600c6244"
X-Mailer: Create Send
X-Complaints-To: abuse@cmail2.com
X-Feedback-ID: CA1-j-dlirill:LI1-j-iljkjr:CL1-j-bihid:createSEND
List-Unsubscribe: <http://unsub.cmail2.com/t/j-u-dlirill-utfituiu/>
Message-ID: <cm.010622.dlirill.utfituiu.j@cmail2.com>
X-SmarterMail-Spam: HostKarma - Yellowlist, SPF_Pass, DK_None, DKIM_Pass
X-SmarterMail-TotalSpamWeight: 10
X-MS-Exchange-Organization-AuthSource: CO2PR04MB652.namprd04.prod.outlook.com
X-MS-Exchange-Organization-AuthAs: Anonymous
X-Auto-Response-Suppress: All
X-MS-Exchange-Organization-Sharing-Instance-Guid: 1a604ff4-8573-4443-a597-945d01facbd4
X-MS-Exchange-Organization-Network-Message-Id: 4ce6650c-363c-4c23-1a20-08d1b2ad58ba
X-MS-Exchange-Organization-AVStamp-Service: 1.0
X-Exchange-Antispam-Report-Test: UriScan:;
X-Forefront-Antispam-Report: LANG:en;LANG:en;
X-MS-Exchange-Organization-MessageDirectionality: Incoming
X-Microsoft-Antispam: UriScan:(61325)(34239);
X-Microsoft-Antispam: BCL:0;PCL:0;RULEID:(120001);SRVR:CO2PR04MB650;

Dave Lerner Replied

10/10/2014 at 5:48 PM

Wow...well, best I can tell, Smartermail is just simply not rejecting mail from IPs that are on RBLs...I mean, take a look at this log snippet:

[2014.10.10] 15:23:49 [22048] Delivery started for diabetesmiracle@nenothe.eu at 3:23:49 PM
[2014.10.10] 15:24:20 [22048] Spam check results: [BARRACUDA - BRBL: passed], [CBL - ABUSE SEAT: passed], [HOSTKARMA - BLACKLIST: failed], [HOSTKARMA - BROWNLIST: passed], [SORBS - ABUSE: passed], [SORBS - DYNAMIC IP: passed], [SORBS - PROXY: passed], [SORBS - SMTP: passed], [SORBS - SOCKS: passed], [SPAMHAUS - CBL: passed], [TRUNCATE: passed], [UCEPROTECT LEVEL 1: passed], [UCEPROTECT LEVEL 2: passed], [UCEPROTECT LEVEL 3: passed], [VIRUS RBL - MSRBL: passed], [_REVERSEDNSLOOKUP: passed], [_SPF: Pass], [_DK: None], [_DKIM: None], [HOSTKARMA - YELLOWLIST: passed], [SURBL - ABUSE BUSTER: passed], [SURBL - JWSPAMSPY: passed], [SURBL - MALWARE: passed], [SURBL - PHISHING: passed], [SURBL - SPAMASSASSIN: passed], [SURBL - SPAMCOP: passed]
[2014.10.10] 15:24:20 [22048] Starting local delivery to andrea@ourdomain.com
[2014.10.10] 15:24:20 [22048] Delivery for diabetesmiracle@nenothe.eu to andrea@outdomain.com has completed (Delivered) Filter: None
[2014.10.10] 15:24:20 [22048] End delivery to andrea@ourdomain.com

I have (as the pictures from the previous post show), all the RBLs set to "enable for incoming SMTP blocking" ...and that log above clearly shows a failure for Hostkarma Blacklist....yet the mail is still delivered!

Can someone from SmarterTools chime in on this? I'm going to ask my client if we can open a ticket on this one.

Dave Lerner Replied

10/10/2014 at 5:53 PM

Any thoughts on this Bruce?

Dave Lerner Replied

10/11/2014 at 12:17 PM

I think I see the problem/issue, Bruce. And it sort of stems from the extremely general nature of the SmarterMail docs...just glosses over a whole bunch of things. In this case, now looking at the "Incoming Weight Threshold" value, on the SMTP Blocking tab, I have the first value ticked, and set to 15...and um, maybe I'm just guessing at this, but I wonder if that means there has to be a spam value of at least 15, *regardless* of whether one of those contributors was an RBL hit!....good grief, if that's the case, this changes everything.

Using your scheme, if I have the RBLs set at 10, and enabled for blocking, then the threshold at 15, and if my *guess* is correct, then a single RBL hit will *NOT* cause an email rejection...since it only contributed a 10 to the score.

This is a little sideways to what your document says; and to what the SM manual implies (it isn't exactly crystal clear).

What I want, is if a single hit happens on an RBL, I want an *immediate* rejection...not just a score input. This is especially true if they hit on Barracuda, and only Barracuda. So, to effect this, one would want to set Barracuda arbitrarily high, and certainly higher than 15 (the threshold).

I've submitted a ticket to Smartertools to get a ruling on this. Too bad you have to use a ticket just to get answers to unclear documentation. Bummer.

Bruce Barnes Replied

10/11/2014 at 2:10 PM

Many of the issues raised by @Dave Lerner can be disposed of by simply reading what I wrote in my antispam document and fully implementing it, and that's where many SmarterMail operators fail in their implementation. Let me explain.

SmarterMail server operators MUST make certain that user's and domain admins CANNOT override settings. Disable the capability of both domains and users to set their own spam settings and push them out to the domains and users.

Anyone who is trying to control spam who continues to allow their domain operators and/or individual users to override spam settings is shooting themselves in the foot, and the solution is going to hurt a whole lot more than the spam.

I am betting that the issue you highlight in your comment,

David Lerner quote regarding spam not being filtered when it should be

is probably related to the fact that the DOMAIN or USER who is receiving the e-mail you are trying to filter has overridden your spam settings with settings of their own
FORGET WHITELISTNG - DON'T USE IT AND DON'T ALLOW IT. Allowing user and domain admins to whitelist only creates backdoors by which spam can come in and, because whitelisting is given priority over spam filters, you'll never control the spam.

In troubleshooting cases like the one described in item #1 above, I find that a domain or IP address has been whitelisted about 15% of the time.

The biggest reason for whitelisting is to bypass SMTP authentication or the resolution of forwarding issues. There is no reason for NOT using SMTP authentication now. Anyone who bypasses SMTP authentication, for any reason, is setting themselves up to be used as a spam generator,
GREYLIST EVERYTHING, FOR EVERYONE - NO EXCEPTIONS. Greylisting's initial deny period can be set as low as one (1) minute and still be effective.

Remember, e-mail is not instant messaging, and forcing someone to wait a few minutes for a first-time sender's e-mail to be delivered is not the end of the world.

Learn how greylisting works and educate your users and customers.
Within SmarterMail, tgere are a couple of ways you can filter spam:

SCORE BASED filtering;
SMTP BLOCKING;
CYRENE Premium antispam;
Spam Assassin pattern based matching;
BAYESIAN filtering, and;
RBL / URIBL / rDNS / SPF testing.

As you can see in the list above, all but RBL / URIBL / rDNS / SPF testing, are marked in RED, because all we use are RBL / URIBL / rDNS / SPF testing.

By combing the RBL / URIBL / rDNS / SPF testing with Greylisting, we eliminate greater than 95% of all spam.

Again, and I cannot repeat this enough, we DO NOT WHITELIST and we DO NOT ALLOW USERS or DOMAINS to override spam settings.

We also run DMARC testing on all incoming messages.

For more information on our antispam measures, see the following documents:

SmarterMail Antispam Settings Document (this is a fast moving topic, so check back frequently)
What is Greylisting

Bruce Barnes

ChicagoNetTech

Portal | Facebook | LinkedIN | Website

Dave Lerner Replied

10/12/2014 at 7:49 PM

Actually, the domains are set so they cannot override.

The user account that the mail in question was delivered to was *my own* ...it was my own personal domain that I use for testing....and its settings are not overidden.

We do not allow admins to override.

We don't use whitelisting; this example was to my own personal test account/domain, and *no* whitelisting was involved.

We use greylisting.

I think my first comment to you probably explains what is happening...and I'm using a ticket to verify that. What I'm talking about is the reply to you regarding how the RBL and "enable for deletion" work in concert with the "incoming threshold". The RBL hit quite simply did not cause a rejection here, and I think that is because the threshold was 15 and the RBL was assigned 15. We'll see what smartermail says, since I am forcing an answer by spending a ticket.

I Will post my findings here.

Steve Reid Replied

10/14/2014 at 5:27 AM

It is common knowledge that Smartermail works the way you see. The spam checks are all score based and completely configurable...

Seems like a waste of a ticket when educating yourself on smartermail by actually reading Bruce's guide would have explained everything.

Dave Lerner Replied

10/14/2014 at 7:41 AM

I read Bruce's document; every word. The comment: "Anything checked in the “ENABLE FOR INCOMING SMTP BLOCKING” column will
UNCEREMONIOUSLY DELETE an incoming message which meets the criteria" is what is curious. An incoming message is deleted...when?...when it is found on an RBL? or when it is found on an RBL *and* it adds enough to the score to meet the threshold? The documentation isn't clear on this. I demonstrated that even with an RBL positive hit, the mail was *not* deleted. That's why I opened the ticket.

Steve Reid Replied

10/14/2014 at 7:46 AM

Yeah but it doesn't say anywhere that it will be deleted. SM uses a scoring system and the email is only deleted if it meets the threshold level that you set. Bruce's document talks about the SMTP blocking settings.

Dave Lerner Replied

10/14/2014 at 8:55 AM

That is precisely what I am getting at. I have worked with over a dozen 'nix type mail servers and have written my own smtp rbl software in the past so I'm quite familiar with this; however, the smartermail system works differently, obviously. What I want is SMTP *blocking* ...I want the smtp session to be dropped, immediately, for any RBL hit. And that is not what's happening. The excerpt above leads me to think it will delete an email if an RBL hit is positive...that's what it says, but it doesn't.

Bruce Barnes Replied

10/14/2014 at 9:31 AM

Dave; If you will contact me directly, I will be happy to take a look at your SmarterMail antispam settings. My document contains contact information.

Joe Wolf Replied

10/14/2014 at 3:26 PM

This is very simple and SmarterMail is doing exactly what it's supposed to do. The message failed HostKarma Yellow and you have a weight of 10 assigned to that test, and you have it enabled for SMTP blocking.

You have your SMTP Blocking threshold set at 15, so it will not SMTP block any message unless it has a weight of 15 or more. In your case the message had a weight of 10 because if failed HostKarma Yellow. 10 is less than 15 so the message was not SMTP blocked.

If you're going to SMTP Block at a weight of 15 there is no point of putting a weight of 45 for BRBL. Anything over 15 is irrelevant.

-Joe

Thanks, -Joe

Dave Lerner Replied

10/15/2014 at 10:47 AM

Thank you Joe. That is an answer that I now realize is correct. I read the documentation for SM and Bruce's document and the SM documentation was unclear in the relationship between the two metrics. Having written over a 100k lines of this sort of code myself, I was biased to the way most of this code works...i.e., an RBL hit is death...the SMTP connection is immediatly *dropped* an the mail is not even received. SM is different, it actually conducts the SMTP conversation, accepts the email, and then does its thing. If you go looking at other MTAs, you will see they do not do this. The SMTP socket is held open *during* the RBL check...if there is a hit, the socket is closed. So the mail is not even accepted...there are no headers, etc...nothing.

So, my entire post, and the ticket, was simply to get this answer. Now that I know this, the solution is trivial.

The ticket, btw, was a waste...they SM folks did not answer my questions nor did they clarify the documentation. Predictably.

Thanks to the community for the clarification, I have my answer and I can now, correctly, configure the settings to get the server to behave the way I want it to.

ciao.

Dave Lerner Replied

10/19/2014 at 1:35 PM

Thank you to all who added constructive, positive, and helpful input. I have used the open ticket to ask a question to the SM folks the following question:

In regard to RBL return value(s), what I am looking for is a way to configure smartermail to do something *regardless* of the value returned. I think I asked about this in the ticket, but let me rephrase it...so, many RBLs return *multiple* values...not just a 127.0.0.1. I'm asking if I *un* check, or clear, that Required Lookup Value checkbox...what happens? If the RBL does not return a value, meaning that the IP was not found, what happens? If the RBL returns *any* other value, other than nothing, what happens (when the checkbox is cleared)?

Most MTAs, including the one that I developed, can check the RBL return value, or *lack* of a return value, and act accordingly. If there is no answer, or a non-listed return value, then fine, the mail is passed on up the stack to the next (if any) check. If, however, there is a 127.0.0.x return value, then you can further act...you can just block right away, or you can perform different actions based on the actual returned value.

The problem with this in SM is that, based on the documentation, you have to list *each* RBL return value, *separately* ...which means that *each* one of those is a completely unnecessary lookup circuit. Take, for example, Spamhaus. Instead of just checking Spamhaus XBL, *once*, for *any* value, as they suggest, with SM, you must (just as Bruce's document shows) list each Spamhaus Zen return value, seperately...meaning for each email, you have to have SM roundtrip like 9 times...this is a waste! You really only need to check Zen only *once*...and *any* 127.0.0.x value you receive, you want to add in your desired weight, and act on it.

In answer to this question, I was told that the question had been sent to the developers....I'll let you know what I find.

Joe Wolf Replied

10/19/2014 at 4:56 PM

Most RBL's will return 127.0.0.2 if the IP Address is on the list. Some can return multiple values (like Zen). If the RBL query of the IP Address is not listed the RBL will return a NXDOMAIN (domain does not exist). So if your intent is to block any IP Address listed on Zen then there's no need to have any Required Lookup Value and the Enabled box not checked.

In the example above if Zen returns ANY value other than NXDOMAIN then the test failed and the weight you assigned to the test will be applied to the message.

If it really matters to you, or you want to set up different weights if the message failed SBL, CSS, XBL, or PBL you can define additional tests, but SmarterMail will make only one DNS query.

If you wanted to check the Zen RBL but apply weight only if the message is listed on the CSS list then you would put a Required Lookup Value as 127.0.0.3 and check the Enabled box.

So, the bottom line is that if you query an RBL and the DNS query receives any value other than NXDOMAIN then SmarterMail will apply the weight to the message unless you have specified a Required Lookup Value and checked the Enabled box and those exact criteria are met.

There are exceptions to every rule of course... in the case of the URIBL at uribl.com will return a lookup value of 127.0.0.1 if your DNS is blocked from using their service (this includes most public DNS resolvers). Generally if you ever get a 127.0.0.1 reply it means you're blocked and does not mean the IP or URL, etc. is listed on the test (a false positive). You should never see a 127.0.0.1 from ANY RBL or URIBL I know of and it would indicate a problem on your end.

-Joe

Thanks, -Joe

Dave Lerner Replied

10/24/2014 at 7:21 AM

I mis-spoke when I said 127.0.0.1 in my comment...obviously that was in error.

I did hear back from the developers...and the response is interesting. They note that RBLs *can* return values other than 127.x.x.x ...so they don't recommend clearing the Enabled checkbox...since this would mean any value returned by the RBL will cause an added score. Some RBLs return, for example, 255.255.255.0 to indicate a temporary DNS block due to a high volume of queries. So, you have to research each one, and set up your system accordingly.

Eric Bourland Replied

2/28/2015 at 1:59 PM

This was a very useful thread. Discussion, and review of documentation, makes me feel like I have some control. I'm getting some weird RBL results in SmarterMail 13.2 these days -- but this thread gave me some perspective. Eric

Back to Community Threads

Please leave this box unchecked

18 Replies

Reply to Thread

Tags

Related Knowledge Base Articles