Network routing issue caused major issue with Smartermail
Problem reported by Paul White - 4/27/2026 at 8:39 AM
Submitted
Yesterday around 4:30 AM, suddenly all traffic to my server started identifying as the Gateway IP.  Under normal operations when connections come in you can see the originating IP, and allow, block, filter based on that.  But in this case everything was showing as my Gateway IP.  This of course means just about every email coming in would fail SPF.  But even more troublesome, is I had SMTP authentication bypass enabled for my entire network, since any connections from the internal IPs should only come from my applications and be considered trusted.  But the problem is I included my entire /26 which includes the gateway IP.  This meant that every connection coming in was suddenly bypassing authentication.  Within a few hours the bots figured this out and blasted thousands of spam emails out.  

Once I figured out what was happening, I removed auth bypass from the gateway IP.  Now with the flood of connections the IP was blocked due to failed attempts, which since it was the only IP accessing the box affected everyone.  Even I couldn't login to smartermail ( too many attempts ), rebooting the service, and restarting the App Pool didn't help either.  

At first I feared my entire server had been compromised.  And since everything even my own office IP was showing as the Gateway IP, I couldn't get into my server through normal RDC.  Luckily I could still access it via IPMI.  I searched the entire system and found nothing installed or running that shouldn't be.  The issue was happening before the data was reaching my server.  I called the Datacenter where I Colo my servers, and they were looking into it.  As of 5:30am this morning everything started routing correctly again.  I am still waiting on an explanation from the datacenter on the cause.

So just a warning to everyone, do not trust the Gateway IP.  If anyone has ever run into this issue before, please share what the cause was.  This happend to both my server and a client's server, at the same time. Both at the same datacenter.  They are both using IPs from the same /22.  I am running 1 network on my server, and my client is running 3 networks on his.  When the connections would come in they would also be the gateway IP of whatever the original destination IP was.  
Kyle Kerst Replied
Employee Post
Hey Paul, sorry to hear you had such a mess on your hands with that! I was initially going to suggest things like a backup MX as a fallback delivery point, but realized it wouldn't have helped in this scenario unless you took the primary environment completely offline until the issue was resolved at the routing level. So, I think the biggest takeaway here is to be super judicious about who/what you whitelist. One compromised host or network misconfiguration can lead to big headaches! Least possible permissions, and least possible exceptions is the way to go here. 
Kyle Kerst
Lead Internal Network/System Administrator
SmarterTools Inc.
Paul White Replied
Hey Kyle,
Yeah there was no quick fix for this.  I am more curious to know if anyone else has run into this same routing issue?  I still haven't received an explanation of the cause from my Colo Provider.  That has me considering moving my server to another datacenter.
Douglas Foster Replied
Yes, it has happened to me.   The problem was eventually blamed on the underlay to our SD-WAN environment, but I was never given any details.  It actually took a couple of weeks to get it fixed.   I was both fortunate and surprised to discover that I detected relatively few problems while either good messages getting blocked or bad messages getting through.
Kyle Kerst Replied
Employee Post
I have seen this happen before but never experienced it personally. From what I understand it typically points to a NAT pinning or routing issue (configuration problem) somewhere and the causes can vary a quite a bit. They could have had a bad update to a switch, a misconfigured firewall rule, poisoned ARP caches, etc. Since they are your service provider though I don't think its crazy to ask them for a full postmortem, they should be able to provide some kind of detailed explanation for the outage. 
Kyle Kerst
Lead Internal Network/System Administrator
SmarterTools Inc.

Reply to Thread

Enter the verification text