Yesterday around 4:30 AM, suddenly all traffic to my server started identifying as the Gateway IP. Under normal operations when connections come in you can see the originating IP, and allow, block, filter based on that. But in this case everything was showing as my Gateway IP. This of course means just about every email coming in would fail SPF. But even more troublesome, is I had SMTP authentication bypass enabled for my entire network, since any connections from the internal IPs should only come from my applications and be considered trusted. But the problem is I included my entire /26 which includes the gateway IP. This meant that every connection coming in was suddenly bypassing authentication. Within a few hours the bots figured this out and blasted thousands of spam emails out.
Once I figured out what was happening, I removed auth bypass from the gateway IP. Now with the flood of connections the IP was blocked due to failed attempts, which since it was the only IP accessing the box affected everyone. Even I couldn't login to smartermail ( too many attempts ), rebooting the service, and restarting the App Pool didn't help either.
At first I feared my entire server had been compromised. And since everything even my own office IP was showing as the Gateway IP, I couldn't get into my server through normal RDC. Luckily I could still access it via IPMI. I searched the entire system and found nothing installed or running that shouldn't be. The issue was happening before the data was reaching my server. I called the Datacenter where I Colo my servers, and they were looking into it. As of 5:30am this morning everything started routing correctly again. I am still waiting on an explanation from the datacenter on the cause.
So just a warning to everyone, do not trust the Gateway IP. If anyone has ever run into this issue before, please share what the cause was. This happend to both my server and a client's server, at the same time. Both at the same datacenter. They are both using IPs from the same /22. I am running 1 network on my server, and my client is running 3 networks on his. When the connections would come in they would also be the gateway IP of whatever the original destination IP was.