I doubt its smartermail that is the cause, but rather a lack of security on the server.

What version of Windows Server is running, is it patched, does it sit behind a firewall which prevents access to all ports other than what are required to be opened.

I also don't think it is a SmarterMail cause.

Windows Server 2012 R2

The recent update was not installed.

We use the Windows Firewall.

We open only SMTP/IMAP/POP3/ RDP and smartermail ports only.

OK, definitely is the RDP port...

Many ransomware use it to hack Servers.

Use some RDP security software or, better, don't open RDP.

Do NOT open RDP ports to the outside. Ever. 

Not sure but it does sound very much like an RDP issue. 

we do not use the default RDP port and the password is very complicated. 

There's malware that scans ALL the open ports to verify if there's a RDP Server behind...

If they find it they start to work on vulnerability and if they find a vulnerability, the password is not needed...

Get a hardware firewall and block ALL unnecessary ports to the world, then use a VPN to access secured ports such as RDP.

I agree with Christopher.

That’s exactly what we do... and only allow RDP access via VPN. 

SmarterMail does not interact with the server at this level, and this behavior indicates a likely root kit on the machine. I agree with previous comments regarding RDP being a potential source of trouble. If you absolutely must have RDP access open I recommend setting up a VPN as has already been suggested, or configure your firewall to drop all attempts on port 3389 if they don't match one of your office IP addresses. 

Aside from the RDP vector, it is possible another application on the server that has not been patched allowed a remote user to escalate their privilege levels and make server level changes. I would start by applying updates to all software running on the server including SmarterMail, Windows and .NET updates, third party applications, etc at the very least. 

Yep yep, we have 3 VYOS-based routers, 2 mains and 1 failover, we use OpenVPN, but a few of us are testing Wireguard which has been a far far better experience than OpenVPN. It's amazing how many systems I see when I do IT consulting where they have RDP/RPC etc. just saying Hi to the public.

We firewall off all RPC/NETBIOS as well as KMS ports on our firewall to ALL servers, including VPS and client dedicated machines, we do not open those period. If a client needs those ports open, buy a VPN addon. We'd love to lock out RDP completely too, but that's not realistic for customers.

That is probably how they managed to get into your server.

The only ports that should be open are those relating to your smartermail service - everything else should not be accessible from outside of a vpn.

There are websites that gather this information and the info is freely available - https://www.shodan.io/  being one of them.