Encrypted SMTP port requires common algorithm

Problem reported by Stephen Clarke - 9/14/2021 at 8:15 PM

Submitted

Hi all, this is a follow-on from a question yesterday.

I am trying to migrate our main server (including SM) across to an Azure VM. They block port 25 by default, I am trying to have this lifted, but it apparently requires establishing billing history first.

If unsuccessful now, the fallback strategy is to migrate most services across to Azure, but keep SM on the existing server for the interim, and use it externally from there.

Our web apps on the server are the only consumers of SmarterMail as an onboard mail server, and I have never had to setup encryption on ports before, as we could simply access it in the clear via localhost. But if we have to access it remotely obviously SSL/TLS support will be essential.

We have SM 12.5, and need SMTP and POP only. We are using System.Net.Mail.SmtpClient for some operations (so must use STARTTLS via port 587, not port 465). Also the jstedfast MailKit SMTP/POP components.

I experimented with adding encryption under SM > Settings > Ports, by making Encryption=TLS for the 587 Submission Port. I am using a SSL certificate generated for the same www subdomain that the mail server is being accessed through (if that is important?).

But then SMTP setup with Outlook 365 fails when configured for STARTTLS, and SmarterMail reports the following in the logs:

[2021.09.15] ... rsp: 220 ...
[2021.09.15] ... connected at 9/15/2021 12:54:26 PM
[2021.09.15] ... cmd: EHLO OFFICENUC
[2021.09.15] ... rsp: ... Hello [...]250-SIZE 31457280250-AUTH LOGIN CRAM-MD5250-STARTTLS250-8BITMIME250 OK
[2021.09.15] ... cmd: STARTTLS
[2021.09.15] ... rsp: 220 Start TLS negotiation
[2021.09.15] ... Exception negotiating TLS session: System.ComponentModel.Win32Exception (0x80004005): The client and server cannot communicate, because they do not possess a common algorithm
[2021.09.15] at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface SecModule, String package, CredentialUse intent, SecureCredential scc)
[2021.09.15] at System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse credUsage, SecureCredential& secureCredential)
[2021.09.15] at System.Net.Security.SecureChannel.AcquireServerCredentials(Byte[]& thumbPrint)
[2021.09.15] at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output)
[2021.09.15] at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count)
[2021.09.15] at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
[2021.09.15] at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
[2021.09.15] at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
[2021.09.15] at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
[2021.09.15] at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
[2021.09.15] at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate)
[2021.09.15] at MailService.TcpServerLib.Common.PooledTcpItem.ConvertToSSL(IPBindingPort setting, Log log, String sessionId)
[2021.09.15] at MailService.TcpServerLib.SMTP.SMTPSession.#W8()

First I thought it might be TLS version mismatch, but then I noticed it lacks a "common algorithm".

What should I be looking for here?

12 Replies

Reply to Thread

Sébastien Riccio Replied

9/14/2021 at 10:59 PM

I think you are on the right track with TLS version mismatch.

As far as I know this kind of error happens when both client and server can't agree with a TLS version or maybe cipher.

Like for example your server is restricted to TLS1.2 and higher and the client (an old one for example) only supports up to TLS 1.1, so they can't agree on a way to communicate securely.

This can also happen when it's server to server communication. If the local server and remote server can't agree on a TLS version to use. In this case I think SmarterMail retries without using SSL.

Sébastien Riccio System & Network Admin https://swisscenter.com

Stephen Clarke Replied

9/14/2021 at 11:22 PM

Thanks Sebastien, after further research, I am thinking so too.

I know that the VM itself currently enables only TLS 1.1 and 1.2 in the SCHANNEL branch of the registry. This is to ensure compliance with PCI scans in the past.

But maybe SM 12.5 supports only TLS 1.0, so there is no intersection? This might explain the error.

I know that SM 12.5 is very old now. We're happy to upgrade if needed but looking for a smooth pathway, would prefer <= v15 for the time being, just because the interface is more familiar.

Would be great to get some insight from staff if possible, on the minimum version needed to achieve contemporary SMTP/STARTTLS and POP security, and how that upgrade pathway looks from 12.5.

Kyle Kerst Replied

9/15/2021 at 1:38 PM

Employee Post

I think you're on the right track! These appear to be mismatches on TLS negotiation. A quick way to resolve that is installing IISCrypto from Nartac Software which has a Best Practices button you can apply on both ends. That said, older versions of Windows/SM may not support the latest and greatest, so I recommend testing this before you roll it out full time.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Stephen Clarke Replied

9/15/2021 at 4:18 PM

Thanks Kyle. IIS Crypto 3.2 recommends the following Best Practices:

Protocols

(TLS 1.0)

TLS 1.1

TLS 1.2

Ciphers

(Triple DES 168)

AES 128/128

AES 256/256

Hashes

MD5

SHA

SHA 256

SHA 384

SHA 512

Key Exchanges

(Diiffie-Hellman)

PKCS

ECDH

All of these are active on the client.
All except the bracketed items are active on the server (these are the deprecated ones disabled for PCI compliance).

So there should be plenty of overlapping combinations.

Is there anything in SM 2.5 that would make it unable to use any of these? How should I diagnose this?

Kyle Kerst Replied

9/15/2021 at 4:58 PM

Employee Post

It does look like you have a good amount of overlap, so I'm not really sure what is holding you up there. Do these versions/types show up in a test from the outside? The only thing I can think of is that the Server OS, network/firewall systems, or SmarterMail itself don't support it along the way. I know that in our latest release we offer the ability to configure those protocol versions or set it to use the system level settings, so this might be a protocol support issue in SM. Unfortunately I don't have a public test server I can roll 12 out on though. Are you familiar with OpenSSL? You could use this to probe the actual SmarterMail ports to see if they answer back with the correct versions.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Stephen Clarke Replied

9/16/2021 at 2:50 PM

Hi Kyle, thanks for the OpenSSL pointer. Apparently the following command is recommended

openssl s_client -connect mail.server.com:587 -starttls smtp

I've just tried this out on my submission port, and got the following

============================================

CONNECTED(0000018C)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 203 bytes and written 354 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

============================================

So is this a problem with the certificate? But the Certificate Path is correct and the Verify Certificate function gives it a pass.

Kyle Kerst Replied

9/16/2021 at 2:56 PM

Employee Post

You're very welcome! I'm not sure if this is telling us there is an error or not. It initially sounds like it might be missing the private key portion of the certificate, but then says it passes validation!

- When you bound the certificate did you bind it to a PFX file export with the private key included?

- If the certificate is passing elsewhere, this would lead us back to the suite versions or auth types again unfortunately.

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Stephen Clarke Replied

9/16/2021 at 3:07 PM

It's bound to a CER file as issued by the CA.

Should it be a PFX exported from the Certificate Store?

Stephen Clarke Replied

9/16/2021 at 3:19 PM

OK, and further to that, it seems that PFX applies only to SM >=16.

Earlier version do need a simple CER file

https://portal.smartertools.com/kb/a3195/configure-ssl-tls-to-secure-smartermail.aspx

Just to be safe, I followed these instructions to create a base64 CER exported from MMC.

Still the same OpenSSL assessment as before.

Kyle Kerst Replied

9/16/2021 at 3:25 PM

Employee Post

CER might be part of our issue, I know we switched to recommending PFX exports in and around v15 (I believe) because of issues like these. It might be time to look at getting an upgrade rolled out. That said, the error still leads me to believe its a mismatch between the versions, but this checked out via IISCrypto so I'm not really sure! Can you submit a ticket on this so we can look at it with you?

Kyle Kerst Acting IT Manager SmarterTools Inc. www.smartertools.com

Stephen Clarke Replied

9/16/2021 at 6:41 PM

Hi Kyle, thanks. But we don't currently have active support, so the page won't let me.

We're happy to get a new license and upgrade to v15 to work on this if you think that is a pathway with good prospects.

Is v12 => v15 a smooth upgrade? Prefer to stick to the old interface if I am going to be retrofitting the existing server. I believe your licensing can apply to any previous version?

Then we would transfer SM to our new Azure VM as the current version when we finally get port 25 opened on that.