Errors in TLS negotiation - is this our server issue?

Problem reported by Merle Wait - 4/29/2026 at 9:45 AM

Resolved

Listed below is a log that reflects an issue that we are experiencing (or seeing) on our inbound email server.
We have our SMTP 25 - setting at START TLS.
Our users are complaining that these inbound requests are delayed.... and I believe this is why.
What I am trying to determine if this is an error on our side or theirs.... and either way.. a good way to correct???
Version: SmarterMail Free 100.0.8797.17797 (Feb 1, 2024)

BUT... the same thing is happening on several other inbound email servers
the other one is running on
100.0.9575.20827 (3/20/2026)

'===================================================
[2026.04.29] 09:20:50.198 [8.31.233.208][25175233] rsp: 220 mx1.ebt.net

[2026.04.29] 09:20:50.198 [8.31.233.208][25175233] connected at 4/29/2026 9:20:50 AM

[2026.04.29] 09:20:50.198 [8.31.233.208][25175233] Country code: US

[2026.04.29] 09:20:50.224 [8.31.233.208][25175233] cmd: EHLO internalmail201.xxxxxxx.com

[2026.04.29] 09:20:50.226 [8.31.233.208][25175233] rsp: 250-mx1.ebt.net Hello [8.31.233.208]250-SIZE 699050666250-AUTH LOGIN CRAM-MD5250-STARTTLS250-8BITMIME250-DSN250 OK

[2026.04.29] 09:20:50.249 [8.31.233.208][25175233] cmd: STARTTLS

[2026.04.29] 09:20:50.249 [8.31.233.208][25175233] rsp: 220 Start TLS negotiation

[2026.04.29] 09:20:50.273 [8.31.233.208][25175233] SNI using fallback binding certificate D25_WildebtV10.pfx for (no hostname passed to SNI).

[2026.04.29] 09:20:50.299 [8.31.233.208][25175233] rsp: 554 Security failure

[2026.04.29] 09:20:50.300 [8.31.233.208][25175233] Exception negotiating TLS session: System.Security.Authentication.AuthenticationException: Cannot determine the frame size or a corrupted frame was received.

[2026.04.29] at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token) at System.Net.Security.SslStream.ReceiveHandshakeFrameAsync[TIOAdapter](CancellationToken cancellationToken) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at SmarterMail.Protocols.Common.PooledTcpItem.ConvertToSSL(db_system_binding_port setting, Log log, String sessionId) at SmarterMail.Protocols.SMTP.SMTPSession.STARTTLS()09:20:50.300 [8.31.233.208][25175233] disconnected at 4/29/2026 9:20:50 AM

7 Replies

Reply to Thread

Proto Replied

5/13/2026 at 6:42 AM

Merle:

I have also been seeing this on the inbound and am looking into it now. Did you find a cause / resolution? Thanks for posting.

SmarterMail(tm)

MAPI over HTTP - Let's flesh it out for Outlook with a full set of Exchange like features!

Merle Wait Replied

5/13/2026 at 7:31 AM

No, nothing.. it is still occurring..

Caden Izbicki Replied

5/13/2026 at 7:38 AM

Employee Post Marked As Resolution

I tested STARTTLS directly against the server using OpenSSL and the TLS negotiation completed successfully.

The server successfully negotiated:

TLS 1.2
Cipher: ECDHE-RSA-AES256-SHA384

The handshake completed normally and returned 250 OK, which indicates the SmarterMail SMTP listener and certificate binding are functioning correctly for STARTTLS.

Based on the logs provided, the error:

Cannot determine the frame size or a corrupted frame was received

is typically seen when the remote sending server starts STARTTLS but then:

sends invalid TLS data
aborts the handshake
or continues sending plain SMTP traffic after STARTTLS was initiated

At this point, this does not appear to be a general SmarterMail STARTTLS issue, especially since the same behavior is occurring across multiple servers and versions.

I would recommend checking whether the issue is isolated to specific sender IPs, gateways, appliances, or security devices performing SMTP/TLS inspection.

Caden Izbicki

System/Network Administrator

SmarterTools Inc.

www.smartertools.com

Proto Replied

5/13/2026 at 8:05 AM

Thank you Caden.

Where we see it on a SmarterMail server (latest version) being used as an incoming gateway to spool incoming messages when the primary is offline, it seems to be affecting all incoming connections. I think your assessment seems correct though. I did see a post I can no longer locate that suggested installing a certificate from another authority manually did resolve the issue (which I know may have nothing to do with the certificate itself). I will try that and report back.

Also, I am not seeing the message on the primary server.

SmarterMail(tm)

MAPI over HTTP - Let's flesh it out for Outlook with a full set of Exchange like features!

Douglas Foster Replied

5/13/2026 at 8:23 AM

Then maybe the remote system is rejecting your certificate chain because it is misconfigured on the backup server

Does openssl confirm that you are sending a server cert and intermediate cert but not any root certs?

Also check your allowed encryption methods between the primary and backup servers

Merle Wait Replied

5/13/2026 at 9:08 AM

Thank you for the research and information..
The sending server is from a well-known "company" .. so we were a bit concerned.
Outside of their servers(the sending company) really haven't seen the issue.
We use a wildcart cert .. as all of inbound mail servers are mx*.domain.net ... so all servers are using the same cert. So while that is good.. it has us questioning if the cert was bad or not.... The various MX servers are on different operating systems/boxes et al... only thing in common was the cert implementation.

Thank you for reearch.

Jeremy Wesley Replied

Today at 8:16 AM

Any update on this. Seeing this as well with N8N trying to connect to imap.

Back to Community Threads

Please leave this box unchecked

Reply to Thread

Enter the verification text