[SOLVED] ATTENTION: blocker issue with update v7412 - Outlook is no longer able to connect in IMAP and SMTP (I have not tried POP3)

Problem reported by Gabriele Maoret - SERSIS - 4/18/2020 at 12:56 AM

Resolved

Just updated from version 7405 to 7412 all our clients using Outlook have no longer been able to connect to the server to receive mail (IMAP protocol) or to send mail (SMTP protocol).

Nobody uses POP3 so I couldn't try.

The ADMINISTRATIVE log is full of errors similar to this:

09:38:28.898 TlRMTVNTUAADAAAAGAAYAJ4AAABmAWYBtgAAACAAIABYAAAACAAIAHgAAAAeAB4AgAAAAAAAAAAcAgAABYKIIgoAukcAAAAPftVohczvBwhweu7iURzQT3YAZQBjAGUAbABsAGkAbwBsAHUAYwBpAGEALgBpAHQAaQBuAGYAbwBEAEUAUwBLAFQATwBQAC0ANABSADAASgBMAFIARQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABANVxmYFJzQW6NRSle4cTJAQEAAAAAAADacihaVBXWAaxRnkijkluaAAAAAAIAFgBTAE0AQQBSAFQARQBSAE0AQQBJAEwAAQAeAFcASQBOAC0AOABIAEMAOABPAEUAVABMADYARQBUAAQAFgBzAG0AYQByAHQAZQByAG0AYQBpAGwAAwA2AFcASQBOAC0AOABIAEMAOABPAEUAVABMADYARQBUAC4AcwBtAGEAcgB0AGUAcgBtAGEAaQBsAAUAFgBzAG0AYQByAHQAZQByAG0AYQBpAGwABwAIANpyKFpUFdYBBgAEAAIAAAAIADAAMAAAAAAAAAABAAAAACAAADrtmOdSpJYTJv8yZkto5qQZnf/+uZGSb+3gx4xy3xwRCgAQAIbroV5Qd8+Med2nbBe1/7MJACgAcABvAHAALwBtAGEAaQBsADMALgBzAGUAcgBzAGkAcwAuAGMAbwBtAAAAAAAAAAAA Non è stato possibile caricare il file o l'assembly 'BouncyCastle.Crypto, Version=1.8.6.0, Culture=neutral, PublicKeyToken=0e99375e54769942' o una delle relative dipendenze. Impossibile trovare il file specificato. in MailService.Protocols.Authentication.NtlmVersion2Response.ValidateHash(String password, String digest, String domainName, String& failure, List`1& logLines, Boolean removeFromCache, Boolean removeFromCacheOnFailure) in MailService.Protocols.Authentication.AuthenticateMessage.TryParse(Byte[] data, String hostname, String& failure, List`1& logLines, Boolean removeFromCache) in MailService.Protocols.Authentication.NegotiateMessages.Parse(Byte[] data, String& response, String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache) in MailService.Protocols.Authentication.AuthenticationManager.Parse(String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache) 09:38:28.976 USER info@vecelliolucia.it L'input non è una stringa Base 64 valida poiché contiene un carattere non Base 64, più di due caratteri di riempimento oppure un carattere non corretto tra i caratteri di riempimento. in System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength) in System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength) in System.Convert.FromBase64String(String s) in MailService.Protocols.Authentication.AuthenticationManager.Parse(String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache) 09:38:39.616 TlRMTVNTUAADAAAAGAAYAKAAAABoAWgBuAAAADYANgBYAAAACAAIAI4AAAAKAAoAlgAAAAAAAAAgAgAABYKIIgYDgCUAAAAPgp+Pi25UezTmP5wPhloFUmYAZABwAGMAbwBzAHQAcgB1AHoAaQBvAG4AaQBlAGwAZQB0AHQAcgBpAGMAaABlAC4AaQB0AGkAbgBmAG8ARABBAFYASQBEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB9s55A8WQcU15VoZP5/j1MBAQAAAAAAAH+ciWBUFdYBa5NzaFY6aO4AAAAAAgAWAFMATQBBAFIAVABFAFIATQBBAEkATAABAB4AVwBJAE4ALQA4AEgAQwA4AE8ARQBUAEwANgBFAFQABAAWAHMAbQBhAHIAdABlAHIAbQBhAGkAbAADADYAVwBJAE4ALQA4AEgAQwA4AE8ARQBUAEwANgBFAFQALgBzAG0AYQByAHQAZQByAG0AYQBpAGwABQAWAHMAbQBhAHIAdABlAHIAbQBhAGkAbAAHAAgAf5yJYFQV1gEGAAQAAgAAAAgAMAAwAAAAAAAAAAEAAAAAIAAArAixmXsJvPWuZO9mGEMKtR26ZgcgS5W0XE9GDvAJ/IgKABAAWDDwQ2sUouVkXthMNOB/4AkAKgBpAG0AYQBwAC8AbQBhAGkAbAAzAC4AcwBlAHIAcwBpAHMALgBjAG8AbQAAAAAAAAAAAA== Non è stato possibile caricare il file o l'assembly 'BouncyCastle.Crypto, Version=1.8.6.0, Culture=neutral, PublicKeyToken=0e99375e54769942' o una delle relative dipendenze. Impossibile trovare il file specificato. in MailService.Protocols.Authentication.NtlmVersion2Response.ValidateHash(String password, String digest, String domainName, String& failure, List`1& logLines, Boolean removeFromCache, Boolean removeFromCacheOnFailure) in MailService.Protocols.Authentication.AuthenticateMessage.TryParse(Byte[] data, String hostname, String& failure, List`1& logLines, Boolean removeFromCache) in MailService.Protocols.Authentication.NegotiateMessages.Parse(Byte[] data, String& response, String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache) in MailService.Protocols.Authentication.AuthenticationManager.Parse(String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache) 09:38:43.132

Uninstal 7412 a reinstall 7405 solve the outlook login issue but introduce a new issue: incoming messages via SMTP are not accepted and in the SMTP LOG there are a lot of errors like this:

"Exception: code is invalid. code: 198 format: bin 32 "

The only way to get back to a fully functional situation was to recover the folder "C:\Program Files (x86)\ SmarterTools\SmarterMail" from backup and then reinstall v7405 over that.

So for now we are unable to test v7412 becuse it breaks Outlook IMAP and SMTP logins.

Gabriele Maoret - Head of SysAdmins at SERSIS

Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

18 Replies

Reply to Thread

Webio Replied

4/18/2020 at 1:33 AM

I can confirm issue with 7412 when it comes to Base-64 errors. On my end I only uninstalled 7412 and installed 7405. I don't see any errors code is invalid. Maybe this has something to do that I use outgoing and incoming gateways (which are also updated to 7412 - so primary is 7405 and gateways in/out are 7412). For update process I've updated latest beta because it fixed issue with bounces not being delivered in my specific environment.

Gabriele Maoret - SERSIS Replied

4/18/2020 at 8:36 AM

Hi Webio, the issue with incoming SMTP mail after downgrading is present if your SmarterMail server is the direct incoming server by itself.

I don't know what's the difference if you have an incoming gateway in front of it. Maybe it can be a workaround.

Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)

Robert Simpson Replied

4/18/2020 at 10:14 AM

I had the same problem with BouncyCastle not being installed. Reinstalling 7412 worked, though.

Webio Replied

4/18/2020 at 10:40 AM

I try maybe tomorrow or on monday and let you know.

Gabriele Maoret - SERSIS Replied

4/18/2020 at 1:59 PM

Hi Robert, so you did a double installation of 7412 and that has solved the issue?

Can you elaborate the procedure?

Tomorrow I'll give it a try...

Robert Simpson Replied

4/18/2020 at 2:30 PM

Yes I just installed 7412 twice using the exe, not the msi.

Gabriele Maoret - SERSIS Replied

4/18/2020 at 2:36 PM

I can't wait tomorrow... :-D

I just installed 7412 like you Robert... And it worked! No issue so far!

I mark your answer as a resolution

Neil Harvey Replied

4/18/2020 at 4:24 PM

I can confirm the same issue, mail server is direct and downgrading to 7405 on it's own does not work. Had to do as Gabriele did and restore a backup of 7405 1st before re-installing 7405.

John Marx Replied

4/19/2020 at 1:31 PM

We just had this issue. We were hoping we would've been lucky but no luck. I had to go back to 7405 as our phones starting going off right after the upgrade.

Sébastien Riccio Replied

4/19/2020 at 6:02 PM

This happens with all clients that still supports NTLM (outlook is one of them).

No issue with emClient, but as soon as a client supports NTLM and tries to use, it it fails.

I still don't get it why NTLM has been recently added to SmarterMail when most of the server and client softwares are deprecating it.

Even MS own servers doesn't announce this auth method anymore (office365) and are advising not to use it anymore.

From:

https://redmondmag.com/articles/2019/06/21/microsoft-disabling-exchange-protocols.aspx

Old Protocols
The old or "legacy" authentication protocols that Microsoft wants organizations to remove include the following, per the announcement:

Basic authentication
Digest authentication
Windows authentication (NTLM and Kerberos)

As it is the auth method that brought a lot of auth problems, I already asked why it was added but got no answer. Is it a requirement to support Outlook 2010 ? If yes we should have a toggle to disable NTLM if the server admin doesn't want it.

"NTLM is not recommended to use in general as it poses some security concerns:NTLM relay, brute forcing, and other vulnerabilities. You can read about general NTLM risks here. As a rule of thumb: try to reduce NTLM usage in your network as much as possible."

[2020.04.20] 02:55:40.567 [x.x.x.x][66803271] response: pvv7 OK CAPABILITY completed
[2020.04.20] 02:55:40.645 [x.x.x.x][66803271] command: dri8 AUTHENTICATE NTLM
[2020.04.20] 02:55:40.676 [x.x.x.x][66803271] NTLM message received TXXXXXXUAABAAAAB4IIogAAAAAAAAAAAAAAAAAAAAAKALpHAAAADw==
[2020.04.20] 02:55:40.676 [x.x.x.x][66803271] NTLM challenge message TlRMTVNTUAACAAAADAAMADgAAAAFgoki/6v9F2Ok7usAAAAAAAAAAJYAlgBEAAAACgA5OXXXXNAEEASQBMADAAMwACABYAUwBNAEEAUgBUAEUAUgBNAEEASQBMAAEADABNAEEASQBMADAAMwAEABYAcwBtAGEAcgB0AGUAcgBtAGEAaQBsAAMAJABNAEEASQBMADAAMwAuAHMAbQBhAHIAdABlAHIAbQBhAGkAbAAFABYAcwBtAGEAcgB0AGUAcgBtAGEAaQBsAAcACAAVT6hprhbWAQAAAAA=
[2020.04.20] 02:55:40.692 [x.x.x.x][66803271] NTLM message received TlRMTVNTUAADAAAAGAAYAI4AAABQAVABpgAAABQAFABYAAAADAAMXXXXXAAAAWABYAeAAAAAAAAAD2AQAABYKIIgoAukcAAAAPDVpxJ4Mm9ojqpHxhGnHEF2EAcgBlAGEAMQAzAC4AYwBvAG0AbQBhAGQAagBpAGsATABPAFYARQBNAEEAQwBIAEkATgBFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFdWzf7uncAW81yt88x1DUoBAQAAAAAAABVPqGmuFtYBDgYsrr1Unk4AAAAAAgAWAFMATQBBAFIAVABFAFIATQBBAEkATAABAAwATQBBAEkATAAwADMABAAWAHMAbQBhAHIAdABlAHIAbQBhAGkAbAADACQATQBBAEkATAAwADMALgBzAG0AYQByAHQAZQByAG0AYQBpAXXXXHMAbQBhAHIAdABlAHIAbQBhAGkAbAAHAAgAFU+oaa4W1gEGAAQAAgAAAAgAMAAwAAAAAAAAAAEAAAAAIAAAhGsS3hGkZZkmHdl6jmo4SpmIVGlKkyfv1RegWxovZxUXXXXrIUKnNpq2jWUCoJ0H49tYwkANgBpAG0AYQBwAC8AbQBhAGkAbAAwADMALgBzAHcAaQBzAHMAYwBlAG4AdABlAHIALgBjAG8AbQAAAAAAAAAAAA==
[2020.04.20] 02:55:40.692 [x.x.x.x][66803271] Exception: (PooledTcpItem.cs) Object reference not set to an instance of an object.
[2020.04.20] 02:55:40.692 [x.x.x.x][66803271] StackTrace:    at MailService.TcpServerLib.IMAP.IMAPSession.AuthNTLMResponseHandler(String encodedResponse)
[2020.04.20]    at MailService.TcpServerLib.IMAP.IMAPSession.ProcessAsyncData(Byte[] memStream)
[2020.04.20]    at MailService.TcpServerLib.Common.PooledTcpItem.ProcessData(Int32 bytes)
[2020.04.20] 02:57:22.901 [x.x.x.x][66803271] disconnected at 20.04.2020 02:57:22

Sébastien Riccio System & Network Admin https://swisscenter.com

Gabriele Maoret - SERSIS Replied

4/19/2020 at 11:34 PM

Hi Sebastien! You are right, the issue is HALF SOLVED, there are still users who report logon errors through NTLM.

SmarterTools, I had reported the problem already in phase 7409 (it was given to me by Larry to try it), but still the problem is not solved.

Please, can you take charge of the problem as soon as possible? It is very urgent!

echoDreamz Replied

4/20/2020 at 12:01 AM

It's 1AM where ST is, you probably wont hear anything for about 8 - 10 hours :)

Gabriele Maoret - SERSIS Replied

4/20/2020 at 12:03 AM

Hi Chris! I know...

Grady Werner Replied

4/20/2020 at 12:06 AM

Employee Post

We have isolated the missing DLL issue to an upgrade of the bouncycastle crypto DLL in our installer configuration. Due to a misconfiguration of the versioning of the previous update they did (the one we've been using for a few months), the update to the newest one was seen by Advanced Installer as being a downgrade, so it refused to update the file.

We have a build ready for QC to go out in the AM. In the meanwhile, uninstalling SmarterMail and reinstalling it again will properly drop the correct bouncycastle crypto DLL in the C:\program files (x86)\SmarterTools\SmarterMail\Service folder.

Side note, we are still investigating the secondary NTLM issue reported above as a high priority item.

Grady Werner SmarterTools Inc. www.smartertools.com

echoDreamz Replied

4/20/2020 at 12:24 AM

Grady, are the builds not tested before being uploaded for us to install? This is becoming quite common with SmarterMail builds and one of the reasons we do not install new builds until after a few days to see if anyone reports any issues.

Sébastien Riccio Replied

4/20/2020 at 6:03 AM

Actually on our side we are still waiting that the product is stable to update our production server to the production ready beta. In the meantime we have a small test environnement running the BETA on which we are testing bugs as they are getting resolved and new introduced bugs...

We're still not conviced we can upgrade our main server when we see such issues. It would be a total disaster if it happened on the production server used by our customers.

This is the situation since mapi beta has began and all pre-MAPI bugs that we reported and that were fixed were only applied to the mapi-BETA and not backported to the stable branch....So we can't upgrade to fix previous bugs as it would bring other problems.

Sébastien Riccio System & Network Admin https://swisscenter.com