Build 6970 Unable to Login to Webmail, Manage Domains, or Impersonate Users

Question asked by Scarab - 2/15/2019 at 12:27 PM

Answered

Since we haven't heard any ideas from Smartertools Support I am hoping that someone in the Community might have some suggestions of what to try.

We upgraded from SmarterMail v16 to v17/100 Build 6970 on Tuesday and got no errors during the data conversion. However, post-conversion roughly 75% of our domains are inaccessible. Users cannot login to webmail, admins cannot manage domains nor impersonate users. Any attempt to do such results in a "404 - Not Found". There has been no change several days later.

Here is an excerpt of our SmarterMail Error Log. These three fields are repeated every time someone with an assumed corrupted domain tries to login to SmarterMail's webmail or a SmarterMail admin attempts to manage an assumed corrupted domain or impersonate a user:

[2019.02.14] [CurrentUser: SYSTEM]
[2019.02.14] [2/14/2019 5:51:08 AM]
[2019.02.14] https://smartermail.scarabmedia.com/api/v1/mail/folders
[2019.02.14]
[2019.02.14] Application Error. Message: The controller for path '/api/v1/mail/folders' was not found or does not implement IController., Stack Trace: at System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(RequestContext requestContext, Type controllerType)
[2019.02.14] at System.Web.Mvc.DefaultControllerFactory.CreateController(RequestContext requestContext, String controllerName)
[2019.02.14] at System.Web.Mvc.MvcHandler.ProcessRequestInit(HttpContextBase httpContext, IController& controller, IControllerFactory& factory)
[2019.02.14] at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
[2019.02.14] at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
[2019.02.14] at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
[2019.02.14] at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[2019.02.14] =======
[2019.02.14] [CurrentUser: SYSTEM]
[2019.02.14] [2/14/2019 5:51:08 AM]
[2019.02.14] https://smartermail.scarabmedia.com/api/v1/rss/feed-card-data
[2019.02.14]
[2019.02.14] Application Error. Message: The controller for path '/api/v1/rss/feed-card-data' was not found or does not implement IController., Stack Trace: at System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(RequestContext requestContext, Type controllerType)
[2019.02.14] at System.Web.Mvc.DefaultControllerFactory.CreateController(RequestContext requestContext, String controllerName)
[2019.02.14] at System.Web.Mvc.MvcHandler.ProcessRequestInit(HttpContextBase httpContext, IController& controller, IControllerFactory& factory)
[2019.02.14] at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
[2019.02.14] at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
[2019.02.14] at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
[2019.02.14] at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

[2019.02.14] =======
[2019.02.14] [CurrentUser: SYSTEM]
[2019.02.14] [2/14/2019 6:01:27 AM]
[2019.02.14] https://smartermail.scarabmedia.com/api/v1/mail/folder
[2019.02.14]
[2019.02.14] Application Error. Message: The controller for path '/api/v1/mail/folder' was not found or does not implement IController., Stack Trace: at System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(RequestContext requestContext, Type controllerType)
[2019.02.14] at System.Web.Mvc.DefaultControllerFactory.CreateController(RequestContext requestContext, String controllerName)
[2019.02.14] at System.Web.Mvc.MvcHandler.ProcessRequestInit(HttpContextBase httpContext, IController& controller, IControllerFactory& factory)
[2019.02.14] at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
[2019.02.14] at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
[2019.02.14] at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
[2019.02.14] at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Thankfully IMAP, POP, and SMTP seem unaffected for these domains, as it only appears to be affecting webmail (not only can customers not login to their webmail we cannot add, modify, or remove users either). Oddly some domains (29 of 273) work fine.

We tried the usual uninstall, reboot, reinstall the smartermail.exe to no avail. It just seems to requeue the same users for Reindexing over and over every time we uninstall/reboot/reinstall. Could we be missing a dependency that wasn't installed with build 6970? Maybe manually reinstall .NET Framework 4.7.2, or MS Visual C++ 2017 Redistributable?

Any ideas?

11 Replies

Reply to Thread

Scarab Replied

2/15/2019 at 5:00 PM

Looking at the IIS Logs it looks like there are a lot of 401 - Unauthorized errors preceeding the 404 - Not Found. All calls to /api/v1/* are coming up 401 (so are all requests to /ews/exchange/). We are also getting 500 - Server Error on all requests for /interface/output/angular-v-100.0.6970.26175.8d687cc0830d500.js and /interface/output/all-v-100.0.6970.26175.8d687cc0830d500.min.css.

Definitely looks to be a permissions issue with Network Service, but problem didn't exist in previous Build 6956 when we tested the conversion. Perms look right on the \MRS folder. Do perms need to be set manually on a different SmarterMail folder perhaps?

I also tried replacing the .exe & .dll files that were missing from Build 6970 with ones from Build 6956 (as below) but it made no noticeable difference.

\Service\Microsoft.AspNet.SignalR.Core.dll
\Service\Microsoft.AspNet.SignalR.Owin.dll
\Service\Microsoft.Owin.Cors.dll
\Service\Microsoft.Owin.Diagnostics.dll
\Service\Microsoft.Owin.dll
\Service\Microsoft.Owin.Host.HttpListener.dll
\Service\Microsoft.Owin.Host.SystemWeb.dll
\Service\Microsoft.Owin.Hosting.dll
\Service\Microsoft.Owin.Security.dll
\Service\Owin.dll
\Service\System.Reactive.Core.dll
\Service\System.Reactive.Interfaces.dll
\Service\System.Reactive.Linq.dll
\Service\System.Reactive.PlatformServices.dll
\Service\System.Web.Cors.dll
\MRS\bin\roslyn\csc.exe
\MRS\bin\roslyn\Microsoft.Build.Tasks.CodeAnalysis.dll
\MRS\bin\roslyn\Microsoft.CodeAnalysis.CSharp.dll
\MRS\bin\roslyn\Microsoft.CodeAnalysis.dll
\MRS\bin\roslyn\Microsoft.CodeAnalysis.VisualBasic.dll
\MRS\bin\roslyn\Microsoft.CSharp.Core.targets
\MRS\bin\roslyn\System.Collections.Immutable.dll
\MRS\bin\roslyn\System.Reflection.Metadata.dll
\MRS\bin\roslyn\vbc.exe
\MRS\bin\roslyn\VBCSCompiler.exe
\MRS\bin\roslyn\VBCSCompiler.exe.config
\MRS\bin\Microsoft.CodeDom.Providers.DotNetCompilerPlatform.dll

So, close but no cigar. We are still dead in the water.

Sébastien Riccio Replied

2/15/2019 at 8:18 PM

Have you tried to downgrade to 6956 that worked for you when you first tried the conversion ?

With some luck it could help you ? Just a guess, we didn't left v16 yet.

http 401 error code is an unauthorized access like if creditentials were missing

Sébastien Riccio System & Network Admin https://swisscenter.com

Sébastien Riccio Replied

2/15/2019 at 8:34 PM

Some other stupid guesses:

Is your install license validated ? Maybe try to reapply it (most of the time when we tried a conversion of our current install, the license we gave at install would not be taken into account)

Have you compared a working domain and a not working domain files to see if there is any difference (in your Domains directory)

files in

D:\SmarterMail\Domains\someworking.com

D:\SmarterMail\Domains\somenotworking.com

maybe settings.json and folders.json in there. Could be some enabled features enabled on these not working domains

--

Or maybe an IIS site config issue ? Tried to remove it and let the installer create it again ?

I hope you will find a way out of the water

Sébastien Riccio System & Network Admin https://swisscenter.com

Tim DeMeza Replied

2/17/2019 at 6:56 AM

Any updates to this? Very curious. ST has been very quiet for a couple weeks. No releases. Just want to know if this is resolved, blown out of proportion or what. I still need to convert and very concerned.

Thanks

Scarab Replied

2/18/2019 at 2:30 AM

We are still having the problem even after attempting to roll back to Build 6956, when problem persisted we then installed Build 6985 (released 2/15/2019) with no change.

I did a clean install on another server without data and did a diff with my Live data. Cleaned up my Live data by deleting the following directories and their contents left behind over the course of the last 333 SM version upgrades we have done:

\MRS\About
\MRS\App_Browsers
\MRS\App_Data\Definitions
\MRS\App_Data\Dictionaries
\MRS\App_Data\Images
\MRS\App_Data\Navigation
\MRS\App_Data\RSS
\MRS\App_Data\Temp
\MRS\App_Data\Translations (anything other than *.json)
\MRS\App_Data\WebSettings.xml
\MRS\aspnet_client
\MRS\Common
\MRS\DomainAdmin
\MRS\EWS
\MRS\Images
\MRS\Javascript
\MRS\m
\MRS\mail
\MRS\MailProcessing
\MRS\Main
\MRS\MasterPages
\MRS\Mobile
\MRS\Public
\MRS\Sync
\MRS\SystemAdmin
\MRS\Temp
\MRS\UserControls
\MRS\WebChat
\MRS\dkim.html
\MRS\Reports\Controls
\Services\App_Data
\Services\Ham
\Services\PlatformDependent
\Services\ReportDefinitions
\Services\Spam
\Services\Temp
\Web Server

That seemed to fix the issue with the three repeating entries in the SM Error Log (probably because there were pre-existing \mail and \rss folders in the \MRS directory). It also resolved the EWS getting a "401 Unauthorized" error in the IIS Logs, probably for the same reason, as well as the "401 Unauthorized" error when attempting to login, manage, or impersonate a user on one of the affected domains, however, we are still getting the "404 Not Found" when attempting to login, manage, or impersonate still. The POST /api/v1/settings/domain/impersonate-user/username@domain.tld fails even though the line right before it for GET /api/v1/settings/sysadmin/user-exists/username@domain.tld succeeds with a server response 200.

I guess in the morning I'll start doing a diff on each individual file in each domain with the copy of our Live data we used to test the conversion to Build 6956.

Still haven't heard from anyone at SmarterTools Support since Wednesday, and even if we do it will have been one week since opening the ticket before the next "Can you try this?". Considering we were also dead in the water when migrating to v16 upon it's initial release (Incoming Gateways no longer worked in Smartermail Mode and were bouncing all inbound email) and although they closed the Support Ticket immediately without any resolution (other than "Don't use that feature") it took them over 13 months to ever fix the issue (v16.3.6698 being the first attempt and v16.3.6729 finally getting it right). I honestly don't have much faith in their Support being of any assistance at this point.

Scarab Replied

2/19/2019 at 11:03 AM

Just as I expected, SmarterTools support got back to us asking "So the problem is resolved?" without any resolution! We have asked to speak with a manager at SmarterTools because having email down for all users and system admins for 250 out of 273 domains for almost a week without any Technical Assistance is a joke!

We have never been so disappointed with SmarterTools (which after the mess from v15 to the first release of v16 I thought was bad).

Scarab Replied

2/19/2019 at 5:29 PM

Marked As Answer

To update everyone Matt and Derek from SmarterTools came to my rescue. After they verified that our Live SmarterMail data was good and couldn't duplicate the issue on their VMs I was finally able to isolate the issue to an IIS RequestFiltering rule (not sure if it was a disallowed File Extension, Verb, or Header Limit but it was one of them) that was causing issues with the API calls used by the web interface in v17/100, but not in previous versions. (Still not sure why it didn't affect all domains, but that was indeed the cause.) Sebastien turned out to have the correct answer!

Hopefully SmarterTools will be able to add a RequestFiltering /clear element to the IIS configuration utility in future builds of SmarterMail to help others avoid the problem we encountered.

We are up and running again, thanks to Matt & Derek.

Sébastien Riccio Replied

2/19/2019 at 6:38 PM

Good to hear that and glad you are now back on rails. Thanks for the update on this topic!

Sébastien Riccio System & Network Admin https://swisscenter.com

Scarab Replied

2/19/2019 at 8:36 PM

After some recollection I isolated the exact RequestFiltering rule. File extensions of *.com (DOS Binaries) were blocked in our hardened IIS configuration. This caused all domains with a .com not to load when v17/100 API used the domain or email address as a file name. This ultimately caused 75% of our domains not to load, whereas any domains that were .net, .org, etc were not blocked which is why those ones still worked. I didn't see the commonality of what made those 28 domains out of 273 work until I realized in retrospect that they weren't .com domains.

Derek Curtis Replied

2/19/2019 at 9:59 PM

Employee Post

Thanks for the update and I'm glad things are back running. In all honesty, it was Rod, another of our support agents, who did the leg work and got the install up and running. He tested it, then we jumped on and tested as well. So while I'm not averse to taking credit where credit is due, this is not one of those cases.

Derek Curtis COO SmarterTools Inc. www.smartertools.com

CTL Replied

2/19/2019 at 11:16 PM

Scarab I hope you have fixed existing installation issue The new build 6985 fixed some of the spam issue as well ( 6970 )

Back to Community Threads

Please leave this box unchecked

Reply to Thread

Enter the verification text

11 Replies

Reply to Thread

Tags

Related Knowledge Base Articles