Autodiscover for Outlook 2013 / 2016 on Windows for IMAP/SMTP Accounts

Question asked by Shaun Peet - 6/15/2017 at 11:55 AM

Unanswered

I have an open support ticket on this but I'm hoping there's somebody in the community that might have experience with this and can provide their ideas as well.

We really want to achieve the "holy grail" or user setup in all devices - which is that they submit their email address and password to the mail client, and all other settings are setup automatically. This would include SMTP authentication and our default / recommended security settings. And it should work for non-EAS accounts.

Try as I might, we just can't get this to work properly, and it looks like it should.

For now I just want to focus on Outlook 2013 / 2016 on Windows since that should be the easiest.

I've created a couple of mailboxes for testing - test@mbsportsweb.info and eas@mbsportsweb.info. Anyone willing to help figure this out can PM me and I'll share the passwords for those accounts. (I've throttled their outbound activity to prevent this from becoming a mess).

Our mail server is located at https://mail.mbsportsweb.ca and it has it's own certificate. We're running SM16 but I don't think that matters. We're forcing all traffic to HTTPS. We've also got a DNS record for http://autodiscover.mbsportsweb.ca but that doesn't work because SM wants to redirect that to HTTPS and there's no certificate for that domain. IIS has bindings for mail.mbsportsweb.ca for both port 80 and 443, the autodiscover binding is for port 80 only.

The DNS settings for the mbsportsweb.info domain with respect to mail are:

MX mbsportsweb.info mail.mbsportsweb.ca priority: 10

CNAME mail -> mail.mbsportsweb.ca

CNAME autodiscover -> mail.mbsportsweb.ca

SRV _autodiscover._tcp.mbsportsweb.ca. -> SRV 0 0 443 mail.mbsportsweb.ca.

In addition to all that, because the first check for an autodiscover.xml file is made at the root domain, I've added a static XML response at the https://mbsportsweb.info/autodiscover/autodiscover.xml as follows:

<?xml version="1.0" encoding="utf-8"?>
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">
<Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
<Account>
<AccountType>email</AccountType>
<Action>redirectUrl</Action>
<RedirectUrl>https://mail.mbsportsweb.ca/autodiscover/autodiscover.xml</RedirectUrl>
</Account>
</Response>
</Autodiscover>

Technically, since this exists the DNS records on the domain name shouldn't matter as the above would short-circuit Outlook to go directly to the mail server to get the settings.

This all appears to work as designed. Using the "Test E-Mail AutoConfiguration" tool (which you can get to by holding the CTRL key and then right-clicking the Outlook icon in the taskbar), the XML response for the Autodiscover file is:

<?xml version="1.0" encoding="utf-8"?>

<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">
   <Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
       <Account>
           <AccountType>email</AccountType>
           <Action>settings</Action>
           <Protocol>
               <ASUrl>https://mail.mbsportsweb.ca/ews/exchange.asmx</ASUrl>
               <DomainName>mbsportsweb.info</DomainName>
               <DomainRequired>on</DomainRequired>
               <EwsUrl>https://mail.mbsportsweb.ca/ews/exchange.asmx</EwsUrl>
               <LoginName>test@mbsportsweb.info</LoginName>
               <OOFUrl>https://mail.mbsportsweb.ca/ews/exchange.asmx</OOFUrl>
               <Server>mail.mbsportsweb.ca</Server>
               <Type>EXPR</Type>
           </Protocol>
           <Protocol>
               <AuthRequired>on</AuthRequired>
               <DomainName>mbsportsweb.info</DomainName>
               <DomainRequired>on</DomainRequired>
               <LoginName>test@mbsportsweb.info</LoginName>
               <Port>143</Port>
               <SPA>off</SPA>
               <SSL>on</SSL>
               <Server>mail.mbsportsweb.ca</Server>
               <Type>IMAP</Type>
           </Protocol>
           <Protocol>
               <AuthRequired>on</AuthRequired>
               <DomainName>mbsportsweb.info</DomainName>
               <DomainRequired>on</DomainRequired>
               <LoginName>test@mbsportsweb.info</LoginName>
               <Port>110</Port>
               <SPA>off</SPA>
               <SSL>on</SSL>
               <Server>mail.mbsportsweb.ca</Server>
               <Type>POP3</Type>
           </Protocol>
           <Protocol>
               <AuthRequired>on</AuthRequired>
               <DomainName>mbsportsweb.info</DomainName>
               <DomainRequired>on</DomainRequired>
               <LoginName>test@mbsportsweb.info</LoginName>
               <Port>465</Port>
               <SPA>off</SPA>
               <SSL>on</SSL>
               <Server>mail.mbsportsweb.ca</Server>
               <Type>SMTP</Type>
           </Protocol>
       </Account>
       <User>
           <DisplayName>test@mbsportsweb.info</DisplayName>
           <EMailAddress>test@mbsportsweb.info</EMailAddress>
       </User>
   </Response>
</Autodiscover>

Everything there looks right and it *should* work. However, it doesn't. Here's a video showing what actually happens in Outlook 2013:

https://youtu.be/EeFopA04FZk

During the automatic setup process, Outlook will test the account by trying to send a message. For whatever reason, either the Autodiscover.xml file or Outlook 2013 refuse to set the "Outgoing server requires authentication" flag. This prevents the sending of the test message, forcing the user to go into manual settings. Once in manual settings, everything looks correct, however, the user needs to click on "More Settings", then to the Outgoing tab, then check the box. For the average user this is simply poor design in Microsoft's part - however, - if things worked correctly with the Autodiscover then it wouldn't be an issue. As the video showcases, the SSL settings are both set to "none" as well, which isn't what it should be.

Has anyone ever gotten this to work the way it's supposed to? I hope so, and then we can get it working in my environment so we can then get onto the EAS account, and then to Android clients, and then to iOS (which I believe is the order of difficulty).

Thanks in advance for anyone who is able to HELP!

13 Replies

Reply to Thread

Shaun Peet Replied

6/15/2017 at 8:07 PM

I wonder if the following property should be added within the SMTP <Protocol> section:

<UsePOPAuth>on</UsePOPAuth>

I've seen that property in a few of the online examples out on the web. FWIW I did try manually creating an autodiscover.xml file with that property and it didn't seem to make a difference. But it might not hurt to have it included in the response.

This is another property I've seen out on the web within the same SMTP <Protocol> section:

<SMTPLast>on</SMTPLast>

I feel like I'm grasping at straws at this point - but if anyone has gotten this to work please share how :)

Shaun Peet Replied

6/15/2017 at 8:58 PM

Found this:

https://social.technet.microsoft.com/Forums/en-US/2230e004-43e2-4531-8ac4-e54c580e446f/outlook-autodiscover-smtp-authentication?forum=outlook

The suggestion there was to remove these:

Might have to test this by creating my own actual static autodiscover.xml response file since I won't be able to modify the response from SmarterMail...

... unless somebody already knows the answer to this :)

Shaun Peet Replied

6/15/2017 at 9:01 PM

More (related) reading...
https://stackoverflow.com/questions/34639133/outlook-autodiscover-smtp-authentication

Shaun Peet Replied

6/16/2017 at 12:09 PM

Some progress...

Turns out that some of our issues were being caused by ports 993, 995, and 465 being blocked by our external firewall and/or Windows Firewall. So setting the <SSL>on</SSL> property for various protocols was causing issues because those ports were blocked.

But in addition to that, the autodiscover settings within SmarterMail for the various protocols were not using the SSL ports anyway. The IMAP was using port 143 with SSL "on" and POP was using 110 also with SSL "on". We've updated both of those within SmarterMail to 993 and 995, respectively.

For SMTP I think that the recommended best practice these days is to use TLS on port 587 rather than SSL on 465 (somebody please correct me if I'm wrong). However, SmarterMail's auto-generated autodiscover.xml file doesn't set TLS anywhere - just <SSL> as either "on" or "off". There's a property available called <Encryption>TSL</Encryption> and that seems to do the trick.

For now, and because we can, we've decided to produce our own autodiscover.xml files at the root of each of the websites we host (that's the advantage we have in hosting our own custom-build CMS for our customer's websites as well as their hosted email).

We've written a custom HTTP Module in our asp.net application to handle URL rewriting and added detection for the following URLs:

/autodiscover/autodiscover.xml

/.well-known/autoconfig/mail/config-v1.1.xml

Upon detection of those incoming URL requests, we've written some custom code to spit out XML files. If the request is an HTTP GET request then we assume it's some sort of guess-smart system (like Thunderbird) and it'll return the config-v1.1.xml file; if it's a POST then we pretty much assume (for now) that it's an Autodiscover request and return an autodiscover.xml file.

While doing that, if we can determine the specific email address being requested, we also query the old SmarterMail web services to 1) verify that we are hosting the domain, 2) verify that the user exists, and 3) determine whether the user is EAS or not.

Assuming the domain & user exists, then depending on whether it's an EAS user or not we serve up two different autodiscover.xml responses.

EAS:

<?xml version="1.0" encoding="utf-8"?>
<Autodiscover xmlns:xsd="http://www.w3.org/2001/XMLSchema"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">;
<Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/mobilesync/responseschema/2006">;
<Culture>en:us</Culture>
<User>
<DisplayName><%= display_name %></DisplayName>
<EMailAddress><%= email_address %></EMailAddress>
</User>
<Action>
<Settings>
<Server>
<Type>MobileSync</Type>
<Url>https://mail.mbsportsweb.ca/Microsoft-Server-ActiveSync</Url>;
<Name>https://mail.mbsportsweb.ca/Microsoft-Server-ActiveSync</Name>;
</Server>
</Settings>
</Action>
</Response>
</Autodiscover>

Non-EAS:

<?xml version="1.0" encoding="UTF-8"?>
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">;
<Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">;
<User>
<DisplayName><%= display_name %></DisplayName>
<EMailAddress><%= email_address %></EMailAddress>
</User>
<Account>
<AccountType>email</AccountType>
<Action>settings</Action>
<Protocol>
<Type>EXPR</Type>
<Server>mail.mbsportsweb.ca</Server>
<ASUrl>https://mail.mbsportsweb.ca/ews/exchange.asmx</ASUrl>;
<EwsUrl>https://mail.mbsportsweb.ca/ews/exchange.asmx</EwsUrl>;
<OOFUrl>https://mail.mbsportsweb.ca/ews/exchange.asmx</OOFUrl>;
<LoginName><%= email_address %></LoginName>
</Protocol>
<Protocol>
<Type>IMAP</Type>
<Server>mail.mbsportsweb.ca</Server>
<Port>993</Port>
<SPA>off</SPA>
<SSL>on</SSL>
<AuthRequired>on</AuthRequired>
<LoginName><%= email_address %></LoginName>
</Protocol>
<Protocol>
<Type>POP3</Type>
<Server>mail.mbsportsweb.ca</Server>
<Port>995</Port>
<SPA>off</SPA>
<SSL>on</SSL>
<AuthRequired>on</AuthRequired>
<LoginName><%= email_address %></LoginName>
</Protocol>
<Protocol>
<Type>SMTP</Type>
<Server>mail.mbsportsweb.ca</Server>
<Port>587</Port>
<SPA>off</SPA>
<SSL>off</SSL>
<Encryption>TLS</Encryption>
<AuthRequired>on</AuthRequired>
<LoginName><%= email_address %></LoginName>
</Protocol>
</Account>
</Response>
</Autodiscover>

Here's the format for the Guess Smart return file:

<?xml version="1.0"?>
<clientConfig version="1.1">
<emailProvider id="mail.mbsportsweb.ca">
<domain>%EMAILDOMAIN%</domain>
<displayName>MBSportsWeb Hosted Email</displayName>
<displayShortName>MBSportsWeb</displayShortName>
<incomingServer type="imap">
<hostname>mail.mbsportsweb.ca</hostname>
<port>993</port>
<socketType>SSL</socketType>
<authentication>password-cleartext</authentication>
<username>%EMAILADDRESS%</username>
</incomingServer>
<incomingServer type="pop3">
<hostname>mail.mbsportsweb.ca</hostname>
<port>995</port>
<socketType>SSL</socketType>
<authentication>password-cleartext</authentication>
<username>%EMAILADDRESS%</username>
</incomingServer>
<outgoingServer type="smtp">
<hostname>mail.mbsportsweb.ca</hostname>
<port>587</port>
<socketType>STARTTLS</socketType>
<username>%EMAILADDRESS%</username>
<authentication>password-cleartext</authentication>
</outgoingServer>
</emailProvider>
</clientConfig>

So far we've tested account creation in Outlook 2013 for both an EAS and a non-EAS account and both worked flawlessly. We've also tested account creation in Thunderbird and for both accounts it created them as IMAP but I think that's because Thunderbird doesn't support EAS accounts.

So now we're on to the next thing, which is trying to get any of this to work on iOS (so far no luck) and Android (no luck there either). We'll also be testing Outlook for iOS and Outlook for Android to see if those work better than the native email apps from either OS. Android is new to me and from what I've heard almost nobody uses the built-in email app because either the Gmail or Outlook ones are much better.

As a side-note, I'm not a big fan of producing our own autodiscover.xml file since I believe this is something that should come out of SmarterMail. But again, since we host both the website and email for our customers, we can get a little creative given that Outlook's first check is to the root domain (aka the website). We'll hopefully end up doing the autodiscover-redirect method (assuming it works on Android) as long as the xml file returned by SmarterMail can start enabling TLS.

One final note for now is that this is way too complicated and there's significant room for improvement in how all this stuff is configured & managed.

Shaun Peet Replied

6/16/2017 at 12:13 PM

I should mention that in our autodiscover.xml return files the variables such as <%= email_address %> are populated with actual values prior to being returned. In the config-v1.1.xml file those %EMAILADDRESS% variables are perfectly legitmate and that's what is actually returned in the file.

Webio Replied

6/16/2017 at 1:45 PM

Have you maybe considered using global IIS URL Rewrite rules instead of per website module? This could allow to host one application which is written for generating proper XML files based on requesting URL.

Shaun Peet Replied

6/16/2017 at 2:31 PM

We've thought about that - we'd still need to create DNS entries for every domain for that to work (CNAME -> common autodiscover application). And we're talking about 430+ domains (and counting). We use CloudFlare for DNS so this could all be scripted. However, for the time being it was easier for us to start creating the files directly at the root of the site - plus that's the first place Outlook checks anyway (and Android supposedly too, but that doesn't seem to necessarily be the case).

Webio Replied

6/16/2017 at 9:52 PM

When mentioning IIS URL Rewrite I was only talking about those two mentioned paths:

/autodiscover/autodiscover.xml
/.well-known/autoconfig/mail/config-v1.1.xml

I'm wondering if this could be handled on global IIS level with rewrite action to some one URL. It would be easier to maintain.

Shaun Peet Replied

7/3/2017 at 9:17 AM

Working with the SmarterMail team to make this better has been great. There's now an option when setting up the various Protocols to choose the Encryption option for SMTP In - with options for "None", "TLS", and "SSL". Selecting the TLS option correctly sets the <Encryption>TLS</Encryption> property in the autodiscover.xml file returned by SmarterMail which therefore sets up Outlook 2013 / 2016 perfectly for IMAP accounts (EAS was never an issue in Outlook).

The question that remains for the greater mail server community (and mail client community) is what method should be used to instruct the client where the autodiscover.xml file resides.

In our particular case we have two options because we host the website (using our custom CMS) and email - so we can intercept the request that Outlook makes to the root domain and handle it. We can also create the DNS records and script the creation of those. I'm not sure which option is "better" although in theory the DNS method would be more universal. The problem is that Android and iOS don't seem to use the DNS methods.

For now we're going to keep returning autodiscover.xml files from our CMS but we'll be testing using the RedirectUrl property of that file to make sure that works in as many scenarios as possible. Hopefully it does because I think it'st still best that the mail server generate the actual configuration file(s) needed to setup the account - the routing of the request is the only thing we should have to deal with.

Michael Muller Replied

3/23/2018 at 11:00 AM

This could also be accomplished with a Virtual Directory within IIS, which can be scripted using AppCmd, which you could batch through all your domains:

c:\windows\system32\inetsrv\AppCmd add vdir /app.name:[IIS-site-name]/ /path:/autodiscover /physicalPath:[path to main site folder]\autodiscover
c:\windows\system32\inetsrv\AppCmd add vdir /app.name:[IIS-site-name]/ /path:/.well-known /physicalPath:[path to main site folder]\.well-known

--- Montague WebWorks Powered by RocketFusion

Michael Muller Replied

3/23/2018 at 11:06 AM

I completely agree. We're on v15x for now until we're ready to upgrade to v17, and we would absolutely need this to function internally. If we need an admin screen at the server level to supply server name (no custom email servers for our customers, all use the same) and all other info. Best also for SM to supply what they need in our DNS, which we also control through a custom system. We tried this and it did not work:

autodiscover IN A [server IP]
autoconfig IN A [server IP]
_autodiscover._tcp SRV 0 0 443 [our server name with a .]

--- Montague WebWorks Powered by RocketFusion

Michael Muller Replied

3/23/2018 at 11:11 AM

I believe this is a major issue that SM needs to support us on. As more and more people move to mobile devices for their work, I am getting more and more calls from clients who have just bought a new phone (Apple or Android) or a new laptop (Outlook, Thunderbird, Windows' awful Mail, MacMail) and need help setting up their email. It would really save me tons of time for it to auto-fill in all the settings.

SmarterTools needs to get in front of this with a bullet-proof, easy to implement solution.

--- Montague WebWorks Powered by RocketFusion

Adam Replied

7/27/2018 at 9:08 PM

Agreed!

Back to Community Threads

Please leave this box unchecked

13 Replies

Reply to Thread

Tags

Related Knowledge Base Articles