Upgrading SmarterMail (Domain Conversion)

When performing an upgrade of SmarterMail from 16.x or prior to the latest supported version, SmarterMail will go through a significant domain conversion process on first load. During this time, XML files will be converted to JSON format, mailboxes will be switched to use the new indexing system, and system files will be modified to help increase SmarterMail speed, reduce disk/CPU and, overall, make for a smoother experience. 

The conversion process make take some time to complete, especially if you have a large amount of data in GRP files, stat files, etc. However, as domains and mailboxes finish the conversion process, they will be available for use. No additional restarts will be necessary. To speed up the upgrade and conversion process, you can do the following:

• Set up a test environment with your existing configuration files only, leaving GRP files and non-configuration files alone.
• To assist with this, we have a handy robocopy command you can use for backing up and restoring just the essentials (ignoring .grp and .stat3 files): robocopy Source Destination /e /r:1 /w:1 /xf *.stat3 *.grp /xd IndexV2. Here's an example of backing up domains and certificates on Windows:

  robocopy C:\SmarterMail\Domains C:\SmarterMail_new\Domains /e /r:1 /w:1 /xf *.stat3 *.grp /xd IndexV2 
  
  robocopy C:\SmarterMail\Certificates C:\SmarterMail_new\Certificates /e /r:1 /w:1 /xf *.stat3 *.grp /xd IndexV2 

You can speed up the process by adding in multi-threading to the command (e.g., /mt:[1-128]) but that can also increase your CPU usage, so do so at your own risk. 

You can access the conversion status page at any time by going to https://your_mail_domain/interface/convert-status. You'll need to log in with a system admin username and password. As the domain conversion runs, you'll see a screen similar to the following:

Once the domain conversion has completed, you'll see something like this: 

Common Issues and Troubleshooting

While SmarterMail will upgrade and run through the conversion process without issues the vast majority of the time, there can be instances where a particular user or domain fails to convert. Thankfully, SmarterMail has some processes in place to prevent the majority of issues we've see. In addition, SmarterMail keeps a temporary archive of old files for each user and each domain so if something does go wrong, it can be resolved quickly and easily. By default, these archives are kept in

C:\SmarterMail\Domains\{domain}\Archived Data\Legacy Config Files.zip.

The most common error seen is for the XML file for a domain or a user to fail to convert over to JSON. If this happens, the Upgrade Process screen will display an error for the domain with the issue. Clicking on that line will open a modal window that should display the issue, such as the user that failed to convert. For example, you'll see a message such as "Domain Failed to Load. As a result, mail will not function and users will not be able to login. For assistance resolving this issue, please refer to the SmarterMail Help." (If no user is displayed, chances are the issues are with the domain level JSON files.)

The 2 files that fail to convert most often are the settings.json and folders.json files. You can verify the file integrity by running them through an online JSON validator such as JSONLint: simply copy the contents of one of those files and paste it in the online validator.

If a particular XML file fails to convert to JSON, and the JSON file is invalid, it's very easy to fix. Simply do the following:

1. Restore the legacy XML file to the folder with the invalid JSON.
2. Stop the mail service.
3. Delete the invalid JSON file.
4. Restart the mail service.

Once the mail service restarts, it looks for missing JSON files and restarts the conversion process. This can be done for an entire domain as well: you stop the mail service, remove the JSON files for the domain, restore the XML files from the domain's archived data, then restart the mail service.

Another issue that's been seen is for a previously deleted domain to show up as not being fully deleted -- it has a folder with a single Resources folder in its root. So, even if it was deleted, it still shows in the list of domains after the upgrade. To solve this, you simply need to manually edit the domains.json file and delete that domain.

One final issue that's been seen, although rarely, is for the conversion to finish but none of the domains are converted and none of the settings are kept. So it's almost like a new version was installed rather than an existing version upgraded. This, too, is easy to resolve:

1. Stop the mail service.
2. Rename the Settings folder (or it can be deleted).
3. Rename the old Mail.cfg file (remove the .old).
4. Restart the mail service.

99.9% of the time, this fixes that issue.

Of course, if you run into errors at any point during the conversion process, and none of the above fixes resolves those issues, please contact the SmarterTools Support Department. You can submit a ticket at the SmarterTools website by clicking on the account icon in the upper right. Once you've logged in, select My Tickets from the dropdown. Be sure to provide a screenshot of the errors you're seeing in the SmarterMail interface. The support team will also need the conversion.log file from your SmarterMail Logs folder. (The default path is c:\SmarterMail\Logs. However, you can find the log path for your specific installation by logging in as the Administrator and going to the Manage icon > Troubleshooting section > Options tab > Log Path setting.) Please send the full log file or copy and paste the snippet of text containing the domains showing an error.

Important Note Regarding New Indexing System

In the latest release, we've updated the indexing format. After your initial upgrade to the current supported version from version 16.x or prior, user mailboxes will be reindexed in order to utilize this new format. The reindexing will occur as users log into webmail, connect to email clients, or send/receive mail. As the reindexing occurs, you will see a temporary spike in CPU. However, please note that a user's experience and the overall server performance should not be negatively impacted, as the indexing thread is prioritized lower than all other threads. In addition, users will still be able to search while this one-time bulk reindex occurs. This is because a user's account will use the old indexing format for searches if the reindexing has not yet been performed for their account. As soon as their account has finished reindexing, it will simply switch to the updated format. 

Because the indexing update is performed in batches and prioritzed lower than all other threads, it's possible that the reindexing process will take a few days to complete. There are two ways to know whether you're still processing the bulk reindex: 

1. Log in to SmarterMail as a Domain Admin and head to Domain Settings > Accounts > Users tab. Click through your users to view the Search Language Indexer setting. If their account has finished reindexing, 'Generic Indexer' will be selected. If they haven't been reindexed yet, 'Old' will be selected instead. (As soon as a user is reindexed, their Search Language Indexer setting will switch to Generic Indexer.) 
2. Log in as a System Admin and head to the Manage icon > Troubleshooting > Mailbox Indexing tab. You'll find a list of users that are currently being indexed.

Please also feel free to turn your Indexing logging to Detailed so you can find any related errors. You'll find this log setting at the Manage icon > Troubleshooting > Options tab.