Back to Community Threads
5
Feature Request: Live-Move Domains to New Disk/Volume
Idea shared by Jay Dubb - 7/2/2021 at 7:06 AM
Proposed
With infrastructure going increasingly virtualized, it is routine to live-move entire servers from chassis to chassis (or datacenter to datacenter) with zero downtime.  Similarly, we can live-move VM storage from one disk volume to another with zero downtime.

However, with SmarterMail, when a domain's storage grows to unexpected levels and we need to move it to another volume, we have to take the entire SmarterMail server offline, affecting all customers for the entire duration of the move-- which can take substantial time depending on size of the domain's storage.

Example:  Customer started small but business prospered and domain storage exploded from around 10 GB to nearly 2 TB, causing space constraints on the server's on-board storage.  Need to move this domain to an iSCSI drive with huge storage available.  Per KB https://portal.smartertools.com/kb/a2727/move-a-domain-to-a-different-hard-drive-on-the-same-server.aspx  -- we have to:

1.)  Stop the SmarterMail service, which affects ALL customers.

2.)  Copy these 2 TB of data across the network to an iSCSI chassis, which in real-world terms even on a 10-GigE network, means downtime exceeding 30 minutes... affecting ALL customers on that server.

3.)   Manually edit the .json files in a text editor for the domain, to point to the new storage location.

4.)  And only then, can we bring the service back online.

Ideally, moving SM domains to other storage would be a live-move, just as with most other virtualized services/servers.  But even if we can't get that far "today", at least give us the ability to specify the new storage location in the web GUI.  Then the SM service can drop just that domain offline, move its storage, then bring it back online-- without disturbing any other customers on that server.

My $0.02.

17 Replies

Reply to Thread
4
2. We used rsync to move the bulk of our data to new storage location days before the "official" date of shutting down SM, once it completed, we ran it again to get any additional stuff missed during the time it took to move the bulk of the data. Then, shutdown SM, ran rsync again to get everything now that the service is off, changed the location details in SM config and restarted the service. Took maybe... 10 minutes (once SM was off).
0
+1
Gabriele Maoret - Head of SysAdmins at SERSIS Currently manages 6 SmarterMail installations (1 in the cloud for SERSIS which provides services to a few hundred third-party email domains + 5 on-premise for customers who prefer to have their mail server in-house)
0
I usually do the same of echoDreamz.
I think it's quite impossible to do it without shutting down SM for few minutes.
2
That would be a nice feature to integrate.  Imagine where a storage-move request could be entered into the admin GUI.  The new storage location would be entered, plus a checkbox asking if the system should delete files in the old location after a successful relocation.  Smartermail would:

1.)  rsync, xcopy, or robocopy (since robocopy is mutli-threaded and native to Windows) the storage and wait for an exit code indicating success.

2.)  Drop just that domain offline to unlock its files.

3.)  Run a final rsync/robocopy..

4.)  As soon as the copy operation is finished with a successful exit code, update the .json file with the new location and bring the domain back online.

5.)  If the admin clicked the box to delete files in the original location, delete now.

6.)  If an error occurs during the copy operation, either in step #1 or #3, send an alert to the admin, leave the domain in its current location, abort the move, and bring the domain back online in its original location.

Or something along those lines.  Since this is a rather simple operation-- just time consuming-- it should be fairly easy to automate, with the tools already native to Windows.
4
I do not believe, that moving domain-file-storage without shutting-down the SmarterMail-server Windows-service will be an easy task to implement into SmarterMail. Our SmarterMail-service is running on Windows-server on VMware ESXi but I really do not see requirements for live-move of domains/storage at all.

In my humble opinion: there are really more essential issues and other (missing) Exchange-features, that should be addressed to make SmarterMail a real Exchange-alternative.
2
Well, we're in that circumstance now.  We ended up doing an initial robocopy, then when done, dropped the service offline, did a final sync, and brought it back up in the new location.  This time we had the luxury of a holiday weekend with very low traffic levels.  I anticipate having to do this again for another customer this month, due to exploding mailbox consumption.

Call me old-school, but I think it's insane to offer 25, 50, even 100 GB email boxes, but that's the only way to be somewhat competitive with Microsoft 365 and other providers offering similarly huge mailboxes.  It's not that people have a million text-based emails.  They treat email like a file cabinet-- tens of thousands of messages with attachments.  Yes it can be done, but it's a lousy (lazy?) way of storing documents, photos, etc.  Seems nobody's heard of saving attachments and deleting the email.

We used to offer a 2 GB mailbox and people barely used a fraction.  Now it's routine to see people hit 25 GB very quickly.  We are reaching the limits of onboard storage on our servers, and need to push some domains off onto (much cheaper, much larger) off-server storage.  We run Samsung Enterprise class SSDs in 8 x SFF disk HP ProLiant servers.  SSDs are really fast and reasonably dependable(**) but far more expensive than bulk storage on our iSCSI arrays.  So for domains who go crazy with email, we're going to start pushing them off the SSDs and onto the storage tanks.

(**) While the Samsung Enterprise SSDs (a mix of SM863 and 883 DCT models) have been reasonably reliable across the several dozen servers we run, they haven't been as bulletproof as the 15k Seagate Cheetahs we used for 15 years (and still do).  For bulk storage, we've been running a few dozen Seagate Exos 10 TB disks across several large RAID storage chassis for about a year, but it's too soon to tell how reliable they'll be.
1
I usually do the same of echoDreamz.
I think it's quite impossible to do it without shutting down SM for few minutes by keeperspro.com
It  is possible but you need a huge technical knowledge

Not really? If you are running and maintaining a mail server, surely running rsync (and if you cant use command line, there are tools like BeyondCompare that make it SUPER easy to do) and editing a JSON file isnt that big a deal.

Currently there is no way to do it that I can think of safely without some way to detach a domain from SM, move it, then reattach it with. Though, we bulk moved over 8TB of mail data with just a few minutes of downtime, was nearly the same as a normal SM upgrade.
1
Don't waste your time responding to Johnny Rock-- he's a spammer.  Brand new account, this is his only post, and you'll notice that when he quoted a prior reply, he stuck in a domain name for no apparent reason.  This not only captures the "curious" lookers, but when Google and other search engines index this page, his domain will be cross-referenced.  Sneaky tricks by SEO scammers.  

In fact, you might want to edit your reply to remove reference to his domain, because every time it's quoted it adds to the number of times it appears in the page and that will only help him in his sleazy tricks.

UPDATE:  Looks like the moderator removed his account and the post.  Thank you, ST.
1
Employee Replied
7/12/2021 at 10:09 AM
Employee Post
Jay Dubb, you might be able to accomplish what you want with the Attach / Detach Domain feature that was introduced in Build 7459 (Jun 3, 2020)--"System Administrators can no Attach and Detach domains without stopping the mail service."

As a system administrator on the Domains list page in the ... button are options to Attach and Detach.  Detaching effectively unloads the domain and prevents any file changes to it or the its users.  You can then move this domain to a new location before attaching domain again.  This is all accomplished without having to restart the mail service.
0
Thank you, Robert.  That certainly gets us a LOT closer to the goal, without having to disrupt any other customers.

Question-- Can the JSON file be modified with the new location while the SM service is still running?

 
0
Employee Replied
7/12/2021 at 11:45 AM
Employee Post
Which JSON file are you referring to?  As a rule of thumb, you shouldn't modify any of the JSON files while the SM service is running because we use a memory cache to store a copy in memory that could potentially overwrite any changes you make.
0
<install directory>\SmarterMail\Service\Settings\domains.json 

So the service would still have to be stopped to modify the disk path of the domain, since a copy is cached in memory.  I was hoping there was a way to effect a physical move of a domain's storage, without affecting any other domains on the server.  

What I'm suggesting as a feature enhancement is not unlike being able to move users to another Exchange database (on another volume) in realtime, without having to take anything offline.
1
Employee Replied
7/12/2021 at 3:51 PM
Employee Post
If you use the Detach Domain option, it will remove it from the domains.json for you.  Then you can move the domain to a new location and Attach Domain specifying the new location of the domain.  This will add it back to the domains.json file.  Because you are doing this through the web interface, you do not have to stop the mail service.
0
Thank you, Robert.  Is there any workaround to the problem of losing domain aliases?  I'm surprised this one particular account config isn't carried over with the rest of the user configs.  Some of the customers we will be moving to another volume are real estate brokers who have a bunch of domain aliases that will be lost (and need to be reconfigured) when the domain is reattached.

I stand by my original suggestion that some type of live-move feature would be a nice one to include in future builds.  Since there is a detach/reattach feature, and robocopy is native to Windows, maybe it could be scripted as a pre-migration robocopy, and when the return code is received indicating success, drop the domain, do another robocopy to catch the last-moment changes, and bring the domain back online (with its aliases).  Just a thought.  Moving users around to different databases in Exchange is drop-dead simple.
 
0
Employee Replied
7/13/2021 at 8:29 AM
Employee Post
Jay, I am unaware of the problem with losing domain aliases?  Can you detail this more so I can get a task created to look into it?
0
When you select a domain and click the Detach Domain option, a warning pops up:

 "Are you sure you want to detach the selected domain? This action will log out any currently logged in users. Domain aliases will be removed and need to be reconfigured after reattaching."

Note:  I tried attaching a screenshot, but it failed.  The image appears in the message edit window, but immediately disappears upon any mouse-click.
0
Employee Replied
7/13/2021 at 2:35 PM
Employee Post
Thank you, Jay.  I will review the code to see if we're able to retain that information rather than losing it.
Back to Community Threads

Reply to Thread

Related Knowledge Base Articles

Move SmarterMail from Hosted to Self-InstalledSmarterMail > Installation and Configuration
Regularly Backup SmarterMail on Windows Using HobocopySmarterMail > Server Configuration and Management
Legal Holds and LitigationSmarterMail > Installation and Configuration
Flex Billing for Microsoft Exchange ActiveSyncGeneral Topics
Deploying Rspamd For Use With SmarterMailSmarterMail > Server Configuration and Management