We are facing this issue very frequently with newer builds >8930. After the above error, the domains shows as 0 mailboxes and mails getting rejected. I need to check and restore the file from backup. Is there any possibility to implement the same in SM itself ?

[2024.09.25] 14:33:23.043 [FileFunctions] File "d:/SmarterMail/Domains/meghaeng.com/Users/lldpe.ho/folders.json" missing. Found TMP file "d:/SmarterMail/Domains/meghaeng.com/Users/lldpe.ho/folders.json.tmp". Copying temp file.

[2024.09.25] 14:33:23.044 [FileFunctions] Exception copying TMP file - SmarterMail.Standard.Files.Exceptions.SMNotFoundException: The process cannot access the file 'd:\SmarterMail\Domains\meghaeng.com\Users\lldpe.ho\folders.json.tmp' because it is being used by another process.

[2024.09.25]  ---> System.IO.IOException: The process cannot access the file 'd:\SmarterMail\Domains\meghaeng.com\Users\lldpe.ho\folders.json.tmp' because it is being used by another process.

[2024.09.25]    at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)

[2024.09.25]    at SmarterMail.Standard.Files.FileX.Copy(String sourceFileName, String destFileName)

[2024.09.25]    --- End of inner exception stack trace ---

[2024.09.25]    at SmarterMail.Standard.Files.FileX.Copy(String sourceFileName, String destFileName)

[2024.09.25]    at SmarterMail.Common.Files.FileFunctions.ReadFileFromDiskAsBytes(String path)

[2024.09.25] 14:33:23.135 [FileFunctions] File "d:/SmarterMail/Domains/meghaeng.com/accounts.json" missing. Found TMP file "d:/SmarterMail/Domains/meghaeng.com/accounts.json.tmp". Copying temp file.

[2024.09.25] 14:33:23.135 [FileFunctions] Exception copying TMP file - SmarterMail.Standard.Files.Exceptions.SMNotFoundException: The process cannot access the file 'd:\SmarterMail\Domains\meghaeng.com\accounts.json.tmp' because it is being used by another process.

[2024.09.25]  ---> System.IO.IOException: The process cannot access the file 'd:\SmarterMail\Domains\meghaeng.com\accounts.json.tmp' because it is being used by another process.

[2024.09.25]    at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)

[2024.09.25]    at SmarterMail.Standard.Files.FileX.Copy(String sourceFileName, String destFileName)

[2024.09.25]    --- End of inner exception stack trace ---

[2024.09.25]    at SmarterMail.Standard.Files.FileX.Copy(String sourceFileName, String destFileName)

[2024.09.25]    at SmarterMail.Common.Files.FileFunctions.ReadFileFromDiskAsBytes(String path)

Have you enabled audit logging to determine what process, exactly, is deleting those .json files?
 

ran perfmon for a day. Not able to find any other process that is touching the json files.

How do you have your filters configured in Process Monitor? Perhaps we can provide guidance on the best options. We have no known or reported issues involving JSON deletion or corruption, so there are only a few possibilities here: 

1. Third-party file access locking JSON files causing corruption/inability to read/write to them. 

2. Corruption of the disk or disk controller. 

3. Operating System concerns. 

Only errors I see is

[2024.09.30] 14:51:53.350 TlRMTVNTUAADAAAAGAAYAIgAAABoAWgBoAAAACQAJABYAAAABgAGAHwAAAAGAAYAggAAAAAAAAAIAgAABYKIogoAOTgAAAAPMzwpMB9xgJqgxagzDjNSDmIAagBlAHIAcgBpAG4AZwAtAGoAZQBuAHMAZQBuAC4AZABrAGIAYgBqAFQAUwAxAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAG+jnW5a/ypdkUa/Hczr11oBAQAAAAAAAEveN4Y3E9sBRq6CrXXgQfQAAAAAAgAWAFMATQBBAFIAVABFAFIATQBBAEkATAABABYAUwBNAEEAUgBUAEUAUgBNAEEASQBMAAQAFgBzAG0AYQByAHQAZQByAG0AYQBpAGwAAwAuAFMATQBBAFIAVABFAFIATQBBAEkATAAuAHMAbQBhAHIAdABlAHIAbQBhAGkAbAAFABYAcwBtAGEAcgB0AGUAcgBtAGEAaQBsAAcACABL3jeGNxPbAQYABAACAAAACAAwADAAAAAAAAAAAAAAAAAwAADpm7KgA4m7Hjo9biLw5IQh9g/8+Lg09WDjpHxxvo1SegoAEAAUKO8b9aEGD4Ph4r4YeFg2CQA6AGkAbQBhAHAALwBzAG0AYQByAHQAZQByAG0AYQBpAGwALgBjAGwAbwB1AGQAcAByAG8AcwAuAGQAawAAAAAAAAAAAA== Object reference not set to an instance of an object.
[2024.09.30]    at SmarterMail.Protocols.Authentication.AuthenticateMessage.TryParse(Byte[] data, String hostname, String& failure, List`1& logLines, Boolean removeFromCache)
[2024.09.30]    at SmarterMail.Protocols.Authentication.NegotiateMessages.Parse(Byte[] data, String& response, String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache, String userAgent)
[2024.09.30]    at SmarterMail.Protocols.Authentication.AuthenticationManager.Parse(String base64, String ip, String hostname, ProtocolTypes types, Boolean removeFromCache, String userAgent) 14:51:53.559

When looking for third-party file access I use these filters:

IF PATH CONTAINS C:\SMARTERMAIL (adjust for your pathing) THEN INCLUDE

This may return some results that are okay and some that are not. For an example, if you see an application listed that you know is using VSS (Volume Shadowcopy Services) this is less concerning than say an antivirus that does hard locks. So, you will still need to interpret the results but that should help narrow things down. 

[2024.10.01] 18:42:13.473 [FileFunctions] File "d:/SmarterMail/Domains/meghaeng.com/accounts.json" missing. Found TMP file "d:/SmarterMail/Domains/meghaeng.com/accounts.json.tmp". Copying temp file.

[2024.10.01] 18:42:13.474 [FileFunctions] Exception copying TMP file - SmarterMail.Standard.Files.Exceptions.SMNotFoundException: Could not find file 'd:\SmarterMail\Domains\meghaeng.com\accounts.json.tmp'.

[2024.10.01]  ---> System.IO.FileNotFoundException: Could not find file 'd:\SmarterMail\Domains\meghaeng.com\accounts.json.tmp'.

[2024.10.01] File name: 'd:\SmarterMail\Domains\meghaeng.com\accounts.json.tmp'

[2024.10.01]    at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)

[2024.10.01]    at SmarterMail.Standard.Files.FileX.Copy(String sourceFileName, String destFileName)

[2024.10.01]    --- End of inner exception stack trace ---

[2024.10.01]    at SmarterMail.Standard.Files.FileX.Copy(String sourceFileName, String destFileName)

[2024.10.01]    at SmarterMail.Common.Files.FileFunctions.ReadFileFromDiskAsBytes(String path)

accounts.json in archived folder is good this time. Why not keep the accounts.json multiple versions in archived folder and copy the archived file whenever the json is missing in the domain folder ?

I have observed that no .tmp is created at all.

Why to create a new empty file instead of taking the backed up version ?

It seems to be there is no failsafe mechanism build into the system. 

There is no answer from SM team why the system is stable up to version 8930 ? Why this issue started after that build. Whatever the reply by the team about we have not observed this issue with any other installation, there may be something wrong with your hardware/antivirus/backup tools is nothing but an excuse from your reluctance to accept that something went wrong after the builds 8930 where you started releasing common builds for win and linux.

There is no answer from SM team that out of 47 domains, why we are experiencing the issue with this domain only ?

Is the SM able to support larger installations ? 

We had a long trouble shooting session with AWS EC2, EBS and Windows support teams. They confirm that there is nothing wrong with instance we are currently running.

It is forcing our clients to look for another alternatives and leaving us. Try to understand loosing a customer with 15000 mail boxes who are happy for 3 years.

The disaster.

[2024.10.01] 18:49:49.730 [52.101.225.119][3622560] rsp: 550 <****aj.v******@meghaeng.com> No such user here

[2024.10.01] 18:50:31.347 [209.85.219.172][28832795] rsp: 550 <****chakrava******@meghaeng.com> No such user here

[2024.10.01] 19:23:34.616 [213.133.232.26][30541429] rsp: 550 <****r******@meghaeng.com> No such user here

[2024.10.01] 19:24:19.696 [213.133.232.26][49140087] rsp: 550 <****r******@meghaeng.com> No such user here

[2024.10.01] 19:24:51.029 [213.133.232.26][66860633] rsp: 550 <****r******@meghaeng.com> No such user here

[2024.10.01] 19:25:34.674 [213.133.232.26][19554153] rsp: 550 <****r******@meghaeng.com> No such user here

[2024.10.01] 19:26:45.570 [125.17.112.217][31629390] rsp: 550 <****galre******@meghaeng.com> No such user here

[2024.10.01] 19:33:25.065 [52.207.50.17][66741342] rsp: 550 <****endore.epotity5c******@meghaeng.com> No such user here

[2024.10.01] 19:33:25.128 [54.235.70.42][50344325] rsp: 550 <****endore.epotity3e******@meghaeng.com> No such user here

[2024.10.01] 19:33:25.162 [35.170.34.129][4634395] rsp: 550 <****endore.epotity0a******@meghaeng.com> No such user here

[2024.10.01] 19:33:26.243 [52.207.50.17][42865538] rsp: 550 <****emarald-abdule-6b******@meghaeng.com> No such user here

[2024.10.01] 19:33:26.393 [35.170.34.129][56538290] rsp: 550 <****emarald-abdule-11******@meghaeng.com> No such user here

[2024.10.01] 19:33:26.421 [54.235.70.42][64367228] rsp: 550 <****emarald-abdule-0c******@meghaeng.com> No such user here

[2024.10.01] 19:38:40.678 [106.10.242.216][3886845] rsp: 550 <****na.ara******@meghaeng.com> No such user here

[2024.10.01] 19:40:56.915 [106.10.240.149][55975226] rsp: 550 <****da******@meghaeng.com> No such user here

[2024.10.01] 19:47:27.735 [209.85.167.45][1058748] rsp: 550 <****eer******@meghaeng.com> No such user here

[2024.10.01] 20:02:37.906 [106.10.240.149][35703115] rsp: 550 <****i.myt******@meghaeng.com> No such user here

[2024.10.01] 21:27:34.570 [209.85.217.53][9473719] rsp: 550 <****nduri.kri******@meghaeng.com> No such user here

[2024.10.01] 22:53:50.114 [209.68.2.254][14279464] rsp: 550 <****ra.revanthr******@meghaeng.com> No such user here

[2024.10.01] 23:27:31.557 [45.84.128.42][38227896] rsp: 550 <****adrishekhar.mo******@meghaeng.com> No such user here

[2024.10.02] 00:05:34.959 [175.158.69.37][9631130] rsp: 550 <****kum******@meghaeng.com> No such user here

[2024.10.02] 00:06:32.647 [142.54.244.228][47935725] rsp: 550 <****aman******@meghaeng.com> No such user here

[2024.10.02] 00:07:38.372 [209.85.208.42][12494242] rsp: 550 <****shant.ch******@meghaeng.com> No such user here

[2024.10.02] 00:09:27.246 [139.138.44.163][36125794] rsp: 550 <****aysriva******@meghaeng.com> No such user here

[2024.10.02] 00:09:27.390 [139.138.44.163][36125794] rsp: 550 

I redacted some of the info. I see you do have a ticket on this. Please keep working with the agent while we look into the issue.