2
[Beta] Client reporting his maillists are gone
Problem reported by echoDreamz - 2/5/2020 at 5:48 PM
Resolved

Client says his maillists are gone and when he tries to view them, he gets this error. He did say it worked about a week ago, so something possibly within the last update or 2 with SmarterMail.

19 Replies

Reply to Thread
0
Sébastien Riccio Replied
We have a clone of our production (pre-BETA) mailserver running with BETA for testing issues.
I checked ~50 customers mailing lists on the clone and can't reproduce the issue.

" seems to be the double quotes char.
The message seems to mean an ending double quote is missing somewhere in a file (probably a json file)

My guess here is maybe that some files related to his mailing list is corrupted, maybe due to the constant crashes you had recently with the crashy build, as SmarterMail tends to leave corrupted json files (really annoying) when crashing.

Have you checked the user's json files related to mailing list?

Maybe one of these is corrupt:
list-subscribers.jsons
mailinglist-data.json
mailinglist-history.jsons

While writing this, I notice there are .jsons file (ending with a s). Is this a thing jsonS files or it's a SmarterMail speciality ?

ps: as we had some crashes on our production server too, I've wrote a little dirty powershell script to check json files integrity recursively from the folder where you start it. This helped us discover broken accounts after a crash before being notified by our customers...

It logs json files that need attention to invalid.txt

I don't know if it would work on .jsons files though, not tested yet as i just discovered there where files named like this.
EDIT: I've edited the script to also check .jsons files as it seems to also work for these files.

Get-ChildItem -Path ".\*.json*" -recurse | ForEach-Object {
    try {
        $powershellRepresentation = Get-Content $_.FullName | ConvertFrom-Json  -ErrorAction Stop;
        $validJson = $true;
    } catch {
        $validJson = $false;
    }

    if ($validJson) {
        Write $_.FullName | Out-File valid.txt -Append 
    } else {
        Write-Host $_.FullName FAILED!
        Write $_.FullName | Out-File invalid.txt -Append 
    } 

} 
 
Sébastien Riccio System & Network Admin https://swisscenter.com
0
echoDreamz Replied
Looks like the file is not completed. Not sure what happened, but the list-subscribers.jsons file looks like it only halfway saved.
0
Kyle Kerst Replied
Employee Post
Thats absolutely right @echoDreamz. Recommendation is to restore the pertinent JSON files from the Archived Data folder. These Archived Data folders contain several days worth of backup JSON files so that you have a restore point! JSON corruption can be caused by a variety of things unfortunately:

- Sudden restart or power failure.
- Third party access to JSON files (AV solutions, backups, etc)
- Drive failure

Hope that helps!
Kyle Kerst IT Coordinator SmarterTools Inc. www.smartertools.com
2
Sébastien Riccio Replied
By experience we also noticed that these corruptions can also happen when mailservice.exe crashes or if you have to kill the process.
But most of the time it's on user settings.json or folders.json.
I guess this happens when the process terminate in the middle of writing to these files :/
Sébastien Riccio System & Network Admin https://swisscenter.com
0
echoDreamz Replied
Several days does no good if it was corrupted weeks ago unfortunately.

no drive failures, no power loss, av has the entire SM data array excluded. My bet is the Sm process crashing. 
0
Alex Clarke Replied
Kyle, is there risk of .json file corruption when manually stopping the SM service (or when uninstalling during an upgrade)?
1
Sébastien Riccio Replied
AFAIK, it's not happening when gracefully stopping the service, only when it crashes/stops abrubtly (kill)


Sébastien Riccio System & Network Admin https://swisscenter.com
0
Sébastien Riccio Replied
As a system admin, I would find it very useful that smartertools provide an official utility for intergrity checking.

Something like scanning these corruptable json(s) files and, if any problem is found, reporting it and also an option to automatically restore them from a valid version from archive folder.

This is something that could be a separate executable or maybe added to the admin troubleshooting pages.

The same for GRP files would also be really nice.

Sébastien Riccio System & Network Admin https://swisscenter.com
0
Alex Clarke Replied
I’m sure I read somewhere that SM would restore corrupt .json files automatically. 

Guessing this isn’t the case!
0
Sébastien Riccio Replied
Alex, that would be interresting to know if it is supposed to do it. However in our experiences it shows that it actually doesn't or at least not all corrupted files.

Sébastien Riccio System & Network Admin https://swisscenter.com
1
Matt Petty Replied
Employee Post
Hello,

    This should no longer occur with settings.json (or any json) file. However I did some curious investigation and realized that these files for mailing lists use a different mechanism for saving, "jsons" file types.

For context
We used to write directly to a file, streaming the data into it. The problem with this comes when the process suddenly shuts down, we now have a file half written. A couple months ago we made changes to "json" files that prevent this behavior. We now write to a secondary file "X.json.tmp" then we tell the OS to swap the files so this happens all at one time from our program's perspective. This could mean a .tmp file gets half-written but on a crash it WONT replace the GOOD json file that's already there, preventing issues.

With the context out of the way, I have added a new task to our tracker to apply these same fixes to jsons files and to evaluate all of our file types and ensure this same "json" behavior applies to all our other files.

Will post here once we have applied these changes, thanks for reporting them.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
Alex Clarke Replied
Matt, what build was this change introduced?

We've noticed issues with the settings.json files containing just blank spaces and also some other .json files being partially written (i.e. missing closing '}' or ']').
0
echoDreamz Replied
Thanks Matt!
1
Matt Petty Replied
Employee Post
We initially introduced this new backend for saving files back in our release of version 17, we applied the above change in mid-October. Nothing about the beta changed the way these files save. Likely the recent crashing issues triggered this behavior.

EDIT: Also Sébastien Riccio I like your idea for an integrity checker, I may write up something that checks to make sure "json" files can be serialized into a JSON object and "jsons" files have a complete JSON object on each line, no half written objects.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
Alex Clarke Replied
Thanks for the update Matt. 
0
echoDreamz Replied
For us, this file went corrupt on December 23rd. Shortly after we did the MAPI beta roll out. When we were having major stability issues.
0
echoDreamz Replied
What is the correct way to remove these lists since they do not load?  We need to rebuild them, but I dont want to go around mucking up anything else.
0
Matt Petty Replied
Employee Post
You can remove the list-subscribers.jsons file, it will start back at zero with your subscribers. Retain the old file just in case you need to reference it. You can also try removing the very last (broken) line in the file.
If you have a complete json object on each line, begins/ends with { and }, then it should load. So you may not even need to remove the file, just the last line of the file. I believe it's ok if the last line is left blank, as long as long as it's an empty line with nothing else in it.

What's effectively lost if not backed up in this case is the broken line and what would've followed it, everything before hand should be valid.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
echoDreamz Replied
Just had another user report this today, this time, their list-subscribers.jsons file went corrupt about 15 minutes ago. Which is also around the same time we stopped the SM service to run the folderfix on 2 broken users.

Appears we shutdown the service as they  were sending out their newsletter. Could this have caused any corruption?

Reply to Thread