Huge amount of RAM consumed in SM 17/100 Build 6928
Problem reported by Ionel Aurelian Rau - January 3 at 5:52 AM
Being Fixed
Hello!

Since updating to the last version of SM 17/100 (Build 6928), we noticed that RAM usage has been constantly increasing. Before updating to SM 17, in SM 16 the same VM had allocated 14GB of RAM and the SM service used in average about 3GB of RAM, with peaks to about 6-7GB.
RAM usage increased after upgrading to SM 17 (it was build 6922 at the time) and SM 17 used more like ~ 10GB in average.

Since then we`ve gone through the other 2 builds and now in Build 6928 the Smartermail service will use any amount of RAM available:
- since RAM usage started to get to above 95%, we increased the RAM allocated to the VM to 24GB from 14GB. After a frest restart, in 5 minutes of operation the RAM usage will again be above 95%. When SM process uses all available RAM, obviously performance for all users craters.
- we increased the RAM allocated to the VM to 32GB -> after 5-10 minutes the SM service uses again more than 95% of RAM. However, usage now fluctuates a bit and sometimes it goes below the max amount, so performance for users is bad, but now not absolutely unusable.
- we increased RAM allocation to the VM to 44GB -> now RAM usage is constantly above 36GB, with peaks to 44GB.. 
Of course that RAM usage fluctuates with the amount of users connected (also, we notices that the calendar view uses the most RAM), but this is ridiculous. I mean, we never exceeded 6-7GB of RAM in SM16 with the same no. of users which performed the same actions. I am sure that if we would allocate more RAM to the VM, SM process would use more. Restarting the SM process will also free up memory temporarily. 

Is anyone else seeing a suspiciously high RAM usage compared to SM 17?

12 Replies

Reply to Thread
0
Robert Emmett Replied
Employee Post
Ionel, what you reported previously with the calendars being inaccessible and one of the error messages returned, I believe there is an issue with that that is causing this memory consumption.  I would strongly recommend that you open a support ticket so that we can access your server and analysis the calendar(s) for your users.
Robert Emmett
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Ionel Aurelian Rau Replied
0
echoDreamz Replied
Us as well, Our server sitting regularly at 15GB of usage on the SM process. v16 was usually around half that.

Christopher

0
Ionel Aurelian Rau Replied
Apparently the high RAM usage was caused by some shared resources that SM 17 could not handle anymore. Detaching these from all the user`s mailboxes lowered the RAM usage to a more sane 9 - 15GB level.

0
Matt Petty Replied
Employee Post
I believe this is calendars that was causing this and more file IO than there needed to be. We have a fix we are sending out to tickets and the news from checking this morning looks like it worked for those users. I can send you this build if you DM me.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Ionel Aurelian Rau Replied
Yes, the calendar definitely has some issues. We are already on custom Build 6942 of SM 17 - do you have a build newer than that?

Either way, the problem needs to be investigated further as RAM is now again increasing to outrageous amounts (spikes to 61 GB). Now the problematic shared resources are no longer mapped to any user, just the new ones. Also, Outlook just stopped syncing again via EAS and the logs get filled with SYNC KEY MISMATCH errors, as well as these:
[2019.01.04] 17:22:39.945 System.OutOfMemoryException: Array dimensions exceeded supported range.   at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired)   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)   at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)   at Newtonsoft.Json.JsonTextReader.ReadAsString()   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)   at SmarterMail.Common.Files.FileFunctions.ReadJsonFile[T](String path, String cacheBusterKey)   at SmarterMail.Common.Files.UserFiles.FoldersContentsFile.Read(BlockIdGenFile genFile, String dir, FolderTypes folder_type, Int64 id, String accountName, String cacheBusterKey)   at MailService.Repositories.DomainRepository.FolderContentsLoadFile(String accountName, FolderTypes type, Int64 folderId)   at MailService.Repositories.DomainRepository.AppointmentsByUID(String accountName, List`1 appointmentUids)   at MailService.Protocols.ActiveSync.Helpers.SyncCalendar.FilterAndGetSoftDeletedItems(Dictionary`2& smSyncItems, Dictionary`2 mappedItems)   at MailService.Protocols.ActiveSync.Helpers.SyncBase.PerformSync(Int32 allowedChanges, Boolean& requiresImmediateResponse, Boolean& saveLastSyncRequest)   at MailService.Protocols.ActiveSync.Commands.Sync.Execute()   at MailService.Protocols.ActiveSync.VersionSpecific.VersionHandlerBase.ProcessCommand()   at MailService.Protocols.ActiveSync.ActiveSyncProcessor.ProcessCommand(EasCommandRequest request)17:22:39.992 [1212729519] redacted@domain.com 865570604B5348D7B74A5566366ACF85 WindowsOutlook15 v14_0 (IP:)
0
Matt Petty Replied
Employee Post
I'd check to make sure you have atleast 100.0.6942.30788 which was built at 5pm MST yesterday. We've got one ticket who came back this morning reporting the CPU and RAM are down. I checked another server this morning and it also appears to be much better, as far as CPU and RAM are concerned. I'll send you a link to this build.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
echoDreamz Replied
Yeah, the "middle of the day" usage is certainly way down, though when our bandwidth and disk usage calculations run, it does seem to cause a major spike in CPU usage. It also does appear that the process handles are also under control now, before they would hit over 30k handles, now we are <10k which is normal.

Christopher

0
echoDreamz Replied


Top one is yesterday's CPU usage until around 8:15PM when we did the upgrade to the build Matt talks about above. The bottom one is today's CPU usage thus far. So certainly way down. However you can see around midnight this morning until almost 3AM the major spike while our usage gather is running.

So well done thus far Matt and team on reducing the CPU usage.

Christopher

0
Sébastien Riccio Replied
Sorry to "hijack" this topic, but I have a question. While reading the topic it's said that some issues are resolved in build 100.0.6942.30788, however the download page shows 6928. 

Has the download pages for new builds changed ?

Thanks for your answer
0
echoDreamz Replied
It was a custom build provided to users having CPU/RAM issues.

Christopher

0
Sébastien Riccio Replied
Oh ok thanks. Then I think it will make it in the next public release.

Sébastien

Reply to Thread