Back to Community Threads
2
Huge amount of RAM consumed in SM 17/100 Build 6928
Problem reported by Ionel Aurelian Rau - 1/3/2019 at 5:52 AM
Resolved
Hello!

Since updating to the last version of SM 17/100 (Build 6928), we noticed that RAM usage has been constantly increasing. Before updating to SM 17, in SM 16 the same VM had allocated 14GB of RAM and the SM service used in average about 3GB of RAM, with peaks to about 6-7GB.
RAM usage increased after upgrading to SM 17 (it was build 6922 at the time) and SM 17 used more like ~ 10GB in average.

Since then we`ve gone through the other 2 builds and now in Build 6928 the Smartermail service will use any amount of RAM available:
- since RAM usage started to get to above 95%, we increased the RAM allocated to the VM to 24GB from 14GB. After a frest restart, in 5 minutes of operation the RAM usage will again be above 95%. When SM process uses all available RAM, obviously performance for all users craters.
- we increased the RAM allocated to the VM to 32GB -> after 5-10 minutes the SM service uses again more than 95% of RAM. However, usage now fluctuates a bit and sometimes it goes below the max amount, so performance for users is bad, but now not absolutely unusable.
- we increased RAM allocation to the VM to 44GB -> now RAM usage is constantly above 36GB, with peaks to 44GB.. 
Of course that RAM usage fluctuates with the amount of users connected (also, we notices that the calendar view uses the most RAM), but this is ridiculous. I mean, we never exceeded 6-7GB of RAM in SM16 with the same no. of users which performed the same actions. I am sure that if we would allocate more RAM to the VM, SM process would use more. Restarting the SM process will also free up memory temporarily. 

Is anyone else seeing a suspiciously high RAM usage compared to SM 17?

22 Replies

Reply to Thread
0
Employee Replied
1/3/2019 at 6:22 AM
Employee Post
Ionel, what you reported previously with the calendars being inaccessible and one of the error messages returned, I believe there is an issue with that that is causing this memory consumption.  I would strongly recommend that you open a support ticket so that we can access your server and analysis the calendar(s) for your users.
0
Ionel Aurelian Rau Replied
1/3/2019 at 7:46 AM
0
echoDreamz Replied
1/3/2019 at 10:52 AM
Us as well, Our server sitting regularly at 15GB of usage on the SM process. v16 was usually around half that.
0
Ionel Aurelian Rau Replied
1/4/2019 at 7:18 AM
Apparently the high RAM usage was caused by some shared resources that SM 17 could not handle anymore. Detaching these from all the user`s mailboxes lowered the RAM usage to a more sane 9 - 15GB level.
0
Matt Petty Replied
1/4/2019 at 8:00 AM
Employee Post
I believe this is calendars that was causing this and more file IO than there needed to be. We have a fix we are sending out to tickets and the news from checking this morning looks like it worked for those users. I can send you this build if you DM me.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
Ionel Aurelian Rau Replied
1/4/2019 at 8:40 AM
Yes, the calendar definitely has some issues. We are already on custom Build 6942 of SM 17 - do you have a build newer than that?

Either way, the problem needs to be investigated further as RAM is now again increasing to outrageous amounts (spikes to 61 GB). Now the problematic shared resources are no longer mapped to any user, just the new ones. Also, Outlook just stopped syncing again via EAS and the logs get filled with SYNC KEY MISMATCH errors, as well as these:
[2019.01.04] 17:22:39.945 System.OutOfMemoryException: Array dimensions exceeded supported range.   at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired)   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)   at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)   at Newtonsoft.Json.JsonTextReader.ReadAsString()   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)   at SmarterMail.Common.Files.FileFunctions.ReadJsonFile[T](String path, String cacheBusterKey)   at SmarterMail.Common.Files.UserFiles.FoldersContentsFile.Read(BlockIdGenFile genFile, String dir, FolderTypes folder_type, Int64 id, String accountName, String cacheBusterKey)   at MailService.Repositories.DomainRepository.FolderContentsLoadFile(String accountName, FolderTypes type, Int64 folderId)   at MailService.Repositories.DomainRepository.AppointmentsByUID(String accountName, List`1 appointmentUids)   at MailService.Protocols.ActiveSync.Helpers.SyncCalendar.FilterAndGetSoftDeletedItems(Dictionary`2& smSyncItems, Dictionary`2 mappedItems)   at MailService.Protocols.ActiveSync.Helpers.SyncBase.PerformSync(Int32 allowedChanges, Boolean& requiresImmediateResponse, Boolean& saveLastSyncRequest)   at MailService.Protocols.ActiveSync.Commands.Sync.Execute()   at MailService.Protocols.ActiveSync.VersionSpecific.VersionHandlerBase.ProcessCommand()   at MailService.Protocols.ActiveSync.ActiveSyncProcessor.ProcessCommand(EasCommandRequest request)17:22:39.992 [1212729519] redacted@domain.com 865570604B5348D7B74A5566366ACF85 WindowsOutlook15 v14_0 (IP:)
0
Matt Petty Replied
1/4/2019 at 9:33 AM
Employee Post
I'd check to make sure you have atleast 100.0.6942.30788 which was built at 5pm MST yesterday. We've got one ticket who came back this morning reporting the CPU and RAM are down. I checked another server this morning and it also appears to be much better, as far as CPU and RAM are concerned. I'll send you a link to this build.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
echoDreamz Replied
1/4/2019 at 10:50 AM
Yeah, the "middle of the day" usage is certainly way down, though when our bandwidth and disk usage calculations run, it does seem to cause a major spike in CPU usage. It also does appear that the process handles are also under control now, before they would hit over 30k handles, now we are <10k which is normal.
0
echoDreamz Replied
1/4/2019 at 11:01 AM


Top one is yesterday's CPU usage until around 8:15PM when we did the upgrade to the build Matt talks about above. The bottom one is today's CPU usage thus far. So certainly way down. However you can see around midnight this morning until almost 3AM the major spike while our usage gather is running.

So well done thus far Matt and team on reducing the CPU usage.
0
Sébastien Riccio Replied
1/4/2019 at 2:04 PM
Sorry to "hijack" this topic, but I have a question. While reading the topic it's said that some issues are resolved in build 100.0.6942.30788, however the download page shows 6928. 

Has the download pages for new builds changed ?

Thanks for your answer
Sébastien Riccio System & Network Admin https://swisscenter.com
0
echoDreamz Replied
1/4/2019 at 2:35 PM
It was a custom build provided to users having CPU/RAM issues.
0
Sébastien Riccio Replied
1/4/2019 at 2:40 PM
Oh ok thanks. Then I think it will make it in the next public release.

Sébastien
Sébastien Riccio System & Network Admin https://swisscenter.com
0
Ionel Aurelian Rau Replied
1/23/2019 at 6:56 AM
Marked As Resolution
Hi all,

In the end it turns out that we had a shared calendar insider our calendar file which had a custom category that duplicated over and over again so that the file reached a size of ~ 800MB. If we removed all shared resources from all users, calendars would work, but with low performance and high system resource usage. But if we mapped any shared calendars, whether the problematic one, or even new ones, then RAM and CPU usage would shoot up to impossible levels (e.g. over 60GB of RAM used when we would normally see 6GB used).

Also, after we mapped any shared resources, Outlook would stop syncing mail via EAS for the users that did so.

Now the calendar file has been fixed and we are on the latest build - resource usage is back to normal levels and user experience is great.

So this issue appears to be fixed, at least after almost a day where users were again able to use shared resources.
1
Matt Petty Replied
1/23/2019 at 7:19 AM
Employee Post
Thanks for the update, its good that all 3 threads were one root cause. Pretty crazy what one bad calendar can do. Also our next minor will have code internally in SM that will passively remove the duplicates. So if anyone else is worried they may have this issue, just grab the next minor and you should be fine.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
Alex Clarke Replied
1/23/2019 at 12:51 PM
Matt, can you confirm the build number of the next minor and when it will be available please?
0
Employee Replied
1/28/2019 at 3:05 PM
Employee Post
Alex, Build 6964 was released on January 25.
0
Alex Clarke Replied
1/29/2019 at 11:48 AM
Thanks Robert. I'm currently on custom build 6967 (Jan 28, 2019) while trying to fix issue with high CPU/memory usage.
0
Hey Matt. Sorry not trying to hijack the thread.  I got a question
So, We are still using SM 14.  Planning to move to the new stuff sometime this summer.  What is the real amount of RAM we should expect to need in the server ?  With the current version 14, we are able to get away with just 2 gig of ram. But then again we have less than 250 email accounts. Is the amount of ram proportiante to the amount of accounts ?  When i see people above talking about 32gig and 60gig of RAM in the server, that is a little bit of a shocker for us as a small non profit org.
www.HawaiianHope.org - Providing technology services to non profit organizations, low income families, homeless shelters, clean and sober houses and prisoner reentry programs. Since 2015, We have refurbished over 11,000 Computers !
1
Ionel Aurelian Rau Replied
2/7/2019 at 4:28 AM
Curtis, the huge 60+ GB of RAM consumed was only during the problem with a broken calendar file. Please see the details above and in the threads I linked in previous comments. And during these issues, I think that the server would have consumed any amount of RAM we would have allocated - but again, this is not the normal behavior, just this particular case where we had a broken file.

After the issues have been fixed, we are seeing RAM usage that ranges between 1.2GB to ~ 10-11 GB with a server instance holding ~ 250 accounts, out of which at peak usage we have about 110 - 150 active users at the same time and most of them are using at least 2 clients (a mobile device and either Outlook and/or the WebMail interface) at the same time.

I am actually all in favor of using more RAM if that increases performance and limits the need to get data off HDD/SSD. SM 17/100 seems to do a decent job of dynamically allocating RAM as it is needed, so we`re quite happy about this and the current performance. There are minor issues, but these are another story.

Each morning we go from ~ 1GB of RAM usage to ~ 10 GB at mid-day, then back to 1GB. So resources are also released when not needed anymore.
0
Matt Petty Replied
2/7/2019 at 8:03 AM
Employee Post
Hello @Curtis,

We have some customers with 5000+, 8000+ domains on them and it can be little scary, even from my perspective. These setups really do squeeze what they can out of SmarterMail, these setups are rockin' dual slot xeon cores that have a processor count that make my eyes go big sometimes and storage setups that are quite impressive.

In most cases they'll actually be like us with about ~5-200 probably, so I just checked our server with almost the same amount of users. We are running in a VM that is partitioned 6x 2.6ghz virtual cores, it's currently floating in the 6%-16% range on CPU and we are at 980mb of RAM used.

I'd say atleast 2-4 cores with maybe 4gb of RAM in your case is all you should need.
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
Matt Petty Replied
2/7/2019 at 8:09 AM
Employee Post
@Ionel

I actually rewrote how we cache mailbox data in SM now, SM does the mailbox garbage collection itself that way I can drop the inactive mailboxes from memory when they haven't been used in a certain amount of time. Our new backend also now keeps a virtual version of most of the json files it writes to in a cache of memory. That means in most cases we aren't even reading directly off the disk and we save to it only when there are changes. This cache is also using .NET's MemoryCache object which is amazing because it will automatically release resources once the system becomes constrained for memory it other places and at I think about 80-85% it will get an alert to start dropping memory.

This is some of the power behind the new SmarterMail, the whole reason we spent months working on this. When we were starting to work on MAPI it became painfully obvious the limitations we had using a backend designed a long time ago. We've had some bumps along the way on the new system as we always do, but once it's ironed out it is gonna flex its muscles. 
Matt Petty Senior Software Developer SmarterTools Inc. www.smartertools.com
0
echoDreamz Replied
2/7/2019 at 10:28 AM
Honestly, If Smartermail was like SQL Server and ate RAM like candy, I wouldnt care, if the caching really helped performance etc. at the cost of some RAM, have at it. Ill toss 128GB of RAM in for SM and smile.
Back to Community Threads

Reply to Thread

Related Knowledge Base Articles

SmarterMail Server Performance TuningSmarterMail > Server Configuration and Management
Recommended SPAM SettingsSmarterMail > Antispam and Antivirus
Timeouts, Crashes and IP Binding Issues Caused by Server HardwareSmarterMail > Troubleshooting
Increase the Total Number of Emails Allowed in the SpoolSmarterMail > Installation and Configuration
500 Error After Upgrading to the Latest BuildSmarterMail > Troubleshooting