What is your indexing activity like?

449 boxes currently listed, but it is stepping through and completing and moving on to the next like normal.

Haven't really paid attention to know what a "normal" load of indexing would be on my system though.

We saw the same issue on one of our servers after upgrading to build 7053. I seems to be related to traffic coming in over IIS as restarting IIS (without restarting smartermail) brings CPU usage back to normal for some time in our case.

Support recommended to run procdump as follows to collect more information:

Download procdump: https://docs.microsoft.com/en-us/sysinternals/downloads/procdump
Run procdump: procdump.exe MailService.exe -ma -accepteula %USERPROFILE%\Desktop\ -c 75 -n 3

I took down the IIS server altogether... 100% CPU remains.

Put back online now.... still 100% CPU.

I've opened a ticket on this, something is not right.

Thanks everyone for the ideas.

Will post results once a fix is figured out.

Rod

Rod, we did make some changes to the indexing max thread count in 7016 and corrected some high CPU causes in 7008 as well, so there were changes between 6985 and current that could explain this. My thoughts are that we're likely in the process of cleaning up the indexes for users who had issues in there to begin with. 

The SM folks are in the server now.

I hear where you are coming from Kyle, my server was still littered with the XML vs JSON config issue to, was that cleanup part of this update too?

Watching what's going on so far it appears to be a combination of the SM service and from  time to time clamd will spike as well.   One thing they have identified is they think we may be configured with too few CPUs in the VM currently.  

Rod, we do keep quite a few of the XMLs around for safety's sake as they do come in handy when restoring old user configurations, etc. I know that cleanup is due to be looked at, but we'll likely still keep at least portions of the files around for restore purposes at least in the short term. 

As to the CPUs, this could definitely cause some of these problems. Any time SmarterMail runs short on resources the server will begin queuing up requests, consuming more memory, using garbage collection more frequently, and subsequently beginning to page for memory due to the increase in resource utilization caused by the queuing. Anything you can do to improve the available CPU cores and memory should help!

And my response to that is... sure, throw more CPU at it to make a CPU issue "vanish".

If it was an issue before the update I could see it, but it wasn't.

I'm waiting for my VMware guys to see if it is an option.

Not my area of expertise at all. :)

We had the exact same issue on all three of our servers immediately after upgrading to 7053. We doubled the CPU cores on the server from 4 to 8 that was Hyper-V but it was happening to two bare-metal servers too. We let it run 8 hours at 100% CPU and it didn't ever decrease, even for a second, so I doubt it is an Indexing issue, especially since we had the exact same 100% CPU issue on our two Incoming Gateways after upgrade (which only have one user). It was causing a > 12 min delivery delay from SmarterMail Gateway to SmarterMail and another > 12 min delivery delay to get to the user's Inbox. Eventually the Spool just filled up to 4000+ messages before we had to pull the plug.

Ultimately, we had to roll back to build 7040.

Post-Script: We originally used the MSI install for 7053. Even tried uninstalling, rebooting, and doing the EXE install with the exact same results. Restarting IIS didn't resolve the issue on any server for us.

Well, techs pulled logs from the server, it is in engineering now is my understanding.

They used the tool Christian Schmit mentioned above as well as installed some dotnet tracing tool as well to capture what was going on in the program while the CPU issue was taking place.

Still not impacting clients so hoping a MOVE FORWARD fix is the answer.

I HATE reverting... it seems to never ever go well.

Rod

Temporary conclusion, but no resolution yet:

The development team seems to think the issue may stem from the White/Black list not being cached.

So every-time an IP comes into the server is having to do a database dip (or file read... I'm not sure which).

They are building a caching mechanism for that now.

No timeline for the fix as of yet... but I'm askin'

Rod

Rod,

That makes a lot more sense for the cause of the high CPU usage. I know we have a *HUGE* Whitelist/Blacklist/SMTP Blocks on all of our SM Servers, including the Gateways...I think @ 13K entries, so that wouldn't surprise me that the build would behave perfectly fine for some installations but then go crazy nuts on other installations, including Gateways. 

Good to know that a fix is in the pipeline. Thanks for keeping us updated.

Yeah ours is pretty big too... 

I have a rule that if I see 20 or more IPs hacking from a class C, the entire class C gets blocked.

May torque some people, but it works.

Quick update... SM has a PATCH 7065 that I'll be installing late night Saturday 5/11/19 ... will post back after that installation and some monitoring / testing.   My understanding is they simply setup some things that were not being cached to be cached.  Guess we'll see how it plays out.

As an aside... for Enterprise Edition customers... shouldn't it really include 2 full licenses of whatever size you have purchased?   Sure would have been nice to be able to simply use failover here to switch traffic to one machine, install patch on other machine, then switch it back... just my $.02.

Heads up... make that patch 7066 will be applied this weekend as an issue was discovered during SM internal testing that had to be repaired.

Rod

Hello,

I have the same problem, after upgrade to 7053.

zito

Hi Luis.  Can you uninstall your current version of SmarterMail, and then install this build.

WooHoo... an earlier guinea pig. :)

Last nights build 7068 has solved the high CPU usage problem for us.

Kyle, should i be loading 7066 or the public released 7068 this weekend?   The release notes seem  to indicate the bl/wl caching is included in the released version.

Rod, please use the public release 7068.  It does include the changes in the custom build.

Through the first peak period of traffic, and things are looking much better.

I have identified a potential issue with ClamAV churning a bit and using all the processors for up to 10 seconds at a time but in general... the 7068 update fixed the biggest part of the High CPU issue.

Thanks SM folks!