7496 trouncing my server
Problem reported by Robert Simpson - 7/12/2020 at 2:43 PM
Submitted
The last couple versions have been just hammering my server into oblivion.  CPU racks up to 90% usage, memory hits to about 1.7gb, and the mailservice becomes unresponsive to web or Outlook.  I grabbed a debug DMP file and ran WINDBG on it ... disk activity is at a minimum, no indexes are being built at the time.  Here's what I'm learning:

0:000> !runaway
 User Mode Time
  Thread       Time
   47:16d0     0 days 0:23:15.968
   51:1070     0 days 0:20:47.906
   53:12d0     0 days 0:16:57.500
   50:18ec     0 days 0:15:43.843
   52:1444     0 days 0:13:14.609
   56:1b38     0 days 0:12:27.953
The service has been running under a day ... CPU activity was extremely high and high memory usage.  These top threads by CPU were all stuck in a GC loop, most looked something like this:

0:053> !mex.t 47
DbgID ThreadID            User  Kernel COM-Initialized             Create Time (UTC)
47    16d0 (0n5840) 23m:15.968 20s.828 APTKIND_MULTITHREADED (MTA) 07/12/2020 07:14:18.026 PM

 # Child-SP         Return           Call Site
 0 000000152c19d338 00007ff9c26e1118 ntdll!NtWaitForSingleObject+0xa
 1 000000152c19d340 00007ff9bb4c74b8 KERNELBASE!WaitForSingleObjectEx+0x94
 2 000000152c19d3e0 00007ff9bb4c746e clr!CLREventWaitHelper2+0x3c
 3 000000152c19d420 00007ff9bb4c73d4 clr!CLREventWaitHelper+0x1f
 4 000000152c19d480 00007ff9bb343569 clr!CLREventBase::WaitEx+0x71
 5 000000152c19d510 00007ff9bb34d10a clr!SVR::gc_heap::wait_for_gc_done+0x5a
 6 000000152c19d540 00007ff9bb2fda8f clr!SVR::GCHeap::Alloc+0x1ae
 7 000000152c19d5b0 00007ff9bb337151 clr!SlowAllocateString+0x1cf
 8 000000152c19d650 00007ff9bb33754f clr!StringObject::NewString+0x25
 9 000000152c19d680 00007ff9bb33738d clr!Int32ToDecStr+0xdf
 a 000000152c19d920 00007ff9ba2d7e4a clr!COMNumber::FormatInt32+0x10d
 b 000000152c19dbf0 00007ff9ba2d7a6b mscorlib_ni!System.Text.StringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ParamsArray)$##6006603+0x38a
 c 000000152c19dcc0 00007ff95d203dce mscorlib_ni!System.String.FormatHelper(System.IFormatProvider, System.String, System.ParamsArray)$##600054D+0x6b

In a VS debugger, the stacks are less detailed, but read:
>	ntdll.dll!NtWaitForSingleObject()	Unknown
 	KERNELBASE.dll!WaitForSingleObjectEx()	Unknown
 	[Managed to Native Transition]	
 	mscorlib.dll!System.Text.StringBuilder.AppendFormatHelper(System.IFormatProvider provider, string format, System.ParamsArray args) Line 1464	C#
 	mscorlib.dll!string.FormatHelper(System.IFormatProvider provider, string format, System.ParamsArray args) Line 3020	C#
 	MailService.exe!MailService.Protocols.MAPI.ServerHandles.FastTransferDownloadContext.GetBytes(int maximumBufferSize, out bool done)	Unknown
 	MailService.exe!MailService.Protocols.MAPI.Rops.FastTransferRops.RopFastTransferSourceGetBuffer.Process(MailService.Protocols.MAPI.ServerHandles.FastTransferDownloadContext fastTransfer)	Unknown
 	MailService.exe!MailService.Protocols.MAPI.Rops.FastTransferRops.RopFastTransferSourceGetBuffer.Process(System.Collections.Generic.List<uint> payloadServerObjectHandleTable)	Unknown
 	MailService.exe!MailService.Protocols.MAPI.RequestTypes.ExecuteRequest.Process()	Unknown
 	MailService.exe!MailService.Protocols.MAPI.MapiProcessor.Process()	Unknown
 	MailService.exe!MailService.WCF.MAPI.MapiService.GetResponse(System.Guid ticketGuid)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(object instance, object[] inputs, out object[] outputs)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(ref System.ServiceModel.Dispatcher.MessageRpc rpc)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(ref System.ServiceModel.Dispatcher.MessageRpc rpc)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(ref System.ServiceModel.Dispatcher.MessageRpc rpc)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.MessageRpc.Process(bool isOperationContextSet)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext request, bool cleanThread, System.ServiceModel.OperationContext currentOperationContext)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext request, System.ServiceModel.OperationContext currentOperationContext)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult result)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult result)	Unknown
 	System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result)	Unknown
 	System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(System.IAsyncResult result)	Unknown
 	System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result)	Unknown
 	System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(object state)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(object state)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete(bool haveResult, int error, int numBytes)	Unknown
 	System.ServiceModel.dll!System.ServiceModel.Channels.OverlappedContext.CompleteCallback(uint error, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped)	Unknown
 	System.ServiceModel.Internals.dll!System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(uint error, uint bytesRead, System.Threading.NativeOverlapped* nativeOverlapped)	Unknown
 	mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP) Line 135	C#
 	[Native to Managed Transition]	
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown
Debugging managed objects, some of the largest objects are:
Object Type									Count	Size (bytes)	Inclusive Size (bytes)
MailService.Protocols.MAPI.ServerHandles.FastTransferContentsSyncContext	42	125,402,192	205,232,312
InternalBufferManager+PooledBufferManager+BufferPool+LargeBufferPool		139	27,542,248	27,542,248
List<ValueTuple<Boolean, UInt64, Boolean, Boolean>>				42	21,498,472	21,498,472
SmarterMail.Standard.Exchange.Ids.GlobalCounter					509,223	20,368,920	20,368,920
I can send you guys this DMP file.  So far 7496 has crashed on me twice with an exception like this one:
0:047> !mex.PrintException2 00000010d34a11f8
Address: 0x00000010d34a11f8
HResult: 0x80131506
Type: System.ExecutionEngineException
Message: 
Stack Trace:
SP IP Function Source

3 Replies

Reply to Thread
0
Robert Simpson Replied
Just in the short time since I restarted the service, it's back up to >2gb of memory usage, and 90% CPU again.  I have only 2 users on MAPI right now.

0
echoDreamz Replied
You may want to open a ticket for this issue.
0
Kyle Kerst Replied
Employee Post
This is being investigated in Support and will follow up here when complete. Please note however that some higher CPU and memory utilization is expected in the latest releases as they have been designed to leverage system resources where possible to give users a better experience, and indexing has recently received some heavy optimizations that will require a reindex of user accounts. If higher than average CPU doesn't clear up in a few days to a week after your upgrade a support ticket would be the best next step. 
Kyle Kerst
Technical Support Specialist
SmarterTools Inc.
(877) 357-6278
www.smartertools.com

Reply to Thread