The last couple versions have been just hammering my server into oblivion. CPU racks up to 90% usage, memory hits to about 1.7gb, and the mailservice becomes unresponsive to web or Outlook. I grabbed a debug DMP file and ran WINDBG on it ... disk activity is at a minimum, no indexes are being built at the time. Here's what I'm learning:
0:000> !runaway
User Mode Time
Thread Time
47:16d0 0 days 0:23:15.968
51:1070 0 days 0:20:47.906
53:12d0 0 days 0:16:57.500
50:18ec 0 days 0:15:43.843
52:1444 0 days 0:13:14.609
56:1b38 0 days 0:12:27.953
The service has been running under a day ... CPU activity was extremely high and high memory usage. These top threads by CPU were all stuck in a GC loop, most looked something like this:
0:053> !mex.t 47
DbgID ThreadID User Kernel COM-Initialized Create Time (UTC)
47 16d0 (0n5840) 23m:15.968 20s.828 APTKIND_MULTITHREADED (MTA) 07/12/2020 07:14:18.026 PM
# Child-SP Return Call Site
0 000000152c19d338 00007ff9c26e1118 ntdll!NtWaitForSingleObject+0xa
1 000000152c19d340 00007ff9bb4c74b8 KERNELBASE!WaitForSingleObjectEx+0x94
2 000000152c19d3e0 00007ff9bb4c746e clr!CLREventWaitHelper2+0x3c
3 000000152c19d420 00007ff9bb4c73d4 clr!CLREventWaitHelper+0x1f
4 000000152c19d480 00007ff9bb343569 clr!CLREventBase::WaitEx+0x71
5 000000152c19d510 00007ff9bb34d10a clr!SVR::gc_heap::wait_for_gc_done+0x5a
6 000000152c19d540 00007ff9bb2fda8f clr!SVR::GCHeap::Alloc+0x1ae
7 000000152c19d5b0 00007ff9bb337151 clr!SlowAllocateString+0x1cf
8 000000152c19d650 00007ff9bb33754f clr!StringObject::NewString+0x25
9 000000152c19d680 00007ff9bb33738d clr!Int32ToDecStr+0xdf
a 000000152c19d920 00007ff9ba2d7e4a clr!COMNumber::FormatInt32+0x10d
b 000000152c19dbf0 00007ff9ba2d7a6b mscorlib_ni!System.Text.StringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ParamsArray)$##6006603+0x38a
c 000000152c19dcc0 00007ff95d203dce mscorlib_ni!System.String.FormatHelper(System.IFormatProvider, System.String, System.ParamsArray)$##600054D+0x6b
In a VS debugger, the stacks are less detailed, but read:
> ntdll.dll!NtWaitForSingleObject () Unknown
KERNELBASE.dll!WaitForSingleObjectEx () Unknown
[Managed to Native Transition]
mscorlib.dll!System.Text.StringBuilder.AppendFormatHelper(System.IFormatProvider provider, string format, System.ParamsArray args) Line 1464 C#
mscorlib.dll!string.FormatHelper(System.IFormatProvider provider, string format, System.ParamsArray args) Line 3020 C#
MailService.exe!MailService.Protocols.MAPI.ServerHandles.FastTransferDownloadContext.GetBytes(int maximumBufferSize, out bool done) Unknown
MailService.exe!MailService.Protocols.MAPI.Rops.FastTransferRops.RopFastTransferSourceGetBuffer.Process(MailService.Protocols.MAPI.ServerHandles.FastTransferDownloadContext fastTransfer) Unknown
MailService.exe!MailService.Protocols.MAPI.Rops.FastTransferRops.RopFastTransferSourceGetBuffer.Process(System.Collections.Generic.List<uint> payloadServerObjectHandleTable) Unknown
MailService.exe!MailService.Protocols.MAPI.RequestTypes.ExecuteRequest.Process() Unknown
MailService.exe!MailService.Protocols.MAPI.MapiProcessor.Process() Unknown
MailService.exe!MailService.WCF.MAPI.MapiService.GetResponse(System.Guid ticketGuid) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(object instance, object[] inputs, out object[] outputs) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(ref System.ServiceModel.Dispatcher.MessageRpc rpc) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(ref System.ServiceModel.Dispatcher.MessageRpc rpc) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(ref System.ServiceModel.Dispatcher.MessageRpc rpc) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.MessageRpc.Process(bool isOperationContextSet) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext request, bool cleanThread, System.ServiceModel.OperationContext currentOperationContext) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext request, System.ServiceModel.OperationContext currentOperationContext) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult result) Unknown
System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult result) Unknown
System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result) Unknown
System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(System.IAsyncResult result) Unknown
System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result) Unknown
System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(object state) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(object state) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete(bool haveResult, int error, int numBytes) Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.OverlappedContext.CompleteCallback(uint error, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped) Unknown
System.ServiceModel.Internals.dll!System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(uint error, uint bytesRead, System.Threading.NativeOverlapped* nativeOverlapped) Unknown
mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP) Line 135 C#
[Native to Managed Transition]
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
Debugging managed objects, some of the largest objects are:
Object Type Count Size (bytes) Inclusive Size (bytes)
MailService.Protocols.MAPI.ServerHandles.FastTransferContentsSyncContext 42 125,402,192 205,232,312
InternalBufferManager+PooledBufferManager+BufferPool+LargeBufferPool 139 27,542,248 27,542,248
List<ValueTuple<Boolean, UInt64, Boolean, Boolean>> 42 21,498,472 21,498,472
SmarterMail.Standard.Exchange.Ids.GlobalCounter 509,223 20,368,920 20,368,920
I can send you guys this DMP file. So far 7496 has crashed on me twice with an exception like this one:
0:047> !mex.PrintException2 00000010d34a11f8
Address: 0x00000010d34a11f8
HResult: 0x80131506
Type: System.ExecutionEngineException
Message:
Stack Trace:
SP IP Function Source