This bug started showing up out of nowhere in September 2014. At the time I was running on 9.5.5091.
I just got around to tracking down the problem and trying to fix it, so I upgraded to 9.5.5318 (the last available version of 9) but the problem still keeps happening.
When this issue happens, there are 3 event log entries: 2 regarding the crash of SmarterTrack and 1 regarding it taking out IIS along with it. One other bit of information is that I've checked the weblogs and there's nothing accessing the web interface when these crashes happen, so it is coming out of some internally triggered event.
----------------------------------------------------------------------------------------------------
Event Log: Application
Source: ASP.NET 4.0.30319.0
Event ID: 1325
1/23/2015 7:12:06 PM
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/2/ROOT
Process ID: 2380
Exception: System.Data.SqlClient.SqlException
Message: Violation of PRIMARY KEY constraint 'PK_st_GlobalSettings'. Cannot insert duplicate key in object 'dbo.st_GlobalSettings'. The duplicate key value is (Options, thread_indexerrunning).
The statement has been terminated.
StackTrace: at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Microsoft.ApplicationBlocks.Data.SqlHelperInternal.ExecuteReader(SqlConnection , SqlTransaction , CommandType , String , SqlParameter[] , #XZc )
at SmarterTrack.Data.GlobalSettingEntityBase.Save_SQL(IDbConnection& connection)
at SmarterTrack.Data.GlobalSettingEntityBase.Save(IDbConnection& connection)
at SmarterTrack.Data.GlobalSettingEntityBase.Save()
at SmarterTrack.Data.GlobalSettingEntity.SetSetting(D v, String val)
at SmarterTrack.MRS.Global.indexEngine_ThreadLoopBeginning(Object sender, ThreadLoopEventArgs e)
at SmarterTools.Indexing.Engine.#cac()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
----------------------------------------------------------------------------------------------------
Event Log: Application
Source: .NET Runtime
Event ID: 1026
1/23/2015 7:12:06 PM
Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Data.SqlClient.SqlException
Stack:
at Microsoft.ApplicationBlocks.Data.SqlHelperInternal.ExecuteReader(System.Data.SqlClient.SqlConnection, System.Data.SqlClient.SqlTransaction, System.Data.CommandType, System.String, System.Data.SqlClient.SqlParameter[], #XZc)
at SmarterTrack.Data.GlobalSettingEntityBase.Save_SQL(System.Data.IDbConnection ByRef)
at SmarterTrack.Data.GlobalSettingEntityBase.Save(System.Data.IDbConnection ByRef)
at SmarterTrack.Data.GlobalSettingEntityBase.Save()
at SmarterTrack.Data.GlobalSettingEntity.SetSetting(D, System.String)
at SmarterTrack.MRS.Global.indexEngine_ThreadLoopBeginning(System.Object, ThreadLoopEventArgs)
at SmarterTools.Indexing.Engine.#cac()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()
----------------------------------------------------------------------------------------------------
Event Log: Application
Source: Application Error
Event ID: 1000
1/23/2015 7:12:07 PM
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18409, time stamp: 0x5315a05a
Exception code: 0xe0434352
Fault offset: 0x000000000000940d
Faulting process id: 0x94c
Faulting application start time: 0x01d036fa96dd367e
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: a174fc6f-a35d-11e4-ad94-001c23b85f7b