SmarterStats: System throws NullReferenceException
Question asked by Fredrik Schweinebarth - November 12, 2013 at 4:14 AM
Answered

Since I've started reading zipped files in combination with unzipped files, I have started to receive the following errors (I usually get hundreds of these, followed by the next log row)

[2013-11-10 06:18:06] (Gather_For_49)
   ===============================================================
   System.NullReferenceException: Object reference not set to an instance of an object.
   at SStatSvc.LogFiles.Smarter.SmarterLogFile.Write(LogFileEntry entryIn)
   at SStatSvc.Control.TaskThreadGather.DoWork()
   ===============================================================
   (1165 Used/2395 Free)
[2013-11-10 06:18:13] (Gather_For_49) site.com - Time = 245s  Read = 384938  Skipped = 1  Records per minute = 94320 (1168 Used/2319 Free)

 

And then we also seem to get (this started when some sites started using zipped format):

2013-11-10 10:29:18] (Gather_For_45)
   ===============================================================
   System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Number.FormatInt64(Int64 value, String format, NumberFormatInfo info)
   at System.Int64.ToString(String format, IFormatProvider provider)
   at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)
   at System.String.Format(IFormatProvider provider, String format, Object[] args)
   at SStatSvc.LogFiles.Smarter.Hit.get_SessionIDString()
   at SStatSvc.LogFiles.Smarter.SessionFileWriter.Process(Hit theHit, String uriStem)
   at SStatSvc.LogFiles.Smarter.SmarterLogFile.Write(LogFileEntry entryIn)
   at SStatSvc.Control.TaskThreadGather.DoWork()
   ===============================================================
   (1552 Used/1963 Free)
[2013-11-10 10:29:20] (Gather_For_55)
   ===============================================================
   System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at ICSharpCode.SharpZipLib.Zip.Compression.Inflater..ctor(Boolean noHeader)
   at ICSharpCode.SharpZipLib.Zip.ZipFile.GetInputStream(Int64 entryIndex)
   at ICSharpCode.SharpZipLib.Zip.ZipFile.GetInputStream(ZipEntry entry)
   at SStatSvc.Stats.ActivityStatsItem.Load(DateTime Day)
   at SStatSvc.Stats.Spiders.StatsDataCollector_Spider.#9V()
   at SStatSvc.Stats.Spiders.StatsDataCollector_Spider.set_Day(DateTime )
   at SStatSvc.Stats.Spiders.StatsDataCollector_Spider.Process(LogFileEntry entry, Hit theHit, SessionItemRecord theSession, SiteSettings details, Boolean shouldExclude)
   at SStatSvc.LogFiles.Smarter.SmarterLogFile.Write(LogFileEntry entryIn)
   at SStatSvc.Control.TaskThreadGather.DoWork()
   ===============================================================
   (1552 Used/1963 Free)
[2013-11-10 10:29:23] (Gather_For_55)
   ===============================================================
   System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder..ctor(String value, Int32 startIndex, Int32 length, Int32 capacity)
   at System.String.Format(IFormatProvider provider, String format, Object[] args)
   at SStatSvc.LogFiles.Smarter.Hit.get_SessionIDString()
   at SStatSvc.LogFiles.Smarter.SessionFileWriter.Process(Hit theHit, String uriStem)
   at SStatSvc.LogFiles.Smarter.SmarterLogFile.Write(LogFileEntry entryIn)
   at SStatSvc.Control.TaskThreadGather.DoWork()
   ===============================================================
   (1552 Used/1963 Free)

 

Eventually the service crashes and need a manual restart. We didn't have an issue before, with the same data volumes. I have increased the memory for this server from 2GB to 4 GB but this is a 32-bit instance so I can't give it more memory without reinstalling. Is this expected? Is it really unable to handle zipped formats without a 64-bit installation? Is SmarterStats even able to run in 64-bit mode?

1 Reply

Reply to Thread
1
Grady Werner Replied
Employee Post Marked As Answer

Several questions here, so I'll do my best to address them all.

First, SmarterStats does in fact run under 64 bit and it is actually our recommendation to do so.  32-bit processes in Microsoft.Net are limited to 1 GB of ram TOTAL and can not exceed it, even for a moment.  64 bit removes that limit, so the process could get temporary memory while it's trying to import logs with lots of complicated data.

OutOfMemoryExceptions can happen because of legitimate out-of-memory errors, but also because of file corruption in your original zip files or in your site data files.  Microsoft.Net is interesting, in that they use OutOfMemory for some cases where it's not really an out of memory problem.

In your case, however, it does look like an out of memory problem.  You can confirm by looking at process monitor while it's running.  I doubt that it's due to your original logs being zipped (that adds very little overhead to the import, memory wise), but rather you just hit a critical mass of data.

Aside from changing your server setup, make sure that you keep up to date with stats.  In recent builds we've made some pretty hefty enhancements to its memory usage.

If these tips don't help, you can always start a support ticket with our staff.  That will enable them to contact you about getting a copy of your SmarterLogs so they can run it through some simulations here to see what's making it take so much memory.

Hope this helps.

Grady Werner
Development Manager
SmarterTools Inc.
(877) 357-6278
www.smartertools.com

Reply to Thread