1
CloudFlare Log Processing Exception
Idea shared by Shaun Peet - November 16, 2014 at 5:06 PM
Proposed
This has been discussed briefly before on the old forums:
 
http://forums.smartertools.com/threads/cloudflare-integration.29148/
http://forums.smartertools.com/threads/site-not-reading-advanced-iis-log-files.39154/
 
In that second link there was mention that ST was thinking of implementing some special processing rules for logs containing CloudFlare's custom headers.  I'm hoping that this can happen sooner than later.
 
The current implementations involve installing modules into IIS to "correct" the logs on an individual site-by-site basis.  While that is feasible for some, for folks like us who host hundreds of sites which all go through CloudFlare it creates multiple points of failure and quite a bit of extra admin work.
 
I know that CloudFlare is not the only DNS proxy service out there so this is potentially opening a bit of a Pandora's box.  However, there is a relatively obscene amount of traffic flowing through their networks these days and chances are that quite a few of those host servers are IIS with logs being processed by SmarterStats.
 
If not doing something specific for CloudFlare's logs (because it opens the door to have to do custom provider-specific coding for other proxies), then perhaps there is a way to include a global configuration option which the end user (ie me) could set which defines custom header mapping when processing the logs.
 
In our case it's really just the one field - the IP address of the visitor - which we are concerned about.  If we could specify that the "CF-Connecting-IP" maps to the IP address of the visitor that would (I think) solve all our problems we get.  These problems are the inability to determine what a "unique" and a "new" visitor is.
 
The log processing code would hopefully be smart enough to check for the custom header first, and if it doesn't exist, revert back to the standard one.
 
And it would probably make things more complicated, but it's possible that some people might want a feature like this to be configurable on a site-by-site basis.  Although I'm not sure how realistic it is that a company who hosts many customer sites would run them through separate proxy services similar to CloudFlare.  We certainly don't since that would be too much to manage.
 
If there are other header fields that cause issues with other SmarterStats customers (or other proxies) then hopefully they chime in here.
 
Thanks for your time - looking forward to the replies.
 

Reply to Thread