Greylisting generating excessive disk activity
Problem reported by David Sovereen - 2/10/2020 at 9:21 PM
Submitted
When greylisting is enabled, SmarterMail outputs the greylist.sbin file every 15 seconds.  We have what I would guess is a moderately busy server.  Over the past 7 days, it has recorded 253 GB of data transfer on the Reports -> Protocol page.  I realize that Bandwidth Activity is not equal to Disk Activity, but there isn't a disk activity report I can use for comparison.  Aside from indexing and logging, most bandwidth activity probably equates to a similar amount of disk activity, so it's a reasonable comparison.

Our greylist.sbin file is 25MB in size (and still growing a few MB per week). Since SmarterMail writes this file every 15 seconds (4 times a minute), just enabling greylisting causes 100 MB a minute, 6 GB an hour, 144 GB a day, and 1,008 GB a week in disk activity.  If bandwidth activity approximates disk activity, just having greylisting enabled generates 4 times as more disk activity than all mail handling on the server itself!

Tim and Company love to talk about how disk efficient their product is, and regularly post Release Notes that start with EFFICIENCY: pointing out how much they've improved disk, CPU, or memory efficiency.  I have brought this gross inefficiency to support's attention and they tell me it isn't a problem.  I'm annoyed because for no good reason, this excessive disk activity will wear out my brand new server's SSD disks 4 times sooner than if they improved the efficiency of greylisting, perhaps by writing the file once an hour or once a day.

SmarterTools: Please make the frequency with which the greylist.sbin file is written Administrator-configurable so that those of us who care about disk efficiency and disk life can set this to an efficient setting.

Thanks,

Dave

11 Replies

Reply to Thread
0
echoDreamz Replied
Strange, we use greylisting on the latest beta and haven’t seen anything unusual with disk IO. In the past we’ve seen issues with it, but that I believe was back in the v16 days. 
0
David Sovereen Replied
We aren’t having problems with greylisting writing to disk. Through a different greylisting issue (support ticket/bug still exists in latest non-beta release), we discovered that greylisting is massively disk inefficient and are annoyed/frustrated that SmarterTools won’t acknowledge and improve it. 
0
echoDreamz Replied
I’ve been monitoring our environment, the disk usage for the greylist file is almost nothing. I see more activity writing spool data. 
0
David Sovereen Replied
It should be just one file every15 seconds.  However, depending on the size of your greylist.sbin, that single file write likely amounts to much more written to disk than all of your spool activity during the previous 15 seconds. 
0
David Sovereen Replied
It should be just one file every15 seconds.  However, depending on the size of your greylist.sbin, that single file write likely amounts to much more written to disk than all of your spool activity during the previous 15 seconds. 
0
Matt Petty Replied
Employee Post
David have you tried copying your sbin file and greylist-filters.json somewhere else and deleting the ones in Settings? Just curious if you see better behavior with it smaller, maybe only do this for a little while to get an idea how it performs then swap your old greylist + filters in so delivery delays don't get too bad. Restarting SmarterMail in between moving files.

Also I don't doubt at all that Chris see's a lot of spool traffic, there is an impressive amount of I/O going on at any point in time on his machine. A lot of our optimizations and improvements have come from being able to work on environments like that.
@Chris how big is your greylist.sbin?
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
echoDreamz Replied
Our greylist size varies. Right now it is around 18MB. We’ve not had any issues with the greylist file like we used too. Where it grow crazy large and Sm would spin its wheels processing it. I do remember at one point long ago we did have to delete it. 
0
David Sovereen Replied
Matt, the greylist.sbin file is saved every 15 seconds regardless of size.

We deleted our greylist.sbin file on December 12.  It still gets written every 15 seconds.  And since then, it has grown to over 25 MB.  It would seem our server is probably busier than echoDreamz.

Still not sure what any of that matters.  The fact of the matter is that writing this file every 15 seconds is ridiculously inefficient and generates approximately 400% more disk activity than all of the mail handling operations of the server.

Dave
0
echoDreamz Replied
Oh I can almost guarantee ours is busier, There aren’t many SmarterMail instances that are as loaded as ours.  we don’t greylist everything though, we have rules on greylisting to prevent greylisting everything and delaying legit mail. Our greylist file used to hit in the 200MB or more range, but this was back in the 16 days. 

I’ve watched ours, it doesn’t updated every 15 seconds like yours does. I’m going to disable some of our greylisting rules and greylist everything and see what happens. 
0
David Sovereen Replied
We don't greylist all connections either, only about 15%.  We use the Greylist Threshold setting at Settings -> Antispam -> Options -> SMTP Blocking to decide when to greylist.

We are running build 7242.  Our server outputs the greylist.sbin file every 15 seconds *almost* like clockwork.  I say almost because I have seen rare instances, always in the middle of the night, where SmarterMail didn't output the file when expected.  I figured it was yet another SmarterMail bug, but not worth reporting.  Sounds like SmarterMail only outputs the file in 15 second intervals when there are changes.  Depending on how busy a server is, that might be every 15 seconds or might be less often.

Either way, outputting this file every 15 seconds on busy servers is inefficient.  The file could be outputted once every hour and on server shutdown and would be far more disk efficient.

Dave
0
Brian Lewis Replied
Build 7369 production beta Mailservice very busy with 36mb greylist

Reply to Thread