Move Messages from spool dashboard fails with several thousand messages
Problem reported by CCC - November 7, 2017 at 9:02 AM
Submitted
From the spool dashboard, I have successfully moved messages from the spool into another folder on disk - however when there are several thousand messages in the spool, the messages do not get moved.
 
Using Version 15.7.6508

4 Replies

Reply to Thread
1
Scarab Replied
Same thing happens on v16 whether you are moving or deleting messages when the Spool exceeds a certain threshold.
 
The Throttling & IDS tools in Smartermail make it really nice and convenient for blocking outgoing mail when an email account is compromised which protects your Mail Server's reputation, but once the spool exceeds 1K-2K messages it basically dies and stops responding to everything even the "Delete Messages" function in v15-16 which means no one is getting inbound or outbound email delivered. It is a quick way to ruin any and all plans for the day.
 
Since SmarterMail becomes entirely unresponsive during high Spool counts we have had to place a batch file script on our Server. We have a simplified one that we can set to an event that triggers when the Spool > 500 messages and just moves all the messages in the subspools to temp folders and re-adds them back to the spool at intervals to allow the SmarterMail Spool the opportunity to process large(r) loads. For compromised accounts we use the following batch script to compensate for SmarterMail's inability to handle large Spools (if no compromised account variable is declared it just runs the resetspool parts as our simplified script does) :
 
:: Clear Spool & Delete Spam Script
:: For Smartermail
::
:: Move E-Mail from the Spool folder to temp Spool folders
:: Delete emails with a string
:: And return it to the Spool in batches
::
:: Filename:: PurgeSpool.bat
::
:: Author:: By Mark Garrison
:: <admin@scarabmedia.com>
::
:: Created 03/13/20007
:: Last Updated 10/26/2017
:: 
:: Run Syntax::
:: "purgespool.bat address@domain.tld"
:: Wherein this Script::
:: 
:: %1 = email address to purge
 
:: SET Host Server Environment Variables
 
@SET PATH=d:\Smartermail\Spool
@SET TEMP=d:\Smartermail\Spool\temp
@SET INTERVAL=300
 
:: DO NOT MODIFY VARIABLES BELOW THIS LINE!
 
:: Allow for asking for usage syntax
@if "%1" == "?" goto usage
@if "%1" == "/?" goto usage
@if "%1" == "/help" goto usage
 
:: Create Temp Directories
:: If They Don't Exist
 
If NOT exist %TEMP%\SubSpool0\ mkdir %TEMP%\SubSpool0\
If NOT exist %TEMP%\SubSpool1\ mkdir %TEMP%\SubSpool1\
If NOT exist %TEMP%\SubSpool2\ mkdir %TEMP%\SubSpool2\
If NOT exist %TEMP%\SubSpool3\ mkdir %TEMP%\SubSpool3\
If NOT exist %TEMP%\SubSpool4\ mkdir %TEMP%\SubSpool4\
If NOT exist %TEMP%\SubSpool5\ mkdir %TEMP%\SubSpool5\
If NOT exist %TEMP%\SubSpool6\ mkdir %TEMP%\SubSpool6\
If NOT exist %TEMP%\SubSpool7\ mkdir %TEMP%\SubSpool7\
If NOT exist %TEMP%\SubSpool8\ mkdir %TEMP%\SubSpool8\
If NOT exist %TEMP%\SubSpool9\ mkdir %TEMP%\SubSpool9\
 
:: Verify that all variables have been declared
@if "%1" == "" goto resetspool
 
@echo
 
:: Stop the Smartermail Process in Windows
 
net stop MailService
 
:: Check to make sure %PATH%\SubSpool folder has contents
:: Then MOVE contents of \SubSpool subfolder to ..\temp\SubSpool.
 
if exist %PATH%\SubSpool0\*.* MOVE %PATH%\SubSpool0\*.* %TEMP%\SubSpool0\
if exist %PATH%\SubSpool1\*.* MOVE %PATH%\SubSpool1\*.* %TEMP%\SubSpool1\
if exist %PATH%\SubSpool2\*.* MOVE %PATH%\SubSpool2\*.* %TEMP%\SubSpool2\
if exist %PATH%\SubSpool3\*.* MOVE %PATH%\SubSpool3\*.* %TEMP%\SubSpool3\
if exist %PATH%\SubSpool4\*.* MOVE %PATH%\SubSpool4\*.* %TEMP%\SubSpool4\
if exist %PATH%\SubSpool5\*.* MOVE %PATH%\SubSpool5\*.* %TEMP%\SubSpool5\
if exist %PATH%\SubSpool6\*.* MOVE %PATH%\SubSpool6\*.* %TEMP%\SubSpool6\
if exist %PATH%\SubSpool7\*.* MOVE %PATH%\SubSpool7\*.* %TEMP%\SubSpool7\
if exist %PATH%\SubSpool8\*.* MOVE %PATH%\SubSpool8\*.* %TEMP%\SubSpool8\
if exist %PATH%\SubSpool9\*.* MOVE %PATH%\SubSpool9\*.* %TEMP%\SubSpool9\
 
:: Restart the SmarterMail Process in Windows
 
net start MailService
 
:: Delete all .eml & .hdr files with email address
:: Move sanitized contents back to Spool
 
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool0\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool0\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool0\*.* %PATH%\SubSpool0\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool1\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool1\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool1\*.* %PATH%\SubSpool1\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool2\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool2\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool2\*.* %PATH%\SubSpool2\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool4\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool4\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool4\*.* %PATH%\SubSpool4\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool5\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool5\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool5\*.* %PATH%\SubSpool5\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool6\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool6\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool6\*.* %PATH%\SubSpool6\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool7\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool7\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool7\*.* %PATH%\SubSpool7\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool8\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool8\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool8\*.* %PATH%\SubSpool8\
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool9\*.eml') do (
@echo %%a
del %%a
)
for /F "delims=;" %%a in ('findstr /m /i "%1" %TEMP%\subspool9\*.hdr') do (
@echo %%a
del %%a
)
MOVE %TEMP%\SubSpool9\*.* %PATH%\SubSpool9\
CLS
@goto end
 
:resetspool
 
:: Stop the Smartermail Process in Windows
 
net stop MailService
 
:: Check to make sure %PATH%\SubSpool folder has contents
:: Then MOVE contents of \SubSpool subfolder to ..\temp\SubSpool.
 
if exist %PATH%\SubSpool0\*.* MOVE %PATH%\SubSpool0\*.* %TEMP%\SubSpool0\
if exist %PATH%\SubSpool1\*.* MOVE %PATH%\SubSpool1\*.* %TEMP%\SubSpool1\
if exist %PATH%\SubSpool2\*.* MOVE %PATH%\SubSpool2\*.* %TEMP%\SubSpool2\
if exist %PATH%\SubSpool3\*.* MOVE %PATH%\SubSpool3\*.* %TEMP%\SubSpool3\
if exist %PATH%\SubSpool4\*.* MOVE %PATH%\SubSpool4\*.* %TEMP%\SubSpool4\
if exist %PATH%\SubSpool5\*.* MOVE %PATH%\SubSpool5\*.* %TEMP%\SubSpool5\
if exist %PATH%\SubSpool6\*.* MOVE %PATH%\SubSpool6\*.* %TEMP%\SubSpool6\
if exist %PATH%\SubSpool7\*.* MOVE %PATH%\SubSpool7\*.* %TEMP%\SubSpool7\
if exist %PATH%\SubSpool8\*.* MOVE %PATH%\SubSpool8\*.* %TEMP%\SubSpool8\
if exist %PATH%\SubSpool9\*.* MOVE %PATH%\SubSpool9\*.* %TEMP%\SubSpool9\
 
:: Restart the SmarterMail Process in Windows
 
net start MailService
 
:: MOVE contents back to Spool one Subspool at a time
:: Wait Interval (default 5 min or 300 seconds) in between to allow processing
 
MOVE %TEMP%\SubSpool0\*.* %PATH%\SubSpool0\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool1\*.* %PATH%\SubSpool1\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool2\*.* %PATH%\SubSpool2\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool3\*.* %PATH%\SubSpool3\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool4\*.* %PATH%\SubSpool4\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool5\*.* %PATH%\SubSpool5\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool6\*.* %PATH%\SubSpool6\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool7\*.* %PATH%\SubSpool7\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool8\*.* %PATH%\SubSpool8\
PING 127.0.0.1 -n %INTERVAL%
MOVE %TEMP%\SubSpool9\*.* %PATH%\SubSpool9\
CLS
@goto end
 
:usage
@echo Clears Spool and purges messages from a spammer.
@echo var1=email address
@echo EXAMPLE purgespool spammer@domain.tld
@echo If no email address is specified it will return items to the Spool at regular intervals.
@goto end
 
:end
 
 
0
Matt Petty Replied
Employee Post
We've made some large improvements to CPU usage when the spool is under a high load. Have you noticed any difference in the past 2 weeks? This is system CPU performance, not browser.
Matt Petty
Software Developer
SmarterTools Inc.
(877) 357-6278
www.smartertools.com
0
Scarab Replied
Although we are running multiple Incoming SmarterMail Gateways that run Anti-Spam checks each on Dual Xeon X5550 Quad Cores to alleviate the load on our Primary SmarterMail Server we still have this happen pretty frequently on just Inbound traffic alone (about 8 times a year).

We are about 1 month behind on minor version updates to SmarterMail however, as we wait until our next Monthly Maintenance window to apply updates. We should be current with v16.3.6508 on the 16th of this month, just in time for the deluge of email for Black Friday junk mail which is one of those 8 times a year that the SmarterMail Spool historically tanks for us on all of our Incoming Gateways.

I guess we'll see then how things go and let you know Matt.
1
J. Dikkenberg Replied
Is this issue still around or fixed by updates?

Reply to Thread