Ok here is a bit of more info regarding this issue, I began to try and identify a possible cause to this issue, we run declude, which stores its held spam emails in the spool folder under a spam directory, after searching for possible posts, one of the SM guys was indicating that quarantined mails in the spool queue could potentially slow a server dowm, so I moved the declude folder out into the smartermail folder, and i deleted all quarantined emails. and moved all mails out the spool folder.
The mailservice memory usage droped down to 300Mb, I slowly added mails that I removed from the spool, and started with small mails, 1k - 1M files, memory usage grew to around 600Mb and returned back to normal, I then began dropping 1Mb to 4Mb mails into the queue, the memory usage spiked up to around 800mb when I bombed around 20 of these mails in and again the memory dropped back down to 300Mb, I got to mails of 7Mb and dropped 4 of these into the spool, BOOM 2.5Gb of memory usage. The memory dropped back down to stop at 500Mb and has stayed there. The processor usage goes to about 40% and hangs around there constantly.
I then began testing as a client had dropped around 50 of these mails out to clients, 1 mail of 7mb = 675Mb usage - 170Mb usage to process a single file ??? 5 Files spiked up to 2.2Gb usage, 13 files = 3.3Gb usage peaking to 4Gb, the weird part was after the spool delay, and in the queue, but not being delivered yet, the usage stayed at around 3Gb, and as they went into the waiting to deliver queue the memory dropped back down to 500Mb.
I then tried a 20Mb file usage peaked up to 900Mb, 3x 22Mb files peaked at 2.6Gb, 5x 22Mb peaked at just over 4Gb.
I then decided to try looking elsewhere, and took a look at our outgoing spam filtering, I turned off custom rules filtering and remote spam assassin and BINGO after dropping a few 38Gb files in the queue the memory stayed at 450Mb. I re-enabled custom rules and the problem was back, disabled it enabled remote spam assassin and the memory was low. For good measure I tried a few times and it is definately the issue.
We have a few custom rules we run, but we receive a fair amount of mail per day, around 22k mails a day per server, but on incoming spam filtering the custom rules do nothing like this, the memory stays low.
So it looks like custom rules is causing issues on outgoing smtp filtering, I checked this on both my 15.3 servers and one 12.5 server same issue and same solution.
I now am busy looking at the rules to see if there is a specific rule that causes this issue, I will report back later.