This really doesn't directly address your issue but I suspect that this is just normal operation in concert with Windows 2008. Jeez, I just opened task manager on my desktop and my FireFox process, alone, is eating up 1.2 GB of memory with eight tabs open! Any production SM server is handling countless concurrent operations and connections all well below 1 GB of memory usage. Sounds pretty darn efficient in my book, all considering.
As you pointed out though, on my 2012 R2 server, memory usage hovers around 330 MB for mailservice.exe and another 250 MB for clamd.exe.