We made a lot of performance/memory fixes in mail patches, so the first order of business should be upgrading to the latest mail 11 build. Mail 12 (in beta) will be released soon, as well, and it uses significantly fewer resources in many use cases. If you have upgrade protection current, keep an eye out for that.
What part of SmarterMail is chewing up memory, IIS or MailService.exe? That's step one for figuring things out. The other issues are likely resolved by moving up to 11.7. The version you're on is from 8 months ago, and a lot of things have been resolved since then. See http://www.smartertools.com/smartermail/releasenotes/v11.aspx for release notes.
Grady Werner
SmarterTools Inc.
www.smartertools.com