I have a SmarterMail v15 test server I maintain checkpoints of for this kind of testing, so I reverted it and set up a 768, 1024, and 2048 bit key on three separate domains before verifying they passed DKIM when sending emails. I then upgraded to our latest SmarterMail public release and saw no DKIM issues on signing or in validation on the remote end in my testing. Because we supported custom selectors in previous versions I also tested with and without this involved in the test but couldn't replicate the behaviors you're noting in those tests. I've reached out on a ticket to see if we can collect some diagnostic details from your environment and hope to get to the bottom of this soon!
Kyle Kerst
IT Coordinator
SmarterTools Inc.
www.smartertools.com