There's no way to rate limit to specific domains that I know of. The problems you have are common... especially with AOL and Yahoo. I rarely have any problems with gmail. I've got a mail server being rate limited by outlook / hotmail right now (not SmarterMail, but an Exim server) and there's no logic behind it... luckily I can redirect messages from any server to another that isn't being limited.
I really think the big guys use rate limiting as a way to put pressure on the smaller guys. They'll never rate limit each other, just the smaller volume servers (that probably have a much smaller spam percentage than the big guys put out).
If you ever find an answer to the problem let all of us know.
-Joe
Thanks,
-Joe