As far as SmarterMail's behavior, this is expected. Bounce messages are supposed to be generated by the last server to accept the message.
When a sending server issues a RCPT TO command during an SMTP session, the receiving server is permitted to check if it can accept mail for the specified recipient. If it cannot, then the receiving server should return an error message. In the case of a recipient who does not exist, that error would probably be something like "550 No such user". That error response indicates that the server does not accept the message for that recipient, and it then becomes the responsibility of the sending server to generate the bounce message.
I can think of three likely reasons for why you are not getting bounce messages in your test. The first is that Gmail is not requesting bounce messages. Bounce messages are failure type Delivery Status Notifications (DSNs) and must be requested during the SMTP session. This leads to the second possibility, that a server in the relay chain does not support DSNs, since support is an optional extension of the SMTP protocol. If some server between the originating server and the destination server does not support DSNs, then a DSN request will get lost. Finally, some servers that officially support DSNs may be configured to not generate failure messages specifically.
As for your final question, since it is not SmarterMail's responsibility to generate the bounce message in your scenario, there is not really any configuration change you can make to ensure bounce messages are generated.
Andrew Barker
Lead Software Developer
SmarterTools Inc.
www.smartertools.com