While I cannot comment on your issue, we migrated away from using SM as the outgoing gateway, we use ZoneMTA now on Linux as our outgoing gateway and it works brilliantly.

I'm trying to just get a SM16 to SM16 relay to work with no luck. Was also testing SM17 before purchase and can see if I can get an SM16 to SM17 to work but guessing no on that too.

I can't confirm for your scenario but I was able to get SM16 and SM17 to do this. I'm testing this today and this is how I have it working.

The SM16 is the primary mail server.

The SM17 box is the free, single domain version with a dummy domain setup for inbound filtering.

Both boxes have SSL/TLS enabled and bound to ports 25, 465, 587 and 993. I have MX records pointing to the SM16 box as primary, and the SM17 box as secondary. Both boxes have the LOCAL IP address of the other box in security IP whitelist settings for SMTP bypass. (Note I have both boxes connected via VPN. If connecting the two across public IP then use their public IP addresses instead)

For the SM16 primary mailserver in Gateway settings I just added an external gateway with the IP of the SM17 box on port 465, enable encryption, priority all. Do not enable authentication or turn on gateway mode.

For the SM17 box I added an internal gateway using domain forward mode. Entered the IP of the primary mailserver, status enabled. I'm using all (web service) for domain verification. Enable gateway mode and enter the full URL of the primary mailserver. Enter the user name and password of a Smartermail Admin. Not a domain admin and turn on web service user verification.