We faced similar issue couple of weeks back when our primary DNS server was unreachable. I think it's a bug and needs to be fixed asap. 

Hello and good afternoon. I tested this here but was not able to reproduce the issues you've reported. I disabled DNS caching completely, then set my primary DNS server address to an unroutable address and sent a test message. I see in the logs that SM was still able to find a valid MX record for the destination domain. Can you confirm your secondary DNS server is reachable from the SM server?

Kyle,

Yes, both are reachable from the server and SM, however I see what occurred better now (and had it a little backwards in my OP):

So, it does appear that SM will roll over from Primary DNS IP to Secondary DNS IP successfully. However, if the Secondary DNS IP then becomes unreachable it does not appear to retry the Primary DNS IP. 

I'm guessing there is a timed mechanism to determine when it will retry a DNS IP after it has previously failed (since it doesn't seem to do it on a per lookup basis)? Knowing what the retry interval is could allow us to stagger DNS reboots accordingly and avoid having this happen in the future.

Interesting findings here, thanks! I'm betting you're right about the retry period being expired. I'll work on confirming this here and will then get it reported to development. 

I simulated (a couple of different ways) having both my primary and secondary DNS servers go down, but found SmarterMail begins reaching out to the previously down DNS servers on the next send attempt:

[2019.10.14] 16:44:00.990 [A]:hostedsmartermail.com | Time: 11003, Exception: No connection could be established to any of the following name servers: 8.8.8.8:53 (Udp: 512), 8.8.4.4:53 (Udp: 512).
[2019.10.14] 16:44:11.996 [MX]:hostedsmartermail.com | Time: 11004, Exception: No connection could be established to any of the following name servers: 8.8.8.8:53 (Udp: 512), 8.8.4.4:53 (Udp: 512).
[2019.10.14] 16:45:25.516 [MX]:hostedsmartermail.com | Time: 11003, Exception: No connection could be established to any of the following name servers: 8.8.8.8:53 (Udp: 512), 8.8.4.4:53 (Udp: 512).
[2019.10.14] 16:45:25.551 [A]:hostedsmartermail.com | Time: 35, Result: 70.167.205.116
[2019.10.14] 16:45:31.089 [MX]:hostedsmartermail.com | Time: 5536, Result: mail.hostedsmartermail.com
[2019.10.14] 16:45:50.578 [MX]:hostedsmartermail.com | Time: 0, Result: mail.hostedsmartermail.com
[2019.10.14] 16:45:50.579 [A]:hostedsmartermail.com | Time: 0, Result: 70.167.205.116

What version are you running in this environment? I see we did a DNSClient update in b7040, and modified DNS lookups again in 7082 to prevent the settings.json being opened/saved repeatedly, perhaps one of these changed how this responds? Let me know what you think when you get a moment.