There isn't a setting, at least that I'm aware of, but there definitely needs to be. We also have had occasions where an RBL is entirely unresponsive, or when we have exceeded hourly thresholds allowed by a RBL where rather than giving an error response that  RBL just doesn't respond, and in both cases SM simply doesn't timeout causing thousands upon thousands of messages to just pile up in the Spool waiting to be processed.

When this happens we just have to find the RBL causing it, disable it, and restart the Spool. There really should be a configurable timeout setting for antispam checks.