I tried a lot of things today, disabling ClamSup updates, removing some signatures files, changing spool Delivery Delay, changing ClamAV Timeout in Antivirus Administration, with several stop/start of the SmarterMail service (waiting for clamd32 to exit), but the failed connection kept rising, and not a single new connection.
Then I simply unchecked "Enable ClamAV" in Antivirus Administration, waited for the failed connection to stop increasing, and enabled ClamAV again. It started to work again...
I'll keep monitoring the connection/failed connection count and see if it stays ok.