The latency issue is tied to PROXY protocol support (we only support v1 atm), that functionality has to be enabled per-server now. This allows you to add the specific proxy servers you intend on routing to SM via the security whitelist. This allows you to put haproxy (or other tcp reverse proxies) in front of SmarterMail and we'll handle the IP tracking correctly. It's also a security measure as we'll only trust PROXY protocol from those servers and since they are always sending the proxy header you should not notice any latency with them. 

(That latency is actually built into the spec for PROXY support, infact they suggest waiting up to 3 seconds but we only wait for 1 second at most)