Lots of possibilities.  Obviously not a software bug or many would be complaining.

Start at the switch level.  Look for mismatch between fixed speed on one side and auto negotiate on the other..  This will cause half duplex mode with many retransmits.

Then look at antivirus, then CPU load 

I've seen the DNS based checks that occur during SMTP can also slow down the accepting of the email when sources are timing out or DNS performance isn't great. Moving some of these SMTP spam checks to the spool can help with this, along with finding out why DNS is not performing well if it was the issue.