Hopefully this explains things a bit. Please understand that it isn't "broken", we just have 2 rules that are combining that may be causing issues...
Basically, you have 2 options:
Option 1: Change values
A simple solution is to simply change the default values for By IP and By Email. One way would be to decrease the number of attempts By IP to be LOWER than the By Email number. This will block the IP before the email address hits its own rule limit. You could also decrease the By IP limit and increase the By Email limit to ensure that, if you're seeing a lot of brute force attempts from multiple IPs on the same account, the account still doesn't get locked out.
Option 2: Disable By Email
Keep the By IP rules in place and disable the By Email rules. This way, IPs will contiue to be blocked when there are brute force attempts, but the email account itself is never blocked. This is a good solution for cases where a particular address (or multiple addresses) are being brute forced by multiple IPs.
These 2 IDS rules were added and CAN work together. It's just a matter of configuring them TO work together.
As an aside, we're looking at the defaults we put for each. Right now I think they're set to the same number, but a future build will have the By IP number smaller than the By Email number by default. That way, if both are turned on there will be some difference between the two to try and eliminate confusing and issues.
I hope that helps.