When I started writing my own filtering rules, I had to decide how to search for reputation data after it gets collected.
The first principle was that, by default, a rule applies to all subdomains. So a rule to block “BadGuys.ru” actually means:
- Block if search string equals BadGuys.ru
- Block if search string ends with “.BadGuys.ru”
- Block if search string ends with “@BadGuys.ru”
The second principle was “longest match”, because every rule has exceptions. For example, one can imagine a configuration with three rules:
- Quarantine for match on top-level domain “ru”
- Block if search string matches “BadGuys.ru”
- Allow if search string matches “Kaspersky.ru”
This means that the rule database needs a match string and a disposition code.
The third principle was that Allow rules must be based on an authenticated identifier, to prevent impersonation. So the previous rule set gets revised to longest match based on:
- Quarantine for match on top-level domain “ru”
- Block if search string matches “BadGuys.ru”
- Allow if search string matches “Kaspersky.ru” and search string is authenticated.
Now, assume that I have a search string of "user@bounce.email.somedomain.ru". I need to consider these match possibilities:
- Match on full address user@bounce.email.somedomain.ru
- Match on domain name: “bounce.email.somedomain.ru"
- Match on parent domain “email.somdomain.ru”
- Match on grandparent domain: “somedomain.ru”
- Match on top-level domain: “ru”
That leave me with 5 match strings, each of which has one exact match and two ends-with matches, so 15 possible comparisons. I expected to develop a large list of rules over time. This means that the rule list cannot be searched sequentially, or performance will steadily degrade. So the datastore must be indexed, but indexing on "ends with" does not work very well.
Below is my solution: I tend to believe that smarter people can come up with a better one, so consider the competition open. If there is no better solution, perhaps the design will be useful to someone else.
Current solution:
- The rules are stored in a SQL table, with fields for match string, disposition, and whether authentication is required.
- The search string is parsed at delimiters, and each possible match is loaded into a SQL temporary table.
- The temporary table is then joined to the rules table, with a “top 1” qualifier so that the longest matching string is returned and the authentication-required parameter is observed.