The recent post about blocking Japanese-language spam has forced me to consider how to know if a message is in an acceptable language. It makes my brain hurt.
The layers of complexity:
- A spoken language is written down using a script. One script may be used for several languages, and one language may use several scripts. Indonesian and Swahili languages only need the script of the English alphabet. Japanese uses katakana, harkara, and kanji scripts.
- A script is encoded and computerized as a character set.
- The character set may be encoded again for transport as an email message.
- The email message should be labelled so that the transport encoding can be removed, and then the bytes can be decoded into a visual script for display to the user.
- A human reader interprets the script in the context of the language to determine meaning. A computerized natural language processor may take the place of the human reader.
Tagging
- Content-language header fields indicate the language
- Content-type header fields with char-set element indicate a character set.
- Content-type header fields with char-set="utf-8" indicate Unicode text encoded using utf-8
- Content-transfer-encoding: base-64 indicates that the content was encoded for transport.
So the analysis needs to figure out:
- If the text is base-64 encoded for transport, the first step is to reverse that encoding.
- If the decoded text is 7-bit only, and therefore using English script (and probably ASCII encoding, since I have never received an email in EBCDIC.) I regularly receive email that is actually 7-bit ASCII but tagged with a non-English character set. When this occurs, the character set tag needs to be ignored.
- If the decoded text uses 8 bits and is not tagged as utf-8, it is essential to know how to interpret the 8th-bit characters, so a content-type tag is essential.
- If the text decoded text uses 8 bits and is tagged as utf-8, then the utf-8 characters need to be converted to Unicode, then the Unicode code points need to be evaluated to detect the languages and other character classes to determine the script used.
- The script needs to be evaluated by a natural language processor to determine the spoken language used to create this script.
During filtering, an appropriate goal is to determine whether the received document is in an acceptable language. In many cases, the issue can be simplified by filtering on acceptable character sets (for non-Unicode) and acceptable script classes for Unicode. Better regex tools can filter Unicode based on defined script classes.
A first-level filter determines which languages are acceptable at the server level. A second level filter determines which languages are acceptable for a particular recipient. If the language is unacceptable at either level, then delivery stops. The filtering system then needs to decide whether to silently discard the message or notify the sender.
On top of all of this, content filters are language-specific. If you want to block messages advertising "Russian brides", you need to consider how you will block it in each supported language.
I have the luxury of running a US-based server with almost exclusively US-English correspondents, so this has not been forced to the top. I am curious what tools are used by administrators who must cope with multiple languages from legitimate senders.