UTS #39: Unicode Security Mechanisms

Unicode Technical Report #36, "Unicode Security Considerations" [UTR36] provides guidelines for detecting and avoiding security problems connected with the use of Unicode. This document specifies mechanisms that are used in that document, and can be used elsewhere. Readers should be familiar with [UTR36] before continuing. See also the Unicode FAQ on Security Issues [FAQSec].

An implementation claiming conformance to this specification must do so in conformance to the following clauses:

UTS-39-C1. An implementation claiming to implement the General Profile for Identifiers shall do so by conforming to either UTS-39-C1-1 or UTS-39-C1-2.

UTS-39-C1-1. The Implementation shall be in accordance with the specifications in Section 3.1, General Security Profile for Identifiers, without change.

UTS-39-C1-2. The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.1, General Security Profile for Identifiers.

UTS-39-C1.1. An implementation claiming to implement the IDN Security Profiles for Identifiers shall do so by conforming to either UTS-39-C1.1-1 or UTS-39-C1.1-2.

UTS-39-C1.1-1. The implementation shall be in accordance with the specifications in Section 3.2, IDN Security Profiles for Identifiers for Identifiers, without change.

UTS-39-C1.1-2. The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.2, IDN Security Profiles for Identifiers.

UTS-39-C1.2. An implementation claiming to implement the Email Security Profiles for Identifiers shall do so by conforming to either UTS-39-C1.2-1 or UTS-39-C1.2-2.

UTS-39-C1.2-1. The implementation shall be in accordance with the specifications in Section 3.3, Email Security Profiles for Identifiers, without change.

UTS-39-C1.2-2. The implementation shall provide a precise list of characters that are added to or removed from the profile, but otherwise be in accordance with the specifications in Section 3.3, Email Security Profiles for Identifiers.

UTS-39-C2. An implementation claiming to implement any of the following confusable-detection functions for Identifiers defined in Section 4, Confusable Detection shall do so by conforming to either UTS-39-C2-1 or UTS-39-C2-2.

UTS-39-C2-1. The implementation of the function shall be in accordance with the specifications in Section 4, Confusable Detection, without change.

UTS-39-C2-2. The implementation shall provide a precise list of character mappings that are added to or removed from those provided, but otherwise be in accordance with the specifications in Section 4, Confusable Detection.

UTS-39-C3. An implementation claiming to detect mixed scripts shall do so by conforming to either UTS-39-C3-1 or UTS-39-C3-2.

UTS-39-C3-1. The implementation shall be in accordance with the specifications in Section 5.1, Mixed-script Detection, without change.

UTS-39-C3-2. The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.1, Mixed-Script Detection.

UTS-39-C4. An implementation claiming to detect Restriction-Levels shall do so by conforming to either UTS-39-C4-1 or UTS-39-C4-2.

UTS-39-C4-1. The implementation shall be in accordance with the specifications in Section 5.2, Restriction-Level Detection, without change.

UTS-39-C4-2. The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.2, Restriction-Level Detection.

UTS-39-C5. An implementation claiming to detect mixed numbers shall do so by conforming to either UTS-39-C5-1 or UTS-39-C5-2.

UTS-39-C5-1. The implementation shall be in accordance with the specifications in Section 5.3, Mixed-Number Detection, without change.

UTS-39-C5-2. The implementation shall provide a precise description of changes in behavior, but otherwise be in accordance with the specifications in Section 5.3, Mixed-Number Detection.

Identifiers ("IDs") are strings used in application contexts to refer to specific entities of certain significance in the given application. In a given application, an identifier will map to at most one specific entity. Many applications have security requirements related to identifiers. A common example is URLs referring to pages or other resources on the Internet: when a user wishes to access a resource, it is important that the user can be certain what resource they are interacting with. For example, they need to know that they are interacting with a particular financial service and not some other entity that is spoofing the intended service for malicious purposes. This illustrates a general security concern for identifiers: potential ambiguity of strings. While a machine has no difficulty distinguishing between any two different character sequences, it could be very difficult for humans to recognize and distinguish identifiers if an application did not limit which Unicode characters could be in identifiers. The focus of this specification is mitigation of such issues related to the security of identifiers.

Deliberately restricting the characters that can be used in identifiers is an important security technique. The exclusion of characters from identifiers does not affect the general use of those characters for other purposes, such as for general text in documents. Unicode Standard Annex #31, "Unicode Identifiers and Syntax" [UAX31] provides a recommended method of determining which strings should qualify as identifiers. The UAX #31 specification extends the common practice of defining identifiers in terms of letters and numbers to the Unicode repertoire.

That specification also permits other protocols to use that method as a base, and to define a profile that adds or removes characters. For example, identifiers for specific programming languages typically add some characters like "$", and remove others like "-" (because of the use as minus), while IDNA removes "_" (among others)—see Unicode Technical Standard #46, "Unicode IDNA Compatibility Processing" [UTS46], as well as [IDNA2003], and [IDNA2008].

This document provides for additional identifier profiles for environments where security is an issue. These are profiles of the extended identifiers based on properties and specifications of the Unicode Standard [Unicode], including:

The data files used in defining these profiles follow the UCD File Format, which has a semicolon-delimited list of data fields associated with given characters, with each field referenced by number. For more details, see [UCDFormat].

The files under [idmod] provide data for a profile of identifiers in environments where security is at issue. The files contain a set of characters recommended to be restricted from use. They also contain a small set of characters that are recommended as additions to the list of characters defined by the XID_Start and XID_Continue properties, because they may be used in identifiers in a broader context than programming identifiers.

The Restricted characters are characters not in common use, and they can be blocked to further reduce the possibilities for visual confusion. They include the following:

The choice of which characters to specify as Restricted starts conservatively, but allows additions in the future as requirements for characters are refined. For information on handling modifications over time, see Section 2.10.1, Backward Compatibility in Unicode Technical Report #36, "Unicode Security Considerations" [UTR36].

An implementation following the General Security Profile does not permit any characters in \p{Identifier_Status=Restricted}, unless it documents the additional characters that it does allow. Such documentation can specify characters via properties, such as \p{Identifier_Type=Technical}, or by explicit lists, or by combinations of these. Implementations may also specify that fewer characters are allowed than implied by \p{Identifier_Status=Allowed}; for example, they can allow only characters permitted by [IDNA2008].

Common candidates for such additions include characters for scripts listed in Table 7, Limited Use Scripts of [UAX31]. However, characters from these scripts have not been a priority for examination for confusables or to determine specialized, non-modern, or uncommon-use characters.

Canonical equivalence is applied when testing candidate identifiers for inclusion of Allowed characters. For example, suppose the candidate string is the sequence