!standard 2.1(4.1/3) 17-04-24 AI12-0004-1/06 !standard 2.3(4/3) !standard A.3.2(4/3) !standard A.3.2(32.5/3) !standard A.3.5(18/3) !standard A.3.5(51/3) !class binding interpretation 15-06-13 !status Amendment 1-2012 16-11-11 !status WG9 Approved 17-06-16 !status ARG Approved 5-0-4 16-10-09 !status work item 11-11-08 !status received 11-06-20 !priority Low !difficulty Medium !subject Normalization and allowed characters for identifiers !summary Ada identifiers that contain characters that are not allowed in Normalization Form KC are illegal. Ada text not in Normalization Form C is implementation-defined. [Editor's note: This AI is formatted in UTF-8 rather than the typical ASCII because the character examples need extended characters.] !question (1) 2.1(4.1/2) says that meaning of program text not in Normalization Form KC is "implementation-defined". An Ada implementation could reject such text, even if it is only used in characters and/or string literals. But Normalization Form KC is very strong, including such things as superscript and subscript characters (so that writing m² rather than m**2 would not be allowed). The Unicode identifier recommendations included by reference in ISO/IEC 10646:2011 suggest that Normalization Form KC (NFKC) should be applied to identifiers (only). There is no reason to apply such a form to the remainder of Ada text, and especially to allow implementations to reject programs that contain characters not allowed by that form. Should this permission be restricted to identifiers? (Yes.) (2) The Unicode identifier recommendations included by reference in ISO/IEC 10646:2011 (this is Annex 31, the current link as of this writing is http://www.unicode.org/reports/tr31/tr31-25.html) suggests using the character classes XID_Start and XID_Continue in identifiers. The classes are defined to be stable, such that future character set standards (Unicode and 10646 now are kept fairly close together) will only make upwardly compatible changes. That is not true of character classes like "letter_uppercase", which is what Ada is currently defined in terms of. Should we change to use these classes? (No.) !recommendation (See summary.) !wording Modify 2.1(4.1/3): The semantics of an Ada program whose text is not in Normalization Form {C}[KC] (as defined by Clause 21 of ISO/IEC 10646:2011) is implementation defined. AARM Ramification: In particular, an implementation can reject such program source. Portable programs should always be written in Normalization Form C. AARM Reason: Normalization Form C ensures that all source is in a unique format; it eliminates ambiguities and security issues potentially caused by source using unusual sequences of characters. Note that WC3 recommends that all Internet content be in Normalization Form C. We don't require this as there is a potentially significant cost to checking, and some implementations may need to be interoperable with tools that produce unnormalized text. [Editor's note: I considered making this Implementation Advice, but that seemed to defy sensible wording without using Bob's hated word - "reject". And the above seemed to be enough anyway.] Add after 2.3(4/3): Legality Rules An identifier shall only contain characters that may be present in Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011). AARM Implementation Note: An implementation can usually detect this during lexical processing. The code points not allowed are those for which Unicode property NFKC_QC (Normalization Form KC Quick_Check) has the value No. We say "might be allowed" so that characters for which the value is Maybe (really, one of the possible values is Maybe) are allowed (these are mainly combining marks). The necessary tables can be found in http://www.unicode.org/Public/UCD/latest/ucd/DerivedNormalizationProps.txt. Versions for older Unicode versions can be found on this site as well; start at http://www.unicode.org/Public/ and find the appropriate version number. Add at the end of A.3.2(4/3): function Is_NFKC (Item : in Character) return Boolean; Add after A.3.2(32.5/3): Is_NFKC True if Item could be present in a string normalized to Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011); this includes all characters except those with positions 160, 168, 170, 175, 178, 179, 180, 181, 184, 185, 186, 188, 189, and 190. Add after A.3.5(18/3): function Is_NFKC (Item : in Wide_Character) return Boolean; Add after A.3.5(51/3): function Is_NFKC (Item : in Wide_Character) return Boolean; Returns True if the Wide_Character designated by Item could be present in a string normalized to Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011), otherwise returns False. AARM Reason: Wide_Characters for which this function returns False are not allowed in identifiers (see 2.3) even if they are categorized as letters or digits. AARM Implementation Note: This function returns False if the Unicode property NFKC Quick Check (NFKC_QC in the files) has the value No. See the Implementation Notes in 2.3 for the source of this property. AARM Discussion: A string for which Is_NFKC is true for every character may still not be in Normalization Form KC, as Is_NFKC returns true for characters that are dependent on characters around them as to whether they are removed by normalization. Ada does not provide a full normalization operation (it is complex and expensive). !discussion For the first question, it's clear that non-identifier text should not be subject to Normalization Form KC (NFKC). NFKC merges characters based on their basic properties, ignoring their visual appearance; normalization form C (NFC) just puts characters into their minimal canonical format. (Minimal here means that characters that can be written as a sequence or as a single character are written as the single character.) NFC never changes the visual appearance of a character; NKFC eliminate visual variants of a character such as subscripting, superscripting, positional variants, and more. Identifier text needs strong normalization so that it can be compared using simple case folding. Other text, which does not need to use case folding, should not be modified (or rejected) in that way. Indeed, both TR31 and TR15 are quite clear that NFKC should only be applied to limited text (like identifiers), not entire programs or documents. (TR15 says that Normalization Form KC "must not be blindly applied to arbitrary text"; TR31 mentions that string literals and comments should be handled separately from identifiers.) Also, we do not want the set of identifiers allowed by Ada to be dependent on the implementation, so we don't want the normalization of identifiers to be implementation-defined. We have to define the normalization used. We believe that the best solution is to define that all identifiers are normalized (that is, identifiers that are not normalized are illegal). The current situation is that Ada implementations might treat some characters differently in identifiers. (In particular, an implementation that normalized all incoming text into Normalization Form KC would convert m² [which is not a legal identifier] into m2 [which IS a legal identifier] before lexical analysis.) This could be a significant portability problem, one that is unnecessary. Thus we adopt a rule that identifiers need to be in Normalization Form KC. However, normalizing text is a very expensive and slow operation (it involves decomposing characters into their canonical parts (with a possible 3x expansion), ordering those parts, and then recomposing them). This is not an operation we want to use frequently [or implement!! - author]. Luckily, Unicode has a short-cut available, the "Quick_Check" properties (one set for each of the four normalization forms available). These are relatively large tables, but it provides a simple answer for each code point. It is simple to scan the text and determine the result. But as usual, here Unicode giveth and Unicode taketh-away. For Normalization Form D, the Quick_Check consists of Yes and No values; if a No is encountered, the string is not normalized and an error can be flagged. But for NFC and NFKC (that is, the useful forms that emphasize the most compact representation), Quick_Check is a three state value: Yes, No, and Maybe. In the Maybe case, an implementation is supposed to normalize the string and see if it changes. Since normalization is very expensive, we don't do that. We rather say "may be present in NFKC", thus allowing all of the Maybe characters. For Unicode 9.0 (current as of this writing), there are only 110 code points that are Maybe in the NFC_QC and NFKC_QC tables. (These are mostly combining code points.) All of the remaining code points are Yes or No. (No meaning of course that they can't appear in normalized text.) There are 1120 No code points for NFC_QC and a whopping 4792 for NFKC_QC (again, for Unicode 9.0). (Many of these code points aren't letters or numbers and can't appear in identifiers anyway.) (These tables can be found at http://www.unicode.org/Public/UCD/latest/ucd/DerivedNormalizationProps.txt; they're about in the middle of the file.) A detailed discussion of Unicode normalization can be found in Unicode Annex 15 (current link as of this writing is http://www.unicode.org/reports/tr15/). It's a lot more readable than the actual Unicode standard text [which I made the mistake of trying to understand - Author]. Additionally, it is directly referenced from Section 21 of 10646:2011, including it by reference. We do retain the rule that Ada source code needs to be normalized in order to be portable; we just use the much weaker Normalization Form C (NFC). WC3 recommends that all Internet content be normalized to NFC; that eliminates duplication and security issues that otherwise might arise. It seems sensible to recommend (but not require) the same of Ada source. === As noted in the second question, Annex U of ISO/IEC 10646:2011 directly references the Unicode Annex 31 document. It's fairly clear that this is the recommendation of the experts in this field, and we should come as close to those recommendations as possible. (But don't forget AI05-0227-1 about case folding and upper case conversions -- we should not undo those corrections.) We, however, don't change to using XId_Start and XID_Continue for several reasons. First, compabilitity is very important to Ada; using these classes would require a number of modifications in order to be compatible. (For instance, underscore is included in XId_Continue.) In addition, we would need to be careful not to undo the corrections made by AI05-0227-1 about case folding and upper case conversion. Secondly, it would require a lot of detailed analysis to figure out how the identifiers allowed by these classes differ from those allowed by Ada 2005 and Ada 2012. While identifiers with non-Latin-1 characters aren't very common, they do exist. This would also make this change less portable. Thirdly, the XId_Start and XId_Continue classes are very similar to the classes that Ada already uses for identifiers; they mainly are created from the same sets of characters that Ada already uses. Thus we have a situation where there is more risk of incompatibility than there is useful differences. Finally, Unicode has now made most properties "stable" unless there is a clear mistake (they claim to have only found 6 so far), so the worries about change to the existing classifications are reduced without making any changes to Ada (so long as most implementations are using Unicode versions newer than 5.2). If these classes had existed in Unicode 4.0, we surely would have used them, but the Unicode people were very confused about identifiers at that time (and we copied much of that confusion, sadly). Recent versions of TR31 straightened out that situation, but somewhat too late for Ada to use. It should be noted that we're actually not that far from the current TR31 recommendations. Our rules are essentially the same as "Filtered Case-Insensitive Identifiers" (that is, simple case folding and requiring normalization). Also, "Filtered Normalized Identifiers" seems to suggest allowing "Maybe" characters unconditionally, which is our solution to the otherwise very expensive implementation. (Unfortunately, TR31 doesn't give a suggested implementation for Filtered Case-Insensitive Identifiers using simple case folding, so we had to figure out one.) !corrigendum 2.1(4.1/3) @drepl The semantics of an Ada program whose text is not in Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011) is implementation defined. @dby The semantics of an Ada program whose text is not in Normalization Form C (as defined by Clause 21 of ISO/IEC 10646:2011) is implementation defined. !corrigendum 2.3(4/3) @dinsa An @fa shall not contain two consecutive characters in category @fa, or end with a character in that category. @dinst @s8<@i> An identifier shall only contain characters that may be present in Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011). !corrigendum A.3.2(4/3) @drepl @xcode< @b Is_Control (Item : @b Character) @b Boolean; @b Is_Graphic (Item : @b Character) @b Boolean; @b Is_Letter (Item : @b Character) @b Boolean; @b Is_Lower (Item : @b Character) @b Boolean; @b Is_Upper (Item : @b Character) @b Boolean; @b Is_Basic (Item : @b Character) @b Boolean; @b Is_Digit (Item : @b Character) @b Boolean; @b Is_Decimal_Digit (Item : @b Character) @b Boolean @b Is_Digit; @b Is_Hexadecimal_Digit (Item : @b Character) @b Boolean; @b Is_Alphanumeric (Item : @b Character) @b Boolean; @b Is_Special (Item : @b Character) @b Boolean; @b Is_Line_Terminator (Item : @b Character) @b Boolean; @b Is_Mark (Item : @b Character) @b Boolean; @b Is_Other_Format (Item : @b Character) @b Boolean; @b Is_Punctuation_Connector (Item : @b Character) @b Boolean; @b Is_Space (Item : @b Character) @b Boolean;> @dby @xcode< @b Is_Control (Item : @b Character) @b Boolean; @b Is_Graphic (Item : @b Character) @b Boolean; @b Is_Letter (Item : @b Character) @b Boolean; @b Is_Lower (Item : @b Character) @b Boolean; @b Is_Upper (Item : @b Character) @b Boolean; @b Is_Basic (Item : @b Character) @b Boolean; @b Is_Digit (Item : @b Character) @b Boolean; @b Is_Decimal_Digit (Item : @b Character) @b Boolean @b Is_Digit; @b Is_Hexadecimal_Digit (Item : @b Character) @b Boolean; @b Is_Alphanumeric (Item : @b Character) @b Boolean; @b Is_Special (Item : @b Character) @b Boolean; @b Is_Line_Terminator (Item : @b Character) @b Boolean; @b Is_Mark (Item : @b Character) @b Boolean; @b Is_Other_Format (Item : @b Character) @b Boolean; @b Is_Punctuation_Connector (Item : @b Character) @b Boolean; @b Is_Space (Item : @b Character) @b Boolean; @b Is_NFKC (Item : @b Character) @b Boolean;> !corrigendum A.3.2(32.5/3) @dinsa @xhang<@xterm True if Item is a character with position 32 (' ') or 160 (No_Break_Space).> @dinst @xhang<@xterm True if Item could be present in a string normalized to Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011); this includes all characters except those with positions 160, 168, 170, 175, 178, 179, 180, 181, 184, 185, 186, 188, 189, and 190.> !corrigendum A.3.5(18/3) @dinsa @xcode< @b Is_Space (Item : Wide_Character) @b Boolean;> @dinst @xcode< @b Is_NFKC (Item : Wide_Character) @b Boolean;> !corrigendum A.3.5(51/3) @dinsa @xindent, otherwise returns False.> @dinss @xcode<@b Is_NFKC (Item : Wide_Character) @b Boolean;> @xindent !ACATS test ACATS B-Test(s) should be written to check that characters in NFC but not in NFKC are not allowed in identifiers, even if they otherwise have the correct character classes. ACATS C-Test(s) should be written to test the Is_NKFC functions. !ASIS Probably no ASIS interface effect. !appendix !topic Inconsistency in text normalization form and allowed characters !reference Ada 2012 RM (Draft 12) 2.1(4.1/2), 2.2(3/2), 2.3(8.c/2), A.3.2(60/3), A.4.6(8/3), AI05-0114-1 !from Howard W. Ludwig 11-06-20 !discussion 2.1(4.1/2) states The semantics of an Ada program whose text is not in Normalization Form KC (as defined by section 24 of ISO/IEC 10646:2003) is implementation defined. This adheres to recommendations for communicating case-insensitive data across a network in a manner minimizing security issues. However, it raises two difficulties (the first of which has shades of leap second): 1. AI-0114-1 indicates a ruling that characters such as the feminine ordinal indicator (Unicode code point u+00AA), the micro sign (u+00B5), and the masculine ordinal indicator (u+00BA) are to be regarded as letters (because Unicode places them in category Ll, lower-case letters) that can be used in identifiers, but are not to be treated as letters by Ada.Characters.Handling.Is_Letter. However, Normalization Form KC does not allow the use of those characters (with ª being replaced by a [u+0061], µ replaced by μ [u+03BC], and º replaced by o [u+006F]). If a programmer wishes to avoid implementation-defined behavior, then the use of these (and some other) characters in identifiers is forbidden. In any case, it would seem that an Ada compiler, due to Ada’s case insensitivity, must at the very least treat the identifier ª as identical to the identifiers A and a. In addition, the AARM 2.3(8.c/2) indicates, in contradiction to the NFKC expectation: An identifier can use any letter defined by ISO-10646:2003, along with several other categories. 2. Normalization Form KC forbids the use of superscripted and subscripted digits, for example. While I understand motivation for this restriction as applied to identifiers, I really would like to be able to use the full Unicode character set in comments. For example, if I declare a variable Area and I would like to indicate in commentary that its units are square meters, I would like to be able to write it as: Area: Float := 0.0; -- m² This is illegal in NFKC. A tool that converts an arbitrary text file to NFKC would convert this to: Area: Float := 0.0; -- m2 unsuperscripting the 2 so that it is no longer obvious that squaring is intended to be indicated. I would like to see comments being explicitly exempted from the Normalization Form KC expectation indicated by 2.1(4.1/2). Given all these issues, it might be more straightforward to use Normalization Form C instead of KC in 2.1(4.1/2). **************************************************************** From: Randy Brukardt Sent: Friday, July 1, 2011 7:39 PM >2.1(4.1/2) states The semantics of an Ada program whose text is not in >Normalization Form KC (as defined by section 24 of ISO/IEC 10646:2003) >is implementation defined. >This adheres to recommendations for communicating case-insensitive data >across a network in a manner minimizing security issues. My recollection is that this was inserted in order to sort-of follow a Unicode recommendation. We originally required the compiler to convert the source to Normalization Form KC before processing. However, there was great concern that this conversion could make identifiers that look very different (and likely compare differently in editors) be treated as the same. Of course, the reverse also could be true. So this rule was added, with the intent that it would leave it up to implementors as to whether or not the normalization was performed. [I should point out that those Unicode recommendations were completely rewritten between Unicode 4 (which is what Ada 2005 uses) and Unicode 5 -- completely reversing the recommendations in some areas. So I don't know what is recommended now on this specific topic.] >...However, it >raises two difficulties (the first of which has shades of leap second): >1. > >AI-0114-1 indicates a ruling that characters such as the feminine >ordinal indicator (Unicode code point u+00AA), the micro sign (u+00B5), >and the masculine ordinal indicator (u+00BA) are to be regarded as >letters (because Unicode places them in category Ll, lower-case >letters) that can be used in identifiers, but are not to be treated as letters by >Ada.Characters.Handling.Is_Letter. >However, Normalization Form KC does not allow the use of those >characters (with ª being replaced by a [u+0061], µ replaced by µ [u+03BC], and º replaced by o [u+006F]). >If a programmer wishes to avoid implementation-defined behavior, then >the use of these (and some other) characters in identifiers is forbidden. I agree with this. >In any case, it would seem that an Ada compiler, due to Ada’s case >insensitivity, must at the very least treat the identifier ª as identical to >the identifiers A and a. Now I've lost you. If the compiler implements the normalization, then it will never see ª in an identifier. Otherwise, the letter (if not normalized) is clearly *not* the same as A (based on the case folding tables). The net effect is that a compiler could do it either way and be within the requirements of the language. After all, "implementation-defined" is just that; the compiler could set off a fireworks show (this being the July 4th weekend) if it wanted to when the source text is not in Normalization Form KC; pretty much anything goes so long as it is documented. We probably would have preferred to restrict the outcomes but that isn't really an option given the framework of the Ada Standard. >In addition, the AARM 2.3(8.c/2) indicates, in contradiction to the NFKC expectation: >An identifier can use any letter defined by ISO-10646:2003, along with several >other categories. The AARM, of course, is not normative, and it is discussing the language ignoring NFKC issues. I'm actually surprised to hear that NFKC eliminates some characters completely, as that seems like a bad thing. In any case, if the compiler does not normalize the source input before processing, then the AARM note is strictly true. (I believe that all existing Ada 2005 compilers don't normalize.) >2. Normalization Form KC forbids the use of superscripted and >subscripted digits, for example. While I understand motivation for this >restriction as applied to identifiers, I really would like to be able >to use the full Unicode character set in comments. For example, if I >declare a variable Area and I would like to indicate in commentary that its >units are square meters, I would like to be able to write it as: > > Area: Float := 0.0; -- m² > >This is illegal in NFKC. A tool that converts an arbitrary text file to >NFKC would convert this to: > > Area: Float := 0.0; -- m2 > > unsuperscripting the 2 so that it is no longer obvious that squaring >is intended to be indicated. I would like to see comments being >explicitly exempted from the Normalization Form KC expectation indicated by 2.1(4.1/2). The intent was that a compiler could normalize all of the source, or leave the source in its original form. In either case, comments would have the same effect. It would greatly complicate the implementation to turn normalization on and off in different places, and it would violate the Unicode recommendation that all source code be intepreted in NFKC. Unless the implementation does something silly (like the fireworks-generating compiler noted earlier), the effect of comments would be the same either way. So I don't think there is any need to avoid any characters in comments. >Given all these issues, it might be more straightforward to use Normalization Form C instead of KC in 2.1(4.1/2). If there actually were any issues, it probably would be better to simply drop any permission for normalization, since it appears to be actively harmful (but it really isn't that harmful, as noted above). I find eliminating superscripts and subscripts to be completely changing the meaning of the text (as you note in your comments), that goes so far beyond normalization that it is simply insane. Moreover, if we are going to ignore the Unicode recommendation in this area (as we had to do for identifiers, see AI05-0227-1 for the sad details), we might as well go all the way and ignore normalization altogether. In any case, this is too late for Ada 2012; the earliest that the ARG could take up this issue would be at the November meeting, and the final draft of Ada 2012 should have been approved before that meeting. So this will be an early Ada 2012 AI and possible Binding Interpretation. **************************************************************** From: Howard W Ludwig Sent: Tuesday, July 5, 2011 2:26 PM ... >[I should point out that those Unicode recommendations were completely >rewritten between Unicode 4 (which is what >Ada 2005 uses) and Unicode >5 -- completely reversing the recommendations in some areas. So I don't >know what is >recommended now on this specific topic.] I understand that the rules for Unicode 5 and Ada 2012 changed from the previous version and why. The changes seem sensible, and it could get problematic for Unicode and Ada to diverge unnecessarily. NFKC is a Unicode recommendation for text interpreted in a case-insensitive manner (NFC for case-sensitive), so I can see how 2.1(4.1/2) was established, although it does have troubling consequences (mentioned below) accompanying certain advantages. ... >>In any case, it would seem that an Ada compiler, due to Ada’s case >>insensitivity, must at the very least treat the identifier ª as >>identical to the identifiers A and a. > >Now I've lost you. If the compiler implements the normalization, then it will >never see ª in an identifier. Otherwise, the letter (if not normalized) is >clearly *not* the same as A (based on the case folding tables). >The net effect is that a compiler could do it either way and be within the >requirements of the language. After all, "implementation-defined" is just >that; the compiler could set off a fireworks show (this being the July 4th >weekend) if it wanted to when the source text is not in Normalization Form KC; >pretty much anything goes so long as it is documented. We probably would have >preferred to restrict the outcomes but that isn't really an option given the >framework of the Ada Standard. My point was based on my interpretation of the implementation-dependence of source code written in other than NFKC [2.1(4.1/2)]. I regarded this as being my responsibility to write the code in NFKC so the compiler would behave in a ARM-defined and -compliant manner. If that is the case, then I must not use ª, µ [u+00B5], and º identifiers in my code, because they are not legal for NFKC. I see your point that the compiler could do the normalization for me (somewhat like a preprocessor); however, if that happens, the compiler would replace ª by a and º by o, effectively making the identifier ª identical to the identifiers A and a--this is what I was really making reference to in my original statement where you lost me. The whole point of normalization is to clarify when two characters are clearly intended to be the same but are written as text with different code points (for example, å can be written as the simple u+00E5 or as the compound u+0061 u+030A, for which NFKC requires the former and a conversion tool u+would recognize the latter and convert it to the former). The different normalization forms provide different prescriptions as to when two different sequences of code points are to be treated as the same character. NFC is very simple and would say, like NFKC, that å should be written as u+00E5 and not as u+0061 u+030A; NFD and NFKD make the opposite choice. However, NFKC goes way beyond NFC and makes many more sequences of code points identical, so that font effects like subscripting, superscripting, bolding, etc. are treated as insignificant to meaning and only the unadorned character version code point is to be accepted (ª gets converted to a, for example). I understand about "implementation-defined" allowing fireworks as long as it is documented. If I comply with NFKC in my code, I will not use those three characters myself, even though the ARM explicitly states they are legal in identifiers; if the compiler does the normalization for me, then it will eliminate those characters (even though explicitly allowed) and remap them as I stated before; otherwise, the compiler can do anything (treat ª as the same as a, treat ª as totally distinct from a, or generate fireworks). That is why I try to avoid implementation- dependence, so I don't have to worry about such variations. In this case, it means I need to write my code as NFKC to avoid one compiler generating fireworks while another compiler does what I had hoped it would do. That is why I am making somewhat of a big deal about this. ... >The intent was that a compiler could normalize all of the source, or leave >the source in its original form. In >either case, comments would have the same effect. >It would greatly complicate the implementation to turn >normalization on and off in different places, and it would violate the >Unicode recommendation that all source code be interpreted in NFKC. > >Unless the implementation does something silly (like the fireworks-generating compiler >noted earlier), the effect of comments would be the same either way. So I don't think there is any need to avoid any characters in comments. If the compiler does the normalization to NFKC and any generated listings are produced after the normalization, my superscripts would be undone in the listing. (My understanding is that Annex H, if supported by a compiler, requires a compiler to be able to generate listings, potentially with generated assembly language, to enable certain security reviews, and it is wide open how that is done, so post-normalization listings are quite plausible.) The other (and to me more important) issue is that, with the implementation-dependence for non-NFKC source code, the compiler might refuse to accept non-NFKC text (with or without accompanying fireworks), which would prevent me from using the superscripts if I want my code compiled, unless there are explicit ARM constraints forbidding rejecting code solely because comments are not written in NFKC. >>Given all these issues, it might be more straightforward to use Normalization Form C instead of KC in 2.1(4.1/2). > >If there actually were any issues, it probably would be better to simply drop >any permission for normalization, since it appears to be actively harmful (but >it really isn't that harmful, as noted above). I find eliminating superscripts >and subscripts to be completely changing the meaning of the text (as you note >in your comments), that goes so far beyond normalization that it is simply >insane. Moreover, if we are going to ignore the Unicode recommendation in this >area (as we had to do for identifiers, see AI05-0227-1 for the sad details), >we might as well go all the way and ignore normalization altogether. I agree with your insanity comment, but that is what NFKC and NFKD do. (NFC and NFD are far more sane in this regard.) However, as I pointed out earlier, the normalization is useful for letting all parties know how to indicate in a clear, unique, unambiguous manner a particular character that otherwise has multiple ways of being expressed, so it seems a little dangerous to throw out the whole thing. I know that WG9 in general and Robert Dewar have had fun dealing with case insensitivity in the context of the German ß versus SS and ss, as well as the Turkish lower-case dotless and dotted i both being written as I in upper case. Character composition and character compatibility in Unicode just extend these issues into a broader realm with more cans of worms (which is why I alluded to the leap second issues that proved difficult to close completely in a self-consistent manner). >In any case, this is too late for Ada 2012; the earliest that the ARG could >take up this issue would be at the November meeting, and the final draft of >Ada 2012 should have been approved before that meeting. So this will be an early >Ada 2012 AI and possible Binding Interpretation. I understand that progress has to be made, and at some point you have to say you are done for a given phase or stage, and you move on. I do not wish to cause a delay in releasing Ada 2012. I still appreciate your consideration at some point, as well as WG9's desire to have Ada becoming an ever better language without breaking what people may already be using. **************************************************************** From: Florian Schanda Sent: Tuesday, June 13, 2016 9:48 PM Just for fun, if you have a strange sense of humour: https://mathiasbynens.be/notes/javascript-unicode **************************************************************** From: Randy Brukardt Sent: Tuesday, August 16, 2016 1:55 AM Having spent the last 3 1/2 hours trying to understand Unicode identifier and normalization rules, I think I've finally got an acceptable update for this AI (as discussed in Pisa). I've attached the result (warning, it is in UTF-8 format as there are a bunch of Unicode characters in the text and !appendix). I'd appreciate a quick review, as I don't know how long I'm going to remember the Unicode details (I tried to write everything significant down, but if I missed anything, it's likely I'll forget it soon). The summary is: Ada identifiers that contain characters that are not allowed in Normalization Form KC are illegal. Ada text not in Normalization Form C is implementation-defined. I tried to explain the reasons for these choices (mostly to avoid implementation burden). The unusual wording of the first item is necessary, as Unicode has characters that *might* be allowed, and we want to allow those so no one has to implement a full normalization algorithm. (That would be unspeakable, the string could expand up to 3x bigger.) P.S. I just noticed that the !summary in the AI uses an obsolete version of the wording. Mentally substitute the above, I'll fix it. **************************************************************** From: Tucker Taft Sent: Tuesday, August 16, 2016 10:43 AM Minor typo in an AARM Reason: > ... interoperable with tools that produc[t]{e} unnormalized text. The wording what "might be allowed" is pretty weird, in 2.3(4/3): "An identifier shall only contain characters that might be allowed in Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011)." I see your explanation, but "might be allowed" is really hard to interpret. I could see "... that may be accepted ...," and that at least links to the "Maybe" notion in the standard. It does seem worth adding an Is_NFKC if it plays such an important role. **************************************************************** From: Randy Brukardt Sent: Tuesday, August 16, 2016 9:10 AM > The wording what "might be allowed" is pretty weird, in 2.3(4/3): > > "An identifier shall only contain characters that might be allowed in > Normalization Form KC (as defined by Clause 21 of ISO/IEC > 10646:2011)." > > I see your explanation, but "might be allowed" is really hard to > interpret. Sure. I would have preferred to exactly say what's meant, but the inability to reference Unicode makes that painful at best. > I could see "... that may be accepted ...," and that at least links to > the "Maybe" notion in the standard. I always try to avoid "may" because it is so easy to use "may not" (which is not allowed). But I don't quite see how "accepted" works here. The full sentence with your change is: An identifier shall only contain characters that may be accepted in Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011). Normalization is a conversion, and the input can have any Unicode characters. It's the *output* of normalization that has a limited character set. When I said "allowed", I meant "allowed in text normalized by Normalization Form KC". The existing text of 2.1(4.1/3) isn't very clear on this point, either. "in Normalization Form KC" really means "in text that has been normalized by Normalization Form KC". I tried to use the same (sloppy) meaning in the new wording, but perhaps we should clarify both?? I'll hold off changing anything here until I hear a better idea. > It does seem worth adding an Is_NFKC if it plays such an important > role. OK. And thanks for the quick response. **************************************************************** From: Tucker Taft Sent: Tuesday, August 16, 2016 9:41 PM > But I don't quite see how "accepted" works here. The full sentence > with your change is: > > An identifier shall only contain characters that may be accepted in > Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011). How about: "An identifier shall only container characters that may be present in Normalization Form KC ..." **************************************************************** From: Randy Brukardt Sent: Tuesday, August 16, 2016 11:38 PM That seems good. Sometimes I don't know how you do it. Thanks. **************************************************************** From: Randy Brukardt Sent: Tuesday, August 18, 2016 7:11 PM ... > How about: > > "An identifier shall only container characters that may be present in > Normalization Form KC ..." I just noticed the typo in the above. Funny how one sees what they expect. And of course that allowed me to muse on what the above actually means. Maybe a "container character" is one of "aceinort"? So 'z' isn't allowed in an identifier? Back to real work... **************************************************************** From: Jean-Pierre Rosen Sent: Wednesday, August 17, 2016 2:54 AM > "An identifier shall only contain characters that might be allowed in > Normalization Form KC (as defined by Clause 21 of ISO/IEC > 10646:2011)." > I'll hold off changing anything here until I hear a better idea. Reverse the proposition: "An identifier shall only contain characters that are not forbidden in Normalization Form KC (as defined by Clause 21 of ISO/IEC 10646:2011)." **************************************************************** From: Randy Brukardt Sent: Tuesday, August 16, 2016 10:21 PM I wrote: > The summary is: > > Ada identifiers that contain characters that are not allowed in > Normalization Form KC are illegal. In researching my response to Tuck, I reread the beginning of TR15. A short summary of the difference between the two Normalization Forms (which I added to the AI) is: NFKC merges characters based on their basic properties, ignoring their visual appearance; normalization form C (NFC) just puts characters into their minimal canonical format. (Minimal here means that characters that can be written as a sequence or as a single character are written as the single character.) NFC never changes the visual appearance of a character; NKFC eliminate visual variants of a character such as subscripting, superscripting, positional variants, and more. I have to wonder if eliminating positional variants (the obvious example being the two forms of lower case sigma in Greek (see 7.5 in the Ada 2005 Rationale for an example) would be bad in Ada identifiers. The special form for ending a word is not allowed in NFKC, and thus would not be allowed in an Ada identifier. We could do that by making the Legality Rule only reject characters not in NFC; case folding will cause the alternative forms to compare the same so that case insensitivity would be preserved. The case that we were mainly worried about of m² being treated the same as m2 isn't possible here; the problem case was where an implementation automatically normalized all program text (or just potential identifier text) with NFKC. Since NFKC changes superscript 2 into regular 2, that would make m² a legal identifier on that implementation and the same as m2. However, NFC doesn't change the superscript 2; and that character has character class Number, Other. A quick look at your RM will show that such a character is not allowed in an identifier, so there is no problem (with the implementation-definedness removed). It's hard to know if there are any other problems with this; most characters that are different probably aren't allowed in identifiers anyway. The font-specific characters (like Mathematical Sans Serif Captial A) would be allowed in identifiers in this scenario, and they'd be *different* from the similar regular characters (indeed, they have no case folding at all, which seems weird -- maybe there's something about those characters - 1D400 .. 1D7FF that I don't understand - there's some extra "" entries in their definition). Anyway, the question is whether making use of positional variants in Ada identifiers illegal is a problem. If so, we need to consider some other rule. Otherwise, the rule I have should be fine. **************************************************************** From: Tucker Taft Sent: Wednesday, August 17, 2016 6:02 AM > ... Anyway, the question is whether making use of positional variants > in Ada identifiers illegal is a problem. If so, we need to consider > some other rule. Otherwise, the rule I have should be fine. German S has a similar situation, I believe. Erhard? I would presume that currently most program identifiers do not expect positional variants to work, but who knows in the future? **************************************************************** From: Erhard Ploedereder Sent: Wednesday, August 17, 2016 8:32 AM > German S has a similar situation, I believe. Erhard? I would presume > that currently most program identifiers do not expect positional > variants to work, but who knows in the future? Indeed (I guess - I did not follow the normalization issues). It is the "scharfe S" ("ß" - which might not show correctly on your screen now), used only in lower-case. The upper-case equivalent is "SS". Replacement in lower case, extremely rarely used in the real word, is "sz". Its similarity to "b" or "B" in appearance gives rise to many jokes, e.g. when you ask for a "Scheibe Brot" (a slice of bread) and you mix up the "b" and the "ß". I won't elaborate. **************************************************************** From: Randy Brukardt Sent: Thursday, August 18, 2016 6:02 PM This isn't involved for this rule, as no letters in Latin-1 are rejected (that is, have the NFKC_QC property = No). The only Latin-1 characters that would be rejected aren't allowed in identifiers anyway (macron, superscript numbers, etc.). Greek seems to be the highest profile language involved; some non-latin-1 "Latin" characters, mostly ligatures, are also rejected. And then characters for languages like Bengali, Tibetian, Gurumukhi, and Devanagari. I was hoping to get a comment from John, who's responsible for the Greek examples I've seen. But I haven't been able to find them (I thought they were in a Rationale version, but definitely not). Anyway, it seems best to leave the rule as proposed until we get some feedback from someone that cares about Greek identifiers (or one of the other languages involved). It probably would be easier to make a special exception for a handful of characters after the fact (less incompatible) than to use a series of increasingly tight rules. Especially as it would be best to get as close as possible to the Unicode rules and properties. **************************************************************** From: Jeff Cousins Sent: Tuesday, October 25, 2016 4:56 AM Maybe someone else spotted this at Pittsburgh and I missed it, I didn't notice this before as the character doesn't appear in my hard-copy print-out, but several places have a capital A with circumflex between the m and the superscript 2. **************************************************************** From: Randy Brukardt Sent: Tuesday, October 25, 2016 2:08 PM Did you miss the note that the AI file is encoded in UTF-8?? (It's at the end of the summary.) It needs to use Unicode characters (of course), so if you read it with a dumb app you'll see various garbage rather than the correct text. OTOH, even Notepad displays it correctly, so you have to be using something *really* ancient or really dumb to get the wrong text. **************************************************************** From: Jeff Cousins Sent: Thursday, May 4, 2017 11:14 AM [Editor's note: This is in response to the request for HoDs to review AIs presented to WG9 for approval. Jeff is presenting UK comments. Only the comments on this AI are shown here.] I'll be off on holiday for the next fortnight, so here are the only comments that I've had so far: --- Sensible all the way back to Unicode 5.2 -- I'd assumed it had been resolved as a 2005 AI. Randy's surprise at "maybe" -- in the 5.2 discussion, it started as "sometimes", but that rather implied them analysing cases in which it was "yes", and when "no": when the workload of that proved both tricky and excessive, they tamed it down to "maybe" so that they could defer the analyses to a later version. Then they never got back to it. The reference to http://www.unicode.org/reports/tr15/tr15.html in http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0004-1.txt?rev=1.9&raw=N now 404s -- it's been replaced by a clean-URL'd generated file at just http://www.unicode.org/reports/tr15/. Sadly, the Unicode guys didn't put in a redirect from the old address -- easy when old addresses are .html. **************************************************************** From: Randy Brukardt Sent: Monday, June 5, 2017 9:05 PM A brief response to the comments > AI12-0004-1/06 2017-04-24 -- Normalization and allowed > characters for identifiers > > Sensible all the way back to Unicode 5.2 -- I'd assumed it > had been resolved as a 2005 AI. The question came in after the deadline for the Ada 2012 work, so it ended up as one of the very first Ada 2012 AIs. It SHOULD have been in the Corrigendum for Ada 2012, but yours truly miscategorized it as a Amendment AI when initially creating it, so we didn't take it up until WG 9 authorized work on Ada 202x. Thus it is just getting completed now. > Randy's surprise at "maybe" -- in the 5.2 discussion, it > started as "sometimes", but that rather implied them > analysing cases in which it was "yes", and when "no": when > the workload of that proved both tricky and excessive, they > tamed it down to "maybe" so that they could defer the > analyses to a later version. Then they never got back to it. I don't think I personally was "surprised" at "maybe" (if you've read TR15, you'll probably be unsurprised by anything character-set related :-), but I thought that most of the potential readers would be. I don't think that there is any other serious option; the entire point is that it is way too expensive to figure out the actual answer. In any case, my primary goal with the AI was to justify the decisions proposed to ARG members who probably did not have the time or inclination to spend 4+ hours reading and understanding TR15 and TR31. They were almost certain to be surprised by some of this. > The reference to > http://www.unicode.org/reports/tr15/tr15.html > in > > http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ai12s/ai12-0004-1.t > xt?rev=1.9&raw=N > now 404s -- it's been replaced by a clean-URL'd generated file at just > http://www.unicode.org/reports/tr15/. > Sadly, the Unicode guys didn't put in a redirect from the old > address -- easy when old addresses are .html. Even people who should know better don't follow good Internet practice. I think we're doomed. ;-) Anyway, I fixed the link in the AI. ****************************************************************