Skip to content
Snippets Groups Projects
  • Jakub Jelinek's avatar
    29341f21
    c++: Implement C++26 P2558R2 - Add @, $, and ` to the basic character set [PR110343] · 29341f21
    Jakub Jelinek authored
    The following patch implements the easy parts of the paper.
    When @$` are added to the basic character set, it means that
    R"@$`()@$`" should now be valid (here I've noticed most of the
    raw string tests were tested solely with -std=c++11 or -std=gnu++11
    and I've tried to change that), and on the other side even if
    by extension $ is allowed in identifiers, \u0024 or \U00000024
    or \u{24} should not be, similarly how \u0041 is not allowed.
    
    The paper in 3.1 claims though that
     #include <stdio.h>
    
     #define STR(x) #x
    
    int main()
    {
      printf("%s", STR(\u0060)); // U+0060 is ` GRAVE ACCENT
    }
    should have been accepted before this paper (and rejected after it),
    but g++ rejects it.
    
    I've tried to understand it, but am confused on what is the right
    behavior and why.
    
    Consider
     #define STR(x) #x
    const char *a = "\u00b7";
    const char *b = STR(\u00b7);
    const char *c = "\u0041";
    const char *d = STR(\u0041);
    const char *e = STR(a\u00b7);
    const char *f = STR(a\u0041);
    const char *g = STR(a \u00b7);
    const char *h = STR(a \u0041);
    const char *i = "\u066d";
    const char *j = STR(\u066d);
    const char *k = "\u0040";
    const char *l = STR(\u0040);
    const char *m = STR(a\u066d);
    const char *n = STR(a\u0040);
    const char *o = STR(a \u066d);
    const char *p = STR(a \u0040);
    
    Neither clang nor gcc emit any diagnostics on the a, c, i and k
    initializers, those are certainly valid (c is invalid in C23 though).  g++
    emits with -pedantic-errors errors on all the others, while clang++ on the
    ones with STR involving \u0041, \u0040 and a\u0066d.  The chosen values are
    \u0040 '@' as something being changed by this paper, \u0041 'A' as basic
    character set char valid in identifiers before/after, \u00b7 as an example
    of character which is pedantically valid in identifiers if not at the start
    and \u066d s something pedantically not valid in identifiers.
    
    Now, https://eel.is/c++draft/lex.charset#6 says that UCN used outside of a
    string/character literal which corresponds to basic character set character
    (or control character) is ill-formed, that would make d, f, h cases invalid
    for C++ and l, n, p cases invalid for C++26.
    
    https://eel.is/c++draft/lex.name states which characters can appear at the
    start of the identifier and which can appear after the start.  And
    https://eel.is/c++draft/lex.pptoken states that preprocessing-token is
    either identifier, or tons of other things, or "each non-whitespace
    character that cannot be one of the above"
    
    Then https://eel.is/c++draft/lex.pptoken#1 says that this last category is
    invalid if the preprocessing token is being converted into token.
    
    And https://eel.is/c++draft/lex.pptoken#2 includes "If any character not in
    the basic character set matches the last category, the program is
    ill-formed."
    
    Now, e.g.  for the C++23 STR(\u0040) case, \u0040 is there not in the basic
    character set, so valid outside of the literals (not the case anymore in
    C++26), but it isn't nondigit and doesn't have XID_Start property, so it
    isn't IMHO an identifier and so must be the "each non-whitespace character
    that cannot be one of the above" case.  Why doesn't the above mentioned
    https://eel.is/c++draft/lex.pptoken#2 sentence make that invalid?  Ignoring
    that, I'd say it would be then stringized and that feels like it is what
    clang++ is doing.  Now, e.g.  for the STR(a\u066d) case, I wonder why that
    isn't lexed as a identifier followed by \u066d "each non-whitespace
    character that cannot be one of the above" token and stringified similarly,
    clang++ rejects that.
    
    What GCC libcpp seems to be doing is that if that forms_identifier_p calls
    _cpp_valid_utf8 or _cpp_valid_ucn with an argument which tells it is first
    or second+ in identifier, and e.g.  _cpp_valid_ucn then for UCNs valid in
    string literals calls
      else if (identifier_pos)
        {
          int validity = ucn_valid_in_identifier (pfile, result, nst);
    
          if (validity == 0)
            cpp_error (pfile, CPP_DL_ERROR,
                       "universal character %.*s is not valid in an identifier",
                       (int) (str - base), base);
          else if (validity == 2 && identifier_pos == 1)
            cpp_error (pfile, CPP_DL_ERROR,
       "universal character %.*s is not valid at the start of an identifier",
                       (int) (str - base), base);
        }
    so basically all those invalid in identifiers cases emit an error and
    pretend to be valid in identifiers, rather than what e.g.  _cpp_valid_utf8
    does for C but not for C++ and only for the chars completely invalid in
    identifiers rather than just valid in identifiers but not at the start:
              /* In C++, this is an error for invalid character in an identifier
                 because logically, the UTF-8 was converted to a UCN during
                 translation phase 1 (even though we don't physically do it that
                 way).  In C, this byte rather becomes grammatically a separate
                 token.  */
    
              if (CPP_OPTION (pfile, cplusplus))
                cpp_error (pfile, CPP_DL_ERROR,
                           "extended character %.*s is not valid in an identifier",
                           (int) (*pstr - base), base);
              else
                {
                  *pstr = base;
                  return false;
                }
    The comment doesn't really match what is done in recent C++ versions because
    there UCNs are translated to characters and not the other way around.
    
    2024-07-25  Jakub Jelinek  <jakub@redhat.com>
    
    	PR c++/110343
    libcpp/
    	* lex.cc: C++26 P2558R2 - Add @, $, and ` to the basic character set.
    	(lex_raw_string): For C++26 allow $@` characters in prefix.
    	* charset.cc (_cpp_valid_ucn): For C++26 reject \u0024 in identifiers.
    gcc/testsuite/
    	* c-c++-common/raw-string-1.c: Use { c || c++11 } effective target,
    	remove c++ specific dg-options.
    	* c-c++-common/raw-string-2.c: Likewise.
    	* c-c++-common/raw-string-4.c: Likewise.
    	* c-c++-common/raw-string-5.c: Likewise.  Expect some diagnostics
    	only for non-c++26, for c++26 expect different.
    	* c-c++-common/raw-string-6.c: Use { c || c++11 } effective target,
    	remove c++ specific dg-options.
    	* c-c++-common/raw-string-11.c: Likewise.
    	* c-c++-common/raw-string-13.c: Likewise.
    	* c-c++-common/raw-string-14.c: Likewise.
    	* c-c++-common/raw-string-15.c: Use { c || c++11 } effective target,
    	change c++ specific dg-options to just -Wtrigraphs.
    	* c-c++-common/raw-string-16.c: Likewise.
    	* c-c++-common/raw-string-17.c: Use { c || c++11 } effective target,
    	remove c++ specific dg-options.
    	* c-c++-common/raw-string-18.c: Use { c || c++11 } effective target,
    	remove -std=c++11 from c++ specific dg-options.
    	* c-c++-common/raw-string-19.c: Likewise.
    	* g++.dg/cpp26/raw-string1.C: New test.
    	* g++.dg/cpp26/raw-string2.C: New test.
    29341f21
    History
    c++: Implement C++26 P2558R2 - Add @, $, and ` to the basic character set [PR110343]
    Jakub Jelinek authored
    The following patch implements the easy parts of the paper.
    When @$` are added to the basic character set, it means that
    R"@$`()@$`" should now be valid (here I've noticed most of the
    raw string tests were tested solely with -std=c++11 or -std=gnu++11
    and I've tried to change that), and on the other side even if
    by extension $ is allowed in identifiers, \u0024 or \U00000024
    or \u{24} should not be, similarly how \u0041 is not allowed.
    
    The paper in 3.1 claims though that
     #include <stdio.h>
    
     #define STR(x) #x
    
    int main()
    {
      printf("%s", STR(\u0060)); // U+0060 is ` GRAVE ACCENT
    }
    should have been accepted before this paper (and rejected after it),
    but g++ rejects it.
    
    I've tried to understand it, but am confused on what is the right
    behavior and why.
    
    Consider
     #define STR(x) #x
    const char *a = "\u00b7";
    const char *b = STR(\u00b7);
    const char *c = "\u0041";
    const char *d = STR(\u0041);
    const char *e = STR(a\u00b7);
    const char *f = STR(a\u0041);
    const char *g = STR(a \u00b7);
    const char *h = STR(a \u0041);
    const char *i = "\u066d";
    const char *j = STR(\u066d);
    const char *k = "\u0040";
    const char *l = STR(\u0040);
    const char *m = STR(a\u066d);
    const char *n = STR(a\u0040);
    const char *o = STR(a \u066d);
    const char *p = STR(a \u0040);
    
    Neither clang nor gcc emit any diagnostics on the a, c, i and k
    initializers, those are certainly valid (c is invalid in C23 though).  g++
    emits with -pedantic-errors errors on all the others, while clang++ on the
    ones with STR involving \u0041, \u0040 and a\u0066d.  The chosen values are
    \u0040 '@' as something being changed by this paper, \u0041 'A' as basic
    character set char valid in identifiers before/after, \u00b7 as an example
    of character which is pedantically valid in identifiers if not at the start
    and \u066d s something pedantically not valid in identifiers.
    
    Now, https://eel.is/c++draft/lex.charset#6 says that UCN used outside of a
    string/character literal which corresponds to basic character set character
    (or control character) is ill-formed, that would make d, f, h cases invalid
    for C++ and l, n, p cases invalid for C++26.
    
    https://eel.is/c++draft/lex.name states which characters can appear at the
    start of the identifier and which can appear after the start.  And
    https://eel.is/c++draft/lex.pptoken states that preprocessing-token is
    either identifier, or tons of other things, or "each non-whitespace
    character that cannot be one of the above"
    
    Then https://eel.is/c++draft/lex.pptoken#1 says that this last category is
    invalid if the preprocessing token is being converted into token.
    
    And https://eel.is/c++draft/lex.pptoken#2 includes "If any character not in
    the basic character set matches the last category, the program is
    ill-formed."
    
    Now, e.g.  for the C++23 STR(\u0040) case, \u0040 is there not in the basic
    character set, so valid outside of the literals (not the case anymore in
    C++26), but it isn't nondigit and doesn't have XID_Start property, so it
    isn't IMHO an identifier and so must be the "each non-whitespace character
    that cannot be one of the above" case.  Why doesn't the above mentioned
    https://eel.is/c++draft/lex.pptoken#2 sentence make that invalid?  Ignoring
    that, I'd say it would be then stringized and that feels like it is what
    clang++ is doing.  Now, e.g.  for the STR(a\u066d) case, I wonder why that
    isn't lexed as a identifier followed by \u066d "each non-whitespace
    character that cannot be one of the above" token and stringified similarly,
    clang++ rejects that.
    
    What GCC libcpp seems to be doing is that if that forms_identifier_p calls
    _cpp_valid_utf8 or _cpp_valid_ucn with an argument which tells it is first
    or second+ in identifier, and e.g.  _cpp_valid_ucn then for UCNs valid in
    string literals calls
      else if (identifier_pos)
        {
          int validity = ucn_valid_in_identifier (pfile, result, nst);
    
          if (validity == 0)
            cpp_error (pfile, CPP_DL_ERROR,
                       "universal character %.*s is not valid in an identifier",
                       (int) (str - base), base);
          else if (validity == 2 && identifier_pos == 1)
            cpp_error (pfile, CPP_DL_ERROR,
       "universal character %.*s is not valid at the start of an identifier",
                       (int) (str - base), base);
        }
    so basically all those invalid in identifiers cases emit an error and
    pretend to be valid in identifiers, rather than what e.g.  _cpp_valid_utf8
    does for C but not for C++ and only for the chars completely invalid in
    identifiers rather than just valid in identifiers but not at the start:
              /* In C++, this is an error for invalid character in an identifier
                 because logically, the UTF-8 was converted to a UCN during
                 translation phase 1 (even though we don't physically do it that
                 way).  In C, this byte rather becomes grammatically a separate
                 token.  */
    
              if (CPP_OPTION (pfile, cplusplus))
                cpp_error (pfile, CPP_DL_ERROR,
                           "extended character %.*s is not valid in an identifier",
                           (int) (*pstr - base), base);
              else
                {
                  *pstr = base;
                  return false;
                }
    The comment doesn't really match what is done in recent C++ versions because
    there UCNs are translated to characters and not the other way around.
    
    2024-07-25  Jakub Jelinek  <jakub@redhat.com>
    
    	PR c++/110343
    libcpp/
    	* lex.cc: C++26 P2558R2 - Add @, $, and ` to the basic character set.
    	(lex_raw_string): For C++26 allow $@` characters in prefix.
    	* charset.cc (_cpp_valid_ucn): For C++26 reject \u0024 in identifiers.
    gcc/testsuite/
    	* c-c++-common/raw-string-1.c: Use { c || c++11 } effective target,
    	remove c++ specific dg-options.
    	* c-c++-common/raw-string-2.c: Likewise.
    	* c-c++-common/raw-string-4.c: Likewise.
    	* c-c++-common/raw-string-5.c: Likewise.  Expect some diagnostics
    	only for non-c++26, for c++26 expect different.
    	* c-c++-common/raw-string-6.c: Use { c || c++11 } effective target,
    	remove c++ specific dg-options.
    	* c-c++-common/raw-string-11.c: Likewise.
    	* c-c++-common/raw-string-13.c: Likewise.
    	* c-c++-common/raw-string-14.c: Likewise.
    	* c-c++-common/raw-string-15.c: Use { c || c++11 } effective target,
    	change c++ specific dg-options to just -Wtrigraphs.
    	* c-c++-common/raw-string-16.c: Likewise.
    	* c-c++-common/raw-string-17.c: Use { c || c++11 } effective target,
    	remove c++ specific dg-options.
    	* c-c++-common/raw-string-18.c: Use { c || c++11 } effective target,
    	remove -std=c++11 from c++ specific dg-options.
    	* c-c++-common/raw-string-19.c: Likewise.
    	* g++.dg/cpp26/raw-string1.C: New test.
    	* g++.dg/cpp26/raw-string2.C: New test.