The PR complains that we do not handle UTF-8 in the suffix for a user-defined
literal, such as:

bool operator ""_π (unsigned long long);

In fact we don't handle any extended identifier characters there, whether
UTF-8, UCNs, or the $ sign. We do handle it fine if the optional space after
the "" tokens is included, since then the identifier is lexed in the
"normal" way as its own token. But when it is lexed as part of the string
token, this is handled in lex_string() with a one-off loop that is not aware
of extended characters.

This patch fixes it by adding a new function scan_cur_identifier() that can
be used to lex an identifier while in the middle of lexing another token.

BTW, the other place that has been mis-lexing identifiers is
lex_identifier_intern(), which is used to implement #pragma push_macro
and #pragma pop_macro. This does not support extended characters either.
I will add that in a subsequent patch, because it can't directly reuse the
new function, but rather needs to lex from a string instead of a cpp_buffer.

With scan_cur_identifier(), we do also correctly warn about bidi and
normalization issues in the extended identifiers comprising the suffix.

libcpp/ChangeLog:

	PR preprocessor/103902
	* lex.cc (identifier_diagnostics_on_lex): New function refactoring
	some common code.
	(lex_identifier_intern): Use the new function.
	(lex_identifier): Don't run identifier diagnostics here, rather let
	the call site do it when needed.
	(_cpp_lex_direct): Adjust the call sites of lex_identifier ()
	acccordingly.
	(struct scan_id_result): New struct.
	(scan_cur_identifier): New function.
	(create_literal2): New function.
	(lit_accum::create_literal2): New function.
	(is_macro): Folded into new function...
	(maybe_ignore_udl_macro_suffix): ...here.
	(is_macro_not_literal_suffix): Folded likewise.
	(lex_raw_string): Handle UTF-8 in UDL suffix via
	scan_cur_identifier ().
	(lex_string): Likewise.

gcc/testsuite/ChangeLog:

	PR preprocessor/103902
	* g++.dg/cpp0x/udlit-extended-id-1.C: New test.
	* g++.dg/cpp0x/udlit-extended-id-2.C: New test.
	* g++.dg/cpp0x/udlit-extended-id-3.C: New test.
	* g++.dg/cpp0x/udlit-extended-id-4.C: New test.

We're (currently) not aware of any actual use of 'ht_identifier's with NUL
characters embedded; its 'len' field appears to exist for optimization
purposes, since "forever".  Before 'struct ht_identifier' was added in
commit 2a967f3d (Subversion r42334), we had in
'gcc/cpplib.h:struct cpp_hashnode': 'unsigned short len', or earlier 'length',
earlier in 'gcc/cpphash.h:struct hashnode': 'unsigned short length', earlier
'size_t length' with comment: "length of token, for quick comparison", earlier
'int length', ever since the 'gcc/cpp*' files were added in
commit 7f2935c7 (Subversion r9191).

This amends commit f3b957ea
"pch: Fix streaming of strings with embedded null bytes".

	gcc/
	* doc/gty.texi (GTY Options) <string_length>: Enhance.
	libcpp/
	* include/symtab.h (struct ht_identifier): Document different
	rationale.

It seems prudent to add C++26 now that the first C++26 papers have been
approved.  I followed commit r11-6920 as well as r8-3237.

Since C++23 is essentially finished and its __cplusplus value has
settled to 202302L, I've updated cpp_init_builtins and marked
-std=c++2b Undocumented and made -std=c++23 no longer Undocumented.

As for __cplusplus, I've chosen 202400L:

  $ xg++ -std=c++26 -dM -E -x c++ - < /dev/null | grep cplusplus
  #define __cplusplus 202400L

I've verified the patch with a simple test, exercising the new
directives.  Don't forget to update your GXX_TESTSUITE_STDS!

This patch does not add -Wc++26-extensions.

gcc/c-family/ChangeLog:

	* c-common.h (cxx_dialect): Add cxx26 as a dialect.
	* c-opts.cc (set_std_cxx26): New.
	(c_common_handle_option): Set options when -std={c,gnu}++2{c,6} is
	enabled.
	(c_common_post_options): Adjust comments.
	* c.opt: Add options for -std=c++26, std=c++2c, -std=gnu++26,
	and -std=gnu++2c.
	(std=c++2b): Mark as Undocumented.
	(std=c++23): No longer Undocumented.

gcc/ChangeLog:

	* doc/cpp.texi (__cplusplus): Document value for -std=c++26 and
	-std=gnu++26.  Document that for C++23, its value is 202302L.
	* doc/invoke.texi: Document -std=c++26 and -std=gnu++26.
	* dwarf2out.cc (highest_c_language): Handle GNU C++26.
	(gen_compile_unit_die): Likewise.

libcpp/ChangeLog:

	* include/cpplib.h (c_lang): Add CXX26 and GNUCXX26.
	* init.cc (lang_defaults): Add rows for CXX26 and GNUCXX26.
	(cpp_init_builtins): Set __cplusplus to 202400L for C++26.
	Set __cplusplus to 202302L for C++23.

gcc/testsuite/ChangeLog:

	* lib/target-supports.exp (check_effective_target_c++23): Return
	1 also if check_effective_target_c++26.
	(check_effective_target_c++23_down): New.
	(check_effective_target_c++26_only): New.
	(check_effective_target_c++26): New.
	* g++.dg/cpp23/cplusplus.C: Adjust expected value.
	* g++.dg/cpp26/cplusplus.C: New test.

Fixes r14-1954 (libcpp: reject codepoints above 0x10FFFF, 2023-06-06)

libcpp/

	* charset.cc: Allow `UCS_LIMIT` in UTF-8 strings.

Reported-by: Damien Guibouret <damien.guibouret@partition-saving.com>
Signed-off-by: Ben Boeckel <ben.boeckel@kitware.com>

Existing text output in GCC has to be implemented by writing
sequentially to a pretty_printer instance.  This makes it
hard to implement some kinds of diagnostic output (see e.g.
diagnostic-show-locus.cc).

This patch adds more flexible ways of creating text output:
- a canvas class, which can be "painted" to via random-access (rather
that sequentially)
- a table class for 2D grid layout, supporting items that span
multiple rows/columns
- a widget class for organizing diagrams hierarchically.

The patch also expands GCC's diagnostics subsystem so that diagnostics
can have "text art" diagrams - think ASCII art, but potentially
including some Unicode characters, such as box-drawing chars.

The new code is in a new "gcc/text-art" subdirectory and "text_art"
namespace.

The patch adds a new "-fdiagnostics-text-art-charset=VAL" option, with
values:
- "none": don't emit diagrams (added to -fdiagnostics-plain-output)
- "ascii": use pure ASCII in diagrams
- "unicode": allow for conservative use of unicode drawing characters
(such as box-drawing characters).
- "emoji" (the default): as "unicode", but potentially allow for
conservative use of emoji in the output (such as U+26A0 WARNING SIGN).
I made it possible to disable emoji separately from unicode as I believe
there's a generation gap in acceptance of these characters (some older
programmers have a visceral reaction against them, whereas younger
programmers may have no problem with them).

Diagrams are emitted to stderr by default.  With SARIF output they are
captured as a location in "relatedLocations", with the diagram as a
code block in Markdown within a "markdown" property of a message.

This patch doesn't add any such diagram usage to GCC, saving that for
followups, apart from adding a plugin to the test suite to exercise the
functionality.

contrib/ChangeLog:
	* unicode/gen-box-drawing-chars.py: New file.
	* unicode/gen-combining-chars.py: New file.
	* unicode/gen-printable-chars.py: New file.

gcc/ChangeLog:
	* Makefile.in (OBJS-libcommon): Add text-art/box-drawing.o,
	text-art/canvas.o, text-art/ruler.o, text-art/selftests.o,
	text-art/style.o, text-art/styled-string.o, text-art/table.o,
	text-art/theme.o, and text-art/widget.o.
	* color-macros.h (COLOR_FG_BRIGHT_BLACK): New.
	(COLOR_FG_BRIGHT_RED): New.
	(COLOR_FG_BRIGHT_GREEN): New.
	(COLOR_FG_BRIGHT_YELLOW): New.
	(COLOR_FG_BRIGHT_BLUE): New.
	(COLOR_FG_BRIGHT_MAGENTA): New.
	(COLOR_FG_BRIGHT_CYAN): New.
	(COLOR_FG_BRIGHT_WHITE): New.
	(COLOR_BG_BRIGHT_BLACK): New.
	(COLOR_BG_BRIGHT_RED): New.
	(COLOR_BG_BRIGHT_GREEN): New.
	(COLOR_BG_BRIGHT_YELLOW): New.
	(COLOR_BG_BRIGHT_BLUE): New.
	(COLOR_BG_BRIGHT_MAGENTA): New.
	(COLOR_BG_BRIGHT_CYAN): New.
	(COLOR_BG_BRIGHT_WHITE): New.
	* common.opt (fdiagnostics-text-art-charset=): New option.
	(diagnostic-text-art.h): New SourceInclude.
	(diagnostic_text_art_charset) New Enum and EnumValues.
	* configure: Regenerate.
	* configure.ac (gccdepdir): Add text-art to loop.
	* diagnostic-diagram.h: New file.
	* diagnostic-format-json.cc (json_emit_diagram): New.
	(diagnostic_output_format_init_json): Wire it up to
	context->m_diagrams.m_emission_cb.
	* diagnostic-format-sarif.cc: Include "diagnostic-diagram.h" and
	"text-art/canvas.h".
	(sarif_result::on_nested_diagnostic): Move code to...
	(sarif_result::add_related_location): ...this new function.
	(sarif_result::on_diagram): New.
	(sarif_builder::emit_diagram): New.
	(sarif_builder::make_message_object_for_diagram): New.
	(sarif_emit_diagram): New.
	(diagnostic_output_format_init_sarif): Set
	context->m_diagrams.m_emission_cb to sarif_emit_diagram.
	* diagnostic-text-art.h: New file.
	* diagnostic.cc: Include "diagnostic-text-art.h",
	"diagnostic-diagram.h", and "text-art/theme.h".
	(diagnostic_initialize): Initialize context->m_diagrams and
	call diagnostics_text_art_charset_init.
	(diagnostic_finish): Clean up context->m_diagrams.m_theme.
	(diagnostic_emit_diagram): New.
	(diagnostics_text_art_charset_init): New.
	* diagnostic.h (text_art::theme): New forward decl.
	(class diagnostic_diagram): Likewise.
	(diagnostic_context::m_diagrams): New field.
	(diagnostic_emit_diagram): New decl.
	* doc/invoke.texi (Diagnostic Message Formatting Options): Add
	-fdiagnostics-text-art-charset=.
	(-fdiagnostics-plain-output): Add
	-fdiagnostics-text-art-charset=none.
	* gcc.cc: Include "diagnostic-text-art.h".
	(driver_handle_option): Handle OPT_fdiagnostics_text_art_charset_.
	* opts-common.cc (decode_cmdline_options_to_array): Add
	"-fdiagnostics-text-art-charset=none" to expanded_args for
	-fdiagnostics-plain-output.
	* opts.cc: Include "diagnostic-text-art.h".
	(common_handle_option): Handle OPT_fdiagnostics_text_art_charset_.
	* pretty-print.cc (pp_unicode_character): New.
	* pretty-print.h (pp_unicode_character): New decl.
	* selftest-run-tests.cc: Include "text-art/selftests.h".
	(selftest::run_tests): Call text_art_tests.
	* text-art/box-drawing-chars.inc: New file, generated by
	contrib/unicode/gen-box-drawing-chars.py.
	* text-art/box-drawing.cc: New file.
	* text-art/box-drawing.h: New file.
	* text-art/canvas.cc: New file.
	* text-art/canvas.h: New file.
	* text-art/ruler.cc: New file.
	* text-art/ruler.h: New file.
	* text-art/selftests.cc: New file.
	* text-art/selftests.h: New file.
	* text-art/style.cc: New file.
	* text-art/styled-string.cc: New file.
	* text-art/table.cc: New file.
	* text-art/table.h: New file.
	* text-art/theme.cc: New file.
	* text-art/theme.h: New file.
	* text-art/types.h: New file.
	* text-art/widget.cc: New file.
	* text-art/widget.h: New file.

gcc/testsuite/ChangeLog:
	* gcc.dg/plugin/diagnostic-test-text-art-ascii-bw.c: New test.
	* gcc.dg/plugin/diagnostic-test-text-art-ascii-color.c: New test.
	* gcc.dg/plugin/diagnostic-test-text-art-none.c: New test.
	* gcc.dg/plugin/diagnostic-test-text-art-unicode-bw.c: New test.
	* gcc.dg/plugin/diagnostic-test-text-art-unicode-color.c: New test.
	* gcc.dg/plugin/diagnostic_plugin_test_text_art.c: New test plugin.
	* gcc.dg/plugin/plugin.exp (plugin_test_list): Add them.

libcpp/ChangeLog:
	* charset.cc (get_cppchar_property): New function template, based
	on...
	(cpp_wcwidth): ...this function.  Rework to use the above.
	Include "combining-chars.inc".
	(cpp_is_combining_char): New function
	Include "printable-chars.inc".
	(cpp_is_printable_char): New function
	* combining-chars.inc: New file, generated by
	contrib/unicode/gen-combining-chars.py.
	* include/cpplib.h (cpp_is_combining_char): New function decl.
	(cpp_is_printable_char): New function decl.
	* printable-chars.inc: New file, generated by
	contrib/unicode/gen-printable-chars.py.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>

When libcpp reports diagnostics whose locus is a macro name (such as for
-Wunused-macros), it uses the location in the cpp_macro object that was
stored by _cpp_new_macro. This is currently set to pfile->directive_line,
which contains the line number only and no column information. This patch
changes the stored location to the src_loc for the token defining the macro
name, which includes the location and range information.

libcpp/ChangeLog:

	PR c++/66290
	* macro.cc (_cpp_create_definition): Add location argument.
	* internal.h (_cpp_create_definition): Adjust prototype.
	* directives.cc (do_define): Pass new location argument to
	_cpp_create_definition.
	(do_undef): Stop passing inferior location to cpp_warning_with_line;
	the default from cpp_warning is better.
	(cpp_pop_definition): Pass new location argument to
	_cpp_create_definition.
	* pch.cc (cpp_read_state): Likewise.

gcc/testsuite/ChangeLog:

	PR c++/66290
	* c-c++-common/cpp/macro-ranges.c: New test.
	* c-c++-common/cpp/line-2.c: Adapt to check for column information
	on macro-related libcpp warnings.
	* c-c++-common/cpp/line-3.c: Likewise.
	* c-c++-common/cpp/macro-arg-count-1.c: Likewise.
	* c-c++-common/cpp/pr58844-1.c: Likewise.
	* c-c++-common/cpp/pr58844-2.c: Likewise.
	* c-c++-common/cpp/warning-zero-location.c: Likewise.
	* c-c++-common/pragma-diag-14.c: Likewise.
	* c-c++-common/pragma-diag-15.c: Likewise.
	* g++.dg/modules/macro-2_d.C: Likewise.
	* g++.dg/modules/macro-4_d.C: Likewise.
	* g++.dg/modules/macro-4_e.C: Likewise.
	* g++.dg/spellcheck-macro-ordering.C: Likewise.
	* gcc.dg/builtin-redefine.c: Likewise.
	* gcc.dg/cpp/Wunused.c: Likewise.
	* gcc.dg/cpp/redef2.c: Likewise.
	* gcc.dg/cpp/redef3.c: Likewise.
	* gcc.dg/cpp/redef4.c: Likewise.
	* gcc.dg/cpp/ucnid-11-utf8.c: Likewise.
	* gcc.dg/cpp/ucnid-11.c: Likewise.
	* gcc.dg/cpp/undef2.c: Likewise.
	* gcc.dg/cpp/warn-redefined-2.c: Likewise.
	* gcc.dg/cpp/warn-redefined.c: Likewise.
	* gcc.dg/cpp/warn-unused-macros-2.c: Likewise.
	* gcc.dg/cpp/warn-unused-macros.c: Likewise.

Unicode does not support such values because they are unrepresentable in
UTF-16.

libcpp/

	* charset.cc: Reject encodings of codepoints above 0x10FFFF.
	UTF-16 does not support such codepoints and therefore all
	Unicode rejects such values.

Signed-off-by: Ben Boeckel <ben.boeckel@kitware.com>

[ This is my third attempt to add this configure option.  The first
version was approved but it came too late in the development cycle.
The second version was also approved, but I had to revert it:
<https://gcc.gnu.org/pipermail/gcc-patches/2022-November/607082.html>.
I've fixed the problem (by moving $(PICFLAG) from INTERNAL_CFLAGS to
ALL_COMPILERFLAGS).  Another change is that since r13-4536 I no longer
need to touch Makefile.def, so this patch is simplified. ]

This patch implements the --enable-host-pie configure option which
makes the compiler executables PIE.  This can be used to enhance
protection against ROP attacks, and can be viewed as part of a wider
trend to harden binaries.

It is similar to the option --enable-host-shared, except that --e-h-s
won't add -shared to the linker flags whereas --e-h-p will add -pie.
It is different from --enable-default-pie because that option just
adds an implicit -fPIE/-pie when the compiler is invoked, but the
compiler itself isn't PIE.

Since r12-5768-gfe7c3ecf, PCH works well with PIE, so there are no PCH
regressions.

When building the compiler, the build process may use various in-tree
libraries; these need to be built with -fPIE so that it's possible to
use them when building a PIE.  For instance, when --with-included-gettext
is in effect, intl object files must be compiled with -fPIE.  Similarly,
when building in-tree gmp, isl, mpfr and mpc, they must be compiled with
-fPIE.

With this patch and --enable-host-pie used to configure gcc:

$ file gcc/cc1{,plus,obj,gm2} gcc/f951 gcc/lto1 gcc/cpp gcc/go1 gcc/rust1 gcc/gnat1
gcc/cc1:     ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=98e22cde129d304aa6f33e61b1c39e144aeb135e, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/cc1plus: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=859d1ea37e43dfe50c18fd4e3dd9a34bb1db8f77, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/cc1obj:  ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1964f8ecee6163182bc26134e2ac1f324816e434, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/cc1gm2:  ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=a396672c7ff913d21855829202e7b02ecf42ff4c, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/f951:    ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=59c523db893186547ac75c7a71f48be0a461c06b, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/lto1:    ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=084a7b77df7be2d63c2d4c655b5bbc3fcdb6038d, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/cpp:     ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3503bf8390d219a10d6653b8560aa21158132168, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/go1:     ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=988cc673af4fba5dcb482f4b34957b99050a68c5, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/rust1:   ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=b6a5d3d514446c4dcdee0707f086ab9b274a8a3c, for GNU/Linux 3.2.0, with debug_info, not stripped
gcc/gnat1:   ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=bb11ccdc2c366fe3fe0980476bcd8ca19b67f9dc, for GNU/Linux 3.2.0, with debug_info, not stripped

I plan to add an option to link with -Wl,-z,now.

Bootstrapped on x86_64-pc-linux-gnu with --with-included-gettext
--enable-host-pie as well as without --enable-host-pie.  Also tested
on a Debian system where the system gcc was configured with
--enable-default-pie.

Co-Authored by: Iain Sandoe  <iain@sandoe.co.uk>

ChangeLog:

	* configure.ac (--enable-host-pie): New check.  Set PICFLAG after this
	check.
	* configure: Regenerate.

c++tools/ChangeLog:

	* Makefile.in: Rename PIEFLAG to PICFLAG.  Set LD_PICFLAG.  Use it.
	Use pic/libiberty.a if PICFLAG is set.
	* configure.ac (--enable-default-pie): Set PICFLAG instead of PIEFLAG.
	(--enable-host-pie): New check.
	* configure: Regenerate.

fixincludes/ChangeLog:

	* Makefile.in: Set and use PICFLAG and LD_PICFLAG.  Use the "pic"
	build of libiberty if PICFLAG is set.
	* configure.ac:
	* configure: Regenerate.

gcc/ChangeLog:

	* Makefile.in: Set LD_PICFLAG.  Use it.  Set enable_host_pie.
	Remove NO_PIE_CFLAGS and NO_PIE_FLAG.  Pass LD_PICFLAG to
	ALL_LINKERFLAGS.  Use the "pic" build of libiberty if --enable-host-pie.
	* configure.ac (--enable-host-shared): Don't set PICFLAG here.
	(--enable-host-pie): New check.  Set PICFLAG and LD_PICFLAG after this
	check.
	* configure: Regenerate.
	* doc/install.texi: Document --enable-host-pie.

gcc/ada/ChangeLog:

	* gcc-interface/Make-lang.in (ALL_ADAFLAGS): Remove NO_PIE_CFLAGS.  Add
	PICFLAG.  Use PICFLAG when building ada/b_gnat1.o and ada/b_gnatb.o.
	* gcc-interface/Makefile.in: Use pic/libiberty.a if PICFLAG is set.
	Remove NO_PIE_FLAG.

gcc/m2/ChangeLog:

	* Make-lang.in: New var, GM2_PICFLAGS.  Use it.

gcc/d/ChangeLog:

	* Make-lang.in: Remove NO_PIE_CFLAGS.

intl/ChangeLog:

	* Makefile.in: Use @PICFLAG@ in COMPILE as well.
	* configure.ac (--enable-host-shared): Don't set PICFLAG here.
	(--enable-host-pie): New check.  Set PICFLAG after this check.
	* configure: Regenerate.

libcody/ChangeLog:

	* Makefile.in: Pass LD_PICFLAG to LDFLAGS.
	* configure.ac (--enable-host-shared): Don't set PICFLAG here.
	(--enable-host-pie): New check.  Set PICFLAG and LD_PICFLAG after this
	check.
	* configure: Regenerate.

libcpp/ChangeLog:

	* configure.ac (--enable-host-shared): Don't set PICFLAG here.
	(--enable-host-pie): New check.  Set PICFLAG after this check.
	* configure: Regenerate.

libdecnumber/ChangeLog:

	* configure.ac (--enable-host-shared): Don't set PICFLAG here.
	(--enable-host-pie): New check.  Set PICFLAG after this check.
	* configure: Regenerate.

libiberty/ChangeLog:

	* configure.ac: Also set shared when enable_host_pie.
	* configure: Regenerate.

zlib/ChangeLog:

	* configure.ac (--enable-host-shared): Don't set PICFLAG here.
	(--enable-host-pie): New check.  Set PICFLAG after this check.
	* configure: Regenerate.

As can be seen in the testcase, we don't diagnose #include/#include_next
of a non-existent header if __has_include/__has_include_next is done for
that header first.
The problem is that we normally error the first time some header is not
found, but in the _cpp_FFK_HAS_INCLUDE case obviously don't want to diagnose
it, just expand it to 0.  And libcpp caches both successful includes and
unsuccessful ones.

The following patch fixes that by remembering that we haven't diagnosed
error when using __has_include* on it, and diagnosing it when using the
cache entry in normal mode the first time.

I think _cpp_FFK_NORMAL is the only mode in which we normally diagnose
errors, for _cpp_FFK_PRE_INCLUDE that open_file_failed isn't reached
and for _cpp_FFK_FAKE neither.

2023-06-15  Jakub Jelinek  <jakub@redhat.com>

	PR preprocessor/80753
libcpp/
	* files.cc (struct _cpp_file): Add deferred_error bitfield.
	(_cpp_find_file): When finding a file in cache with deferred_error
	set in _cpp_FFK_NORMAL mode, call open_file_failed and clear the flag.
	Set deferred_error in _cpp_FFK_HAS_INCLUDE mode if open_file_failed
	hasn't been called.
gcc/testsuite/
	* c-c++-common/missing-header-5.c: New test.

	* ru.po: Update.

This reverts commit e7ce7c49.

From 0821df518b264e754d698d399f98be1a62945e32 Mon Sep 17 00:00:00 2001
From: Longjun Luo <luolongjuna@gmail.com>
Date: Thu, 12 Jan 2023 23:59:54 +0800
Subject: [PATCH] libcpp: suppress builtin macro redefined warnings for
 __LINE__

As implied in
gcc.gnu.org/legacy-ml/gcc-patches/2008-09/msg00076.html,
gcc provides -Wno-builtin-macro-redefined to suppress warning when
redefining builtin macro. However, at that time, there was no
scenario for __LINE__ macro.

But, when we try to build a live-patch, we compare sections by using
-ffunction-sections. Some same functions are considered changed because
of __LINE__ macro.

At present, to detect such a changed caused by __LINE__ macro, we
have to analyse code and maintain a function list. For example,
in kpatch, check this commit
github.com/dynup/kpatch/commit/0e1b95edeafa36edb7bcf11da6d1c00f76d7e03d.

So, in this scenario, when we try to compared sections, it would
be better to support suppress builtin macro redefined warnings for
__LINE__ macro.

libcpp:
	* init.cc (builtin_array): Do not always warn for a redefinition
	of __LINE__.

gcc/testsuite

	* gcc.dg/builtin-redefine.c: Test for redefintion warnings
	for __LINE__.
	* gcc.dg/builtin-redefine-1.c: New test.

PR analyzer/109098 notes that the SARIF spec mandates that .sarif
files are UTF-8 encoded, but -fdiagnostics-format=sarif-file naively
assumes that the source files are UTF-8 encoded when quoting source
artefacts in the .sarif output, which can lead to us writing out
.sarif files with non-UTF-8 bytes in them (which break my reporting
scripts).

The root cause is that sarif_builder::maybe_make_artifact_content_object
was using maybe_read_file to load the file content as bytes, and
assuming they were UTF-8 encoded.

This patch reworks both overloads of this function (one used for the
whole file, the other for snippets of quoted lines) so that they go
through input.cc's file cache, which attempts to decode the input files
according to the input charset, and then encode as UTF-8.  They also
check that the result actually is UTF-8, for cases where the input
charset is missing, or incorrectly specified, and omit the quoted
source for such awkward cases.

Doing so fixes all of the cases I've encountered.

The patch adds a new:
  { dg-final { verify-sarif-file } }
directive to all SARIF test cases in the test suite, which verifies
that the output is UTF-8 encoded, and is valid JSON.  In particular
it verifies that when we complain about encoding problems, the .sarif
report we emit is itself correctly encoded.

gcc/ChangeLog:
	PR analyzer/109098
	* diagnostic-format-sarif.cc (read_until_eof): Delete.
	(maybe_read_file): Delete.
	(sarif_builder::maybe_make_artifact_content_object): Use
	get_source_file_content rather than maybe_read_file.
	Reject it if it's not valid UTF-8.
	* input.cc (file_cache_slot::get_full_file_content): New.
	(get_source_file_content): New.
	(selftest::check_cpp_valid_utf8_p): New.
	(selftest::test_cpp_valid_utf8_p): New.
	(selftest::input_cc_tests): Call selftest::test_cpp_valid_utf8_p.
	* input.h (get_source_file_content): New prototype.

gcc/testsuite/ChangeLog:
	PR analyzer/109098
	* c-c++-common/diagnostic-format-sarif-file-1.c: Add
	verify-sarif-file directive.
	* c-c++-common/diagnostic-format-sarif-file-2.c: Likewise.
	* c-c++-common/diagnostic-format-sarif-file-3.c: Likewise.
	* c-c++-common/diagnostic-format-sarif-file-4.c: Likewise.
	* c-c++-common/diagnostic-format-sarif-file-Wbidi-chars.c: New
	test case, adapted from Wbidi-chars-1.c.
	* c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-1.c:
	New test case.
	* c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-2.c:
	New test case.
	* c-c++-common/diagnostic-format-sarif-file-bad-utf8-pr109098-3.c:
	New test case, adapted from cpp/Winvalid-utf8-1.c.
	* c-c++-common/diagnostic-format-sarif-file-valid-CP850.c: New
	test case, adapted from gcc.dg/diagnostic-input-charset-1.c.
	* gcc.dg/plugin/crash-test-ice-sarif.c: Add verify-sarif-file
	directive.
	* gcc.dg/plugin/crash-test-write-though-null-sarif.c: Likewise.
	* gcc.dg/plugin/diagnostic-test-paths-5.c: Likewise.
	* lib/scansarif.exp (verify-sarif-file): New procedure.
	* lib/verify-sarif-file.py: New support script.

libcpp/ChangeLog:
	PR analyzer/109098
	* charset.cc (cpp_valid_utf8_p): New function.
	* include/cpplib.h (cpp_valid_utf8_p): New prototype.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>

I've noticed I forgot to update copyright years when updating from
Unicode 15.0.0 (and makeucnid.cc had it hopelessly obsolete).

2023-03-16  Jakub Jelinek  <jakub@redhat.com>

	* makeucnid.cc (write_copyright): Update Unicode copyright years
	up to 2022.
	* makeuname2c.cc (write_copyright): Likewise.
	* ucnid.h: Regenerated.
	* uname2c.h: Regenerated.

get__Pragma_string() in directives.cc is responsible for lexing the parens
and the string argument from a _Pragma("...") operator. This function does
not handle the case when the closing paren is not on the same line as the
string; in that case, libcpp will by default reuse the token buffer it
previously used for the string, so that the string token returned by
get__Pragma_string() may be corrupted, as shown in the testcase. Fix using
the existing keep_tokens mechanism that temporarily disables the reuse of
token buffers.

libcpp/ChangeLog:

	PR preprocessor/67046
	* directives.cc (_cpp_do__Pragma): Increment pfile->keep_tokens to
	ensure the returned string token is valid.

gcc/testsuite/ChangeLog:

	PR preprocessor/67046
	* c-c++-common/cpp/pr67046.c: New test.

Updates cpp_wcwidth() to Unicode 15, following the procedure in
contrib/unicode/README mechanically without incident.

contrib/ChangeLog:

	* unicode/DerivedCoreProperties.txt: Update to Unicode 15.
	* unicode/DerivedNormalizationProps.txt: Likewise.
	* unicode/EastAsianWidth.txt: Likwise.
	* unicode/PropList.txt: Likewise.
	* unicode/README: Likewise.
	* unicode/UnicodeData.txt: Likewise.

libcpp/ChangeLog:

	* generated_cpp_wcwidth.h: Regenerated for Unicode 15.

	* sr.po, sv.po: Update.

gcc/po/
	* be.po, da.po, de.po, el.po, es.po, fi.po, fr.po, hr.po, id.po,
	ja.po, nl.po, ru.po, sr.po, sv.po, tr.po, uk.po, vi.po, zh_CN.po,
	zh_TW.po: Update.

libcpp/po/
	* be.po, ca.po, da.po, de.po, el.po, eo.po, es.po, fi.po, fr.po,
	id.po, ja.po, ka.po, nl.po, pt_BR.po, ro.po, ru.po, sr.po, sv.po,
	tr.po, uk.po, vi.po, zh_CN.po, zh_TW.po: Update.

gcc/po/
	* gcc.pot: Regenerate.

libcpp/po/
	* cpplib.pot: Regenerate.

	* ka.po: New.

	* eo.po: Update.

2022 -> 2023