libcpp: Implement P2362R3 - Remove non-encodable wide character literals and multicharacter [PR106647]

My understanding of the paper is that we just want to promote the CPP_WCHAR
"character constant too long for its type" warning to error as it is already
error for u8, u and U literals.

2022-08-26  Jakub Jelinek  <jakub@redhat.com>

	PR c++/106647
	* charset.cc (wide_str_to_charconst): Implement P2362R3 - Remove
	non-encodable wide character literals and multicharacter.  For
	C++23 use CPP_DL_ERROR instead of CPP_DL_WARNING for
	"character constant too long for its type" diagnostics on CPP_WCHAR
	literals.

	* g++.dg/cpp23/wchar-multi1.C: New test.
	* g++.dg/cpp23/wchar-multi2.C: New test.

The following patch implements the
C++23 P2071R2 - Named universal character escapes
paper to support \N{LATIN SMALL LETTER E} etc.
I've used Unicode 14.0, there are 144803 character name properties
(including the ones generated by Unicode NR1 and NR2 rules)
and correction/control/alternate aliases, together with zero terminators
that would be 3884745 bytes, which is clearly unacceptable for libcpp.
This patch instead contains a generator which from the UnicodeData.txt
and NameAliases.txt files emits a space optimized radix tree (208765
bytes long for 14.0), a single string literal dictionary (59418 bytes),
maximum name length (currently 88 chars) and two small helper arrays
for the NR1/NR2 name generation.
The radix tree needs 2 to 9 bytes per node, the exact format is
described in the generator program.  There could be ways to shrink
the dictionary size somewhat at the expense of slightly slower lookups.

Currently the patch implements strict matching (that is what is needed
to actually implement it on valid code) and Unicode UAX44-LM2 algorithm
loose matching to provide hints (that algorithm essentially ignores
hyphens in between two alphanumeric characters, spaces and underscores
(with one exception for hyphen) and does case insensitive matching).
In the attachment is a WIP patch that shows how to implement also
spellcheck.{h,cc} style discovery of misspellings, but I'll need to talk
to David Malcolm about it, as spellcheck.{h,cc} is in gcc/ subdir
(so the WIP incremental patch instead prints all the names to stderr).

2022-08-26  Jakub Jelinek  <jakub@redhat.com>

	PR c++/106648
libcpp/
	* charset.cc: Implement C++23 P2071R2 - Named universal character
	escapes.  Include uname2c.h.
	(hangul_syllables, hangul_count): New variables.
	(struct uname2c_data): New type.
	(_cpp_uname2c, _cpp_uname2c_uax44_lm2): New functions.
	(_cpp_valid_ucn): Use them.  Handle named universal character escapes.
	(convert_ucn): Adjust comment.
	(convert_escape): Call convert_ucn even for \N.
	(_cpp_interpret_identifier): Handle named universal character escapes.
	* lex.cc (get_bidi_ucn): Fix up function comment formatting.
	(get_bidi_named): New function.
	(forms_identifier_p, lex_string): Handle named universal character
	escapes.
	* makeuname2c.cc: New file.  Small parts copied from makeucnid.cc.
	* uname2c.h: New generated file.
gcc/c-family/
	* c-cppbuiltin.cc (c_cpp_builtins): Predefine
	__cpp_named_character_escapes to 202207L.
gcc/testsuite/
	* c-c++-common/cpp/named-universal-char-escape-1.c: New test.
	* c-c++-common/cpp/named-universal-char-escape-2.c: New test.
	* c-c++-common/cpp/named-universal-char-escape-3.c: New test.
	* c-c++-common/cpp/named-universal-char-escape-4.c: New test.
	* c-c++-common/Wbidi-chars-25.c: New test.
	* gcc.dg/cpp/named-universal-char-escape-1.c: New test.
	* gcc.dg/cpp/named-universal-char-escape-2.c: New test.
	* g++.dg/cpp/named-universal-char-escape-1.C: New test.
	* g++.dg/cpp/named-universal-char-escape-2.C: New test.
	* g++.dg/cpp23/feat-cxx2b.C: Test __cpp_named_character_escapes.

On Thu, Aug 18, 2022 at 11:02:44PM +0000, Joseph Myers wrote:
> ISO C2x standardizes the existing #warning extension.  Arrange
> accordingly for it not to be diagnosed with -std=c2x -pedantic, but to
> be diagnosed with -Wc11-c2x-compat.

And here is the corresponding C++ version.
Don't pedwarn about this for C++23/GNU++23 and tweak the diagnostics
for C++ otherwise, + testsuite coverage.
The diagnostic wording is similar e.g. to the #elifdef diagnostics.

2022-08-24  Jakub Jelinek  <jakub@redhat.com>

	PR c++/106646
	* init.cc: Implement C++23 P2437R1 - Support for #warning.
	(lang_defaults): Set warning_directive for GNUCXX23 and CXX23.
	* directives.cc (directive_diagnostics): Use different wording of
	#warning pedwarn for C++.

	* g++.dg/cpp/warning-1.C: New test.
	* g++.dg/cpp/warning-2.C: New test.
	* g++.dg/cpp/warning-3.C: New test.

The following patch implements the C++23 P2290R3 paper.

2022-08-20  Jakub Jelinek  <jakub@redhat.com>

	PR c++/106645
libcpp/
	* include/cpplib.h (struct cpp_options): Implement
	P2290R3 - Delimited escape sequences.  Add delimite_escape_seqs
	member.
	* init.cc (struct lang_flags): Likewise.
	(lang_defaults): Add delim column.
	(cpp_set_lang): Copy over delimite_escape_seqs.
	* charset.cc (extend_char_range): New function.
	(_cpp_valid_ucn): Use it.  Handle delimited escape sequences.
	(convert_hex): Likewise.
	(convert_oct): Likewise.
	(convert_ucn): Use extend_char_range.
	(convert_escape): Call convert_oct even for \o.
	(_cpp_interpret_identifier): Handle delimited escape sequences.
	* lex.cc (get_bidi_ucn_1): Likewise.  Add end argument, fill it in.
	(get_bidi_ucn): Adjust get_bidi_ucn_1 caller.  Use end argument to
	compute num_bytes.
gcc/testsuite/
	* c-c++-common/cpp/delimited-escape-seq-1.c: New test.
	* c-c++-common/cpp/delimited-escape-seq-2.c: New test.
	* c-c++-common/cpp/delimited-escape-seq-3.c: New test.
	* c-c++-common/Wbidi-chars-24.c: New test.
	* gcc.dg/cpp/delimited-escape-seq-1.c: New test.
	* gcc.dg/cpp/delimited-escape-seq-2.c: New test.
	* g++.dg/cpp/delimited-escape-seq-1.C: New test.
	* g++.dg/cpp/delimited-escape-seq-2.C: New test.

ISO C2x standardizes the existing #warning extension.  Arrange
accordingly for it not to be diagnosed with -std=c2x -pedantic, but to
be diagnosed with -Wc11-c2x-compat.

Bootstrapped with no regressions for x86_64-pc-linux-gnu.

gcc/testsuite/
	* gcc.dg/cpp/c11-warning-1.c, gcc.dg/cpp/c11-warning-2.c,
	gcc.dg/cpp/c11-warning-3.c, gcc.dg/cpp/c11-warning-4.c,
	gcc.dg/cpp/c2x-warning-1.c, gcc.dg/cpp/c2x-warning-2.c,
	gcc.dg/cpp/gnu11-warning-1.c, gcc.dg/cpp/gnu11-warning-2.c,
	gcc.dg/cpp/gnu11-warning-3.c, gcc.dg/cpp/gnu11-warning-4.c,
	gcc.dg/cpp/gnu2x-warning-1.c, gcc.dg/cpp/gnu2x-warning-2.c: New
	tests.

libcpp/
	* include/cpplib.h (struct cpp_options): Add warning_directive.
	* init.cc (struct lang_flags, lang_defaults): Add
	warning_directive.
	* directives.cc (DIRECTIVE_TABLE): Mark #warning as STDC2X not
	EXTENSION.
	(directive_diagnostics): Diagnose #warning with -Wc11-c2x-compat,
	or with -pedantic for a standard not supporting #warning.

Gcc's '#pragma GCC diagnostic' directives are processed in "early mode"
(see handle_pragma_diagnostic_early) for the C++ frontend and, as such,
require that the target diagnostic option be enabled for the preprocessor
(see c_option_is_from_cpp_diagnostics).  This change modifies the
-Wc++20-compat option definition to register it as a preprocessor option
so that its associated diagnostics can be suppressed.  The changes also
implicitly disable the option in C++20 and later modes.  These changes
are consistent with the definition of the -Wc++11-compat option.

This support is motivated by the need to suppress the following diagnostic
otherwise issued in C++17 and earlier modes due to the char8_t typedef
present in the uchar.h header file in glibc 2.36.
  warning: identifier ‘char8_t’ is a keyword in C++20 [-Wc++20-compat]

Tests are added to validate suppression of both -Wc++11-compat and
-Wc++20-compat related diagnostics (fixes were only needed for the C++20
case).

	PR c++/106423

gcc/c-family/ChangeLog:
	* c-opts.cc (c_common_post_options): Disable -Wc++20-compat
	diagnostics in C++20 and later.
	* c.opt (Wc++20-compat): Enable hooks for the preprocessor.

gcc/cp/ChangeLog:
	* parser.cc (cp_lexer_saving_tokens): Add comment regarding
	diagnostic requirements.

gcc/testsuite/ChangeLog:
	* g++.dg/cpp0x/keywords2.C: New test.
	* g++.dg/cpp2a/keywords2.C: New test.

libcpp/ChangeLog:
	* include/cpplib.h (cpp_warning_reason): Add CPP_W_CXX20_COMPAT.
	* init.cc (cpp_create_reader): Add cpp_warn_cxx20_compat.

This patch corrects handling of UTF-8 character literals in preprocessing
directives so that they are treated as unsigned types in char8_t enabled
C++ modes (C++17 with -fchar8_t or C++20 without -fno-char8_t). Previously,
UTF-8 character literals were always treated as having the same type as
ordinary character literals (signed or unsigned dependent on target or use
of the -fsigned-char or -funsigned char options).

	PR preprocessor/106426

gcc/c-family/ChangeLog:
	* c-opts.cc (c_common_post_options): Assign cpp_opts->unsigned_utf8char
	subject to -fchar8_t, -fsigned-char, and/or -funsigned-char.

gcc/testsuite/ChangeLog:
	* g++.dg/ext/char8_t-char-literal-1.C: Check signedness of u8 literals.
	* g++.dg/ext/char8_t-char-literal-2.C: Check signedness of u8 literals.

libcpp/ChangeLog:
	* charset.cc (narrow_str_to_charconst): Set signedness of CPP_UTF8CHAR
	literals based on unsigned_utf8char.
	* include/cpplib.h (cpp_options): Add unsigned_utf8char.
	* init.cc (cpp_create_reader): Initialize unsigned_utf8char.

This adjusts the API of label_text so that the data members are private
and cannot be modified by callers.  Add accessors for them instead, and
make the accessors const-correct.  Also rename moved_from () to the more
idiomatic release ().  Also remove the unused take_or_copy () member
function which has confusing ownership semantics.

gcc/analyzer/ChangeLog:

	* call-info.cc (call_info::print): Adjust to new label_text API.
	* checker-path.cc (checker_event::dump): Likewise.
	(region_creation_event::get_desc): Likewise.
	(state_change_event::get_desc): Likewise.
	(superedge_event::should_filter_p): Likewise.
	(start_cfg_edge_event::get_desc): Likewise.
	(call_event::get_desc): Likewise.
	(return_event::get_desc): Likewise.
	(warning_event::get_desc): Likewise.
	(checker_path::dump): Likewise.
	(checker_path::debug): Likewise.
	* diagnostic-manager.cc (diagnostic_manager::prune_for_sm_diagnostic):
	Likewise.
	(diagnostic_manager::prune_interproc_events): Likewise.
	* engine.cc (feasibility_state::maybe_update_for_edge):
	Likewise.
	* program-state.cc (sm_state_map::to_json): Likewise.
	* region-model-impl-calls.cc (region_model::impl_call_analyzer_describe): Likewise.
	(region_model::impl_call_analyzer_dump_capacity): Likewise.
	* region.cc (region::to_json): Likewise.
	* sm-malloc.cc (inform_nonnull_attribute): Likewise.
	* store.cc (binding_map::to_json): Likewise.
	(store::to_json): Likewise.
	* supergraph.cc (superedge::dump): Likewise.
	* svalue.cc (svalue::to_json): Likewise.

gcc/c-family/ChangeLog:

	* c-format.cc (class range_label_for_format_type_mismatch):
	Adjust to new label_text API.

gcc/ChangeLog:

	* diagnostic-format-json.cc (json_from_location_range): Adjust
	to new label_text API.
	* diagnostic-format-sarif.cc (sarif_builder::make_location_object):
	Likewise.
	* diagnostic-show-locus.cc (struct pod_label_text): Likewise.
	(layout::print_any_labels): Likewise.
	* tree-diagnostic-path.cc (class path_label): Likewise.
	(struct event_range): Likewise.
	(default_tree_diagnostic_path_printer): Likewise.
	(default_tree_make_json_for_path): Likewise.

libcpp/ChangeLog:

	* include/line-map.h (label_text::take_or_copy): Remove.
	(label_text::moved_from): Rename to release.
	(label_text::m_buffer, label_text::m_owned): Make private.
	(label_text::get, label_text::is_owned): New accessors.

std::move in a return statement can prevent the NRVO:
<https://developers.redhat.com/blog/2019/04/12/understanding-when-not-to-stdmove-in-c>

PR106272 reports that we have two such cases in class label_text's
member functions.  We have -Wpessimizing-move that's supposed to detect
problematic std::move uses, but in this case it didn't trigger.  I've filed
PR106276 to track that.

	PR preprocessor/106272

libcpp/ChangeLog:

	* include/line-map.h (class label_text): Don't std::move in a return
	statement.

The handling of #pragma GCC diagnostic uses input_location, which is not always
as precise as needed; in particular the relative location of some tokens and a
_Pragma directive will crucially determine whether a given diagnostic is enabled
or suppressed in the desired way. PR97498 shows how the C frontend ends up with
input_location pointing to the beginning of the line containing a _Pragma()
directive, resulting in the wrong behavior if the diagnostic to be modified
pertains to some tokens found earlier on the same line. This patch fixes that by
addressing two issues:

    a) libcpp was not assigning a valid location to the CPP_PRAGMA token
    generated by the _Pragma directive.
    b) C frontend was not setting input_location to something reasonable.

With this change, the C frontend is able to change input_location to point to
the _Pragma token as needed.

This is just a two-line fix (one for each of a) and b)), the testsuite changes
were needed only because the location on the tested warnings has been somewhat
improved, so the tests need to look for the new locations.

gcc/c/ChangeLog:

	PR preprocessor/97498
	* c-parser.cc (c_parser_pragma): Set input_location to the
	location of the pragma, rather than the start of the line.

libcpp/ChangeLog:

	PR preprocessor/97498
	* directives.cc (destringize_and_run): Override the location of
	the CPP_PRAGMA token from a _Pragma directive to the location of
	the expansion point, as is done for the tokens lexed from it.

gcc/testsuite/ChangeLog:

	PR preprocessor/97498
	* c-c++-common/pr97498.c: New test.
	* c-c++-common/gomp/pragma-3.c: Adapt for improved warning locations.
	* c-c++-common/gomp/pragma-5.c: Likewise.
	* gcc.dg/pragma-message.c: Likewise.

libgomp/ChangeLog:

	* testsuite/libgomp.oacc-c-c++-common/reduction-5.c: Adapt for
	improved warning locations.
	* testsuite/libgomp.oacc-c-c++-common/vred2d-128.c: Likewise.

libcpp's class label_text stores a char * for a string and a flag saying
whether it owns the buffer.  I added this class before we could use
C++11, and so to avoid lots of copying it required an explicit call
to label_text::maybe_free to potentially free the buffer.

Now that we can use C++11, this patch removes label_text::maybe_free in
favor of doing the cleanup in the destructor, and using C++ move
semantics to avoid any copying.  This allows lots of messy cleanup code
to be eliminated in favor of implicit destruction (mostly in the
analyzer).

No functional change intended.

gcc/analyzer/ChangeLog:
	* call-info.cc (call_info::print): Update for removal of
	label_text::maybe_free in favor of automatic memory management.
	* checker-path.cc (checker_event::dump): Likewise.
	(checker_event::prepare_for_emission): Likewise.
	(state_change_event::get_desc): Likewise.
	(superedge_event::should_filter_p): Likewise.
	(start_cfg_edge_event::get_desc): Likewise.
	(warning_event::get_desc): Likewise.
	(checker_path::dump): Likewise.
	(checker_path::debug): Likewise.
	* diagnostic-manager.cc
	(diagnostic_manager::prune_for_sm_diagnostic): Likewise.
	(diagnostic_manager::prune_interproc_events): Likewise.
	* program-state.cc (sm_state_map::to_json): Likewise.
	* region.cc (region::to_json): Likewise.
	* sm-malloc.cc (inform_nonnull_attribute): Likewise.
	* store.cc (binding_map::to_json): Likewise.
	(store::to_json): Likewise.
	* svalue.cc (svalue::to_json): Likewise.

gcc/c-family/ChangeLog:
	* c-format.cc (range_label_for_format_type_mismatch::get_text):
	Update for removal of label_text::maybe_free in favor of automatic
	memory management.

gcc/ChangeLog:
	* diagnostic-format-json.cc (json_from_location_range): Update for
	removal of label_text::maybe_free in favor of automatic memory
	management.
	* diagnostic-format-sarif.cc
	(sarif_builder::make_location_object): Likewise.
	* diagnostic-show-locus.cc (struct pod_label_text): New.
	(class line_label): Convert m_text from label_text to pod_label_text.
	(layout::print_any_labels): Move "text" to the line_label.
	* tree-diagnostic-path.cc (path_label::get_text): Update for
	removal of label_text::maybe_free in favor of automatic memory
	management.
	(event_range::print): Likewise.
	(default_tree_diagnostic_path_printer): Likewise.
	(default_tree_make_json_for_path): Likewise.

libcpp/ChangeLog:
	* include/line-map.h: Include <utility>.
	(class label_text): Delete maybe_free method in favor of a
	destructor.  Add move ctor and assignment operator.  Add deletion
	of the copy ctor and copy-assignment operator.  Rename field
	m_caller_owned to m_owned.  Add std::move where necessary; add
	moved_from member function.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>

This patch updates ucnid.h from Unicode 13 to Unicode 14.  Additionally, the
procedure detailed in contrib/unicode/README, which updates
generated_wcwidth.h, has been expanded with instructions for updating this
file as well, so that both may be done at the same time conveniently.  Two
additional Unicode data files which are needed to create ucnid.h are also
added to source control in contrib/unicode.

contrib/ChangeLog:

	* unicode/README: Added instructions for updating ucnid.h.
	* unicode/DerivedCoreProperties.txt: New file added to source
	control from Unicode 14.0 release.
	* unicode/DerivedNormalizationProps.txt: Likewise.

libcpp/ChangeLog:

	* ucnid.h: Regenerated for Unicode 14.0.

The procedure detailed in contrib/unicode/README was followed with nothing
notable coming up. The glibc scripts did not require any update, so the
only change was retrieving new versions of the Unicode data files and
rerunning gen_wcwidth.py.

contrib/ChangeLog:

	* unicode/EastAsianWidth.txt: Update to Unicode 14.0.0.
	* unicode/PropList.txt: Likewise.
	* unicode/README: Likewise.
	* unicode/UnicodeData.txt: Likewise.

libcpp/ChangeLog:

	* generated_cpp_wcwidth.h: Generated from updated Unicode data files.

The first part of the following testcase (m1-m3 macros and its use)
regressed with my PR89971 fix, but as the m1,m4-m5 and its use part shows,
the problem isn't new, we can emit a CPP_PADDING token to avoid it from
being adjacent to whatever comes after the __VA_OPT__ (in this case there
is nothing afterwards, true).

In most cases these CPP_PADDING tokens don't matter, all other
callers of cpp_get_token_with_location either ignore CPP_PADDING tokens
completely (e.g. c_lex_with_flags) or they just remember them and
take them into account when printing stuff whether there should be
added whitespace or not (scan_translation_unit + token_streamer::stream).
So, I think we should just ignore CPP_PADDING tokens the same way in
_cpp_parse_expr.

2022-05-27  Jakub Jelinek  <jakub@redhat.com>

	PR preprocessor/105732
	* expr.cc (_cpp_parse_expr): Handle CPP_PADDING by just another
	token.

	* c-c++-common/cpp/va-opt-10.c: New test.

As of GCC 11 onwards we have required a C++11 compiler, such as GCC 4.8
or later.  On the assumption that any such compiler correctly implements
"final" and "override", this patch updates the source tree to stop using
the FINAL and OVERRIDE macros from ansidecl.h, in favor of simply using
"final" and "override" directly.

libcpp/ChangeLog:
	* lex.cc: Replace uses of "FINAL" and "OVERRIDE" with "final" and
	"override".

gcc/analyzer/ChangeLog:
	* analyzer-pass.cc: Replace uses of "FINAL" and "OVERRIDE" with
	"final" and "override".
	* call-info.h: Likewise.
	* checker-path.h: Likewise.
	* constraint-manager.cc: Likewise.
	* diagnostic-manager.cc: Likewise.
	* engine.cc: Likewise.
	* exploded-graph.h: Likewise.
	* feasible-graph.h: Likewise.
	* pending-diagnostic.h: Likewise.
	* region-model-impl-calls.cc: Likewise.
	* region-model.cc: Likewise.
	* region-model.h: Likewise.
	* region.h: Likewise.
	* sm-file.cc: Likewise.
	* sm-malloc.cc: Likewise.
	* sm-pattern-test.cc: Likewise.
	* sm-sensitive.cc: Likewise.
	* sm-signal.cc: Likewise.
	* sm-taint.cc: Likewise.
	* state-purge.h: Likewise.
	* store.cc: Likewise.
	* store.h: Likewise.
	* supergraph.h: Likewise.
	* svalue.h: Likewise.
	* trimmed-graph.h: Likewise.
	* varargs.cc: Likewise.

gcc/c-family/ChangeLog:
	* c-format.cc: Replace uses of "FINAL" and "OVERRIDE" with "final"
	and "override".
	* c-pretty-print.h: Likewise.

gcc/cp/ChangeLog:
	* cxx-pretty-print.h: Replace uses of "FINAL" and "OVERRIDE" with
	"final" and "override".
	* error.cc: Likewise.

gcc/jit/ChangeLog:
	* jit-playback.h: Replace uses of "FINAL" and "OVERRIDE" with
	"final" and "override".
	* jit-recording.cc: Likewise.
	* jit-recording.h: Likewise.

gcc/ChangeLog:
	* config/aarch64/aarch64-sve-builtins-base.cc: Replace uses of
	"FINAL" and "OVERRIDE" with "final" and "override".
	* config/aarch64/aarch64-sve-builtins-functions.h: Likewise.
	* config/aarch64/aarch64-sve-builtins-shapes.cc: Likewise.
	* config/aarch64/aarch64-sve-builtins-sve2.cc: Likewise.
	* diagnostic-path.h: Likewise.
	* digraph.cc: Likewise.
	* gcc-rich-location.h: Likewise.
	* gimple-array-bounds.cc: Likewise.
	* gimple-loop-versioning.cc: Likewise.
	* gimple-range-cache.cc: Likewise.
	* gimple-range-cache.h: Likewise.
	* gimple-range-fold.cc: Likewise.
	* gimple-range-fold.h: Likewise.
	* gimple-range-tests.cc: Likewise.
	* gimple-range.h: Likewise.
	* gimple-ssa-evrp.cc: Likewise.
	* input.cc: Likewise.
	* json.h: Likewise.
	* read-rtl-function.cc: Likewise.
	* tree-complex.cc: Likewise.
	* tree-diagnostic-path.cc: Likewise.
	* tree-ssa-ccp.cc: Likewise.
	* tree-ssa-copy.cc: Likewise.
	* tree-vrp.cc: Likewise.
	* value-query.h: Likewise.
	* vr-values.h: Likewise.

Signed-off-by: David Malcolm <dmalcolm@redhat.com>

	* es.po: Update.

gcc/po/
	* be.po, da.po, de.po, el.po, es.po, fi.po, fr.po, hr.po, id.po,
	ja.po, nl.po, ru.po, sr.po, sv.po, tr.po, uk.po, vi.po, zh_CN.po,
	zh_TW.po: Update.

libcpp/po/
	* be.po, ca.po, da.po, de.po, el.po, eo.po, es.po, fi.po, fr.po,
	id.po, ja.po, nl.po, pt_BR.po, ru.po, sr.po, sv.po, tr.po, uk.po,
	vi.po, zh_CN.po, zh_TW.po: Update.

gcc/po/
	* gcc.pot: Regenerate.

libcpp/po/
	* cpplib.pot: Regenerate.

The logic in libcpp/Makefile.in listing diagnostic functions in a call
to xgettext was missing cpp_warning_at, cpp_pedwarning_at and
cpp_error_at, so resulting in some messages not being extracted for
translation; add those functions to those for which messages are
extracted.

Tested with "make cpplib.pot".

	* Makefile.in (po/$(PACKAGE).pot): Also handle cpp_warning_at,
	cpp_pedwarning_at and cpp_error_at.

As mentioned in the PR, in some cases we preprocess incorrectly when we
encounter an identifier which is defined as function-like macro, followed
by at least 2 CPP_PADDING tokens and then some other identifier.
On the following testcase, the problem is in the 3rd funlike_invocation_p,
the tokens are CPP_NAME Y, CPP_PADDING (the pfile->avoid_paste shared token),
CPP_PADDING (one created with padding_token, val.source is non-NULL and
val.source->flags & PREV_WHITE is non-zero) and then another CPP_NAME.
funlike_invocation_p remembers there was a padding token, but remembers the
first one because of its condition, then the next token is the CPP_NAME,
which is not CPP_OPEN_PAREN, so the CPP_NAME token is backed up, but as we
can't easily backup more tokens, it pushes into a new context the padding
token (the pfile->avoid_paste one).  The net effect is that when Y is not
defined as fun-like macro, we read Y, avoid_paste, padding_token, Y,
while if Y is fun-like macro, we read Y, avoid_paste, avoid_paste, Y
(the second avoid_paste is because that is how we handle end of a context).
Now, for stringify_arg that is unfortunately a significant difference,
which handles CPP_PADDING tokens with:
      if (token->type == CPP_PADDING)
        {
          if (source == NULL
              || (!(source->flags & PREV_WHITE)
                  && token->val.source == NULL))
            source = token->val.source;
          continue;
        }
and later on
      /* Leading white space?  */
      if (dest - 1 != BUFF_FRONT (pfile->u_buff))
        {
          if (source == NULL)
            source = token;
          if (source->flags & PREV_WHITE)
            *dest++ = ' ';
        }
      source = NULL;
(and c-ppoutput.cc has similar code).
So, when Y is not fun-like macro, ' ' is added because padding_token's
val.source->flags & PREV_WHITE is non-zero, while when it is fun-like
macro, we don't add ' ' in between, because source is NULL and so
used from the next token (CPP_NAME Y), which doesn't have PREV_WHITE set.

Now, the funlike_invocation_p condition
       if (padding == NULL
           || (!(padding->flags & PREV_WHITE) && token->val.source == NULL))
        padding = token;
looks very similar to that in stringify_arg/c-ppoutput.cc, so I assume
the intent was to prefer do the same thing and pick the right padding.
But there are significant differences.  Both stringify_arg and c-ppoutput.cc
don't remember the CPP_PADDING token, but its val.source instead, while
in funlike_invocation_p we want to remember the padding token that has the
significant information for stringify_arg/c-ppoutput.cc.
So, IMHO we want to overwrite padding if:
1) padding == NULL (remember that there was any padding at all)
2) padding->val.source == NULL (this matches the source == NULL
   case in stringify_arg)
3) !(padding->val.source->flags & PREV_WHITE) && token->val.source == NULL
   (this matches the !(source->flags & PREV_WHITE) && token->val.source == NULL
   case in stringify_arg)

2022-02-01  Jakub Jelinek  <jakub@redhat.com>

	PR preprocessor/104147
	* macro.cc (funlike_invocation_p): For padding prefer a token
	with val.source non-NULL especially if it has PREV_WHITE set
	on val.source->flags.  Add gcc_assert that CPP_PADDING tokens
	don't have PREV_WHITE set in flags.

	* c-c++-common/cpp/pr104147.c: New test.

The funlike_invocation_p macro never triggered, the other
asserts did on some tests, see below for a full list.
This seems to be caused by #pragma/_Pragma handling.
do_pragma does:
          pfile->directive_result.src_loc = pragma_token_virt_loc;
          pfile->directive_result.type = CPP_PRAGMA;
          pfile->directive_result.flags = pragma_token->flags;
          pfile->directive_result.val.pragma = p->u.ident;
when it sees a pragma, while start_directive does:
  pfile->directive_result.type = CPP_PADDING;
and so does _cpp_do__Pragma.
Now, for #pragma lex.cc will just ignore directive_result if
it has CPP_PADDING type:
              if (_cpp_handle_directive (pfile, result->flags & PREV_WHITE))
                {
                  if (pfile->directive_result.type == CPP_PADDING)
                    continue;
                  result = &pfile->directive_result;
                }
but destringize_and_run does not:
  if (pfile->directive_result.type == CPP_PRAGMA)
    {
...
    }
  else
    {
      count = 1;
      toks = XNEW (cpp_token);
      toks[0] = pfile->directive_result;
and from there it will copy type member of CPP_PADDING, but all the
other members from the last CPP_PRAGMA before it.
Small testcase for it with no option (at least no -fopenmp or -fopenmp-simd).
 #pragma GCC push_options
 #pragma GCC ignored "-Wformat"
 #pragma GCC pop_options
void
foo ()
{
  _Pragma ("omp simd")
  for (int i = 0; i < 64; i++)
    ;
}

Here is a patch that replaces those
      toks = XNEW (cpp_token);
      toks[0] = pfile->directive_result;
lines with
      toks = &pfile->avoid_paste;

2022-02-01  Jakub Jelinek  <jakub@redhat.com>

	* directives.cc (destringize_and_run): Push &pfile->avoid_paste
	instead of a copy of pfile->directive_result for the CPP_PADDING
	case.

Stephan Bergmann reported that our -Wbidi-chars breaks the build
of LibreOffice because we warn about UCNs even when their usage
is correct: LibreOffice constructs strings piecewise, as in:

  aText = u"\u202D" + aText;

and warning about that is overzealous.  Since no editor (AFAIK)
interprets UCNs to show them as Unicode characters, there's less
risk in misinterpreting them, and so perhaps we shouldn't warn
about them by default.  However, identifiers containing UCNs or
programs generating other programs could still cause confusion,
so I'm keeping the UCN checking.  To turn it on, you just need
to use -Wbidi-chars=unpaired,ucn or -Wbidi-chars=any,ucn.

The implementation is done by using the new EnumSet feature.

	PR preprocessor/104030

gcc/c-family/ChangeLog:

	* c.opt (Wbidi-chars): Mark as EnumSet.  Also accept =ucn.

gcc/ChangeLog:

	* doc/invoke.texi: Update documentation for -Wbidi-chars.

libcpp/ChangeLog:

	* include/cpplib.h (enum cpp_bidirectional_level): Add
	bidirectional_ucn.  Set values explicitly.
	* internal.h (cpp_reader): Adjust warn_bidi_p.
	* lex.cc (maybe_warn_bidi_on_close): Don't warn about UCNs
	unless UCN checking is on.
	(maybe_warn_bidi_on_char): Likewise.

gcc/testsuite/ChangeLog:

	* c-c++-common/Wbidi-chars-10.c: Turn on UCN checking.
	* c-c++-common/Wbidi-chars-11.c: Likewise.
	* c-c++-common/Wbidi-chars-14.c: Likewise.
	* c-c++-common/Wbidi-chars-16.c: Likewise.
	* c-c++-common/Wbidi-chars-17.c: Likewise.
	* c-c++-common/Wbidi-chars-4.c: Likewise.
	* c-c++-common/Wbidi-chars-5.c: Likewise.
	* c-c++-common/Wbidi-chars-6.c: Likewise.
	* c-c++-common/Wbidi-chars-7.c: Likewise.
	* c-c++-common/Wbidi-chars-8.c: Likewise.
	* c-c++-common/Wbidi-chars-9.c: Likewise.
	* c-c++-common/Wbidi-chars-ranges.c: Likewise.
	* c-c++-common/Wbidi-chars-18.c: New test.
	* c-c++-common/Wbidi-chars-19.c: New test.
	* c-c++-common/Wbidi-chars-20.c: New test.
	* c-c++-common/Wbidi-chars-21.c: New test.
	* c-c++-common/Wbidi-chars-22.c: New test.
	* c-c++-common/Wbidi-chars-23.c: New test.