powerpc: Fix up *branch_anddi3_dot for -m32 -mpowerpc64 [PR109566]

The following testcase reduced from newlib ICEs on powerpc-linux,
with -O2 -m32 -mpowerpc64 since r12-6433 PR102239 optimization was
added and on the original testcase since some ranger improvements in
GCC 13 made it no longer latent on newlib.
The problem is that the *branch_anddi3_dot define_insn_and_split
relies on the *rotldi3_mask_dot define_insn_and_split being recognized
during splitting.  The rs6000_is_valid_rotate_dot_mask function checks whether
the mask is a CONST_INT which is a valid mask, but *rotl<mode>3_mask_dot in
addition to checking that it is a valid mask also has
  (<MODE>mode == Pmode || UINTVAL (operands[3]) <= 0x7fffffff)
test in the condition.  For TARGET_64BIT that doesn't add any further
requirements, but for !TARGET_64BIT && TARGET_POWERPC64 if the AND
second operand is larger than INT_MAX it will not be recognized.

The rs6000_is_valid_rotate_dot_mask function is used solely in one spot,
condition of *branch_anddi3_dot, so the following patch adjusts it
to check for that as well.

2023-04-25  Jakub Jelinek  <jakub@redhat.com>

	PR target/109566
	* config/rs6000/rs6000.cc (rs6000_is_valid_rotate_dot_mask): For
	!TARGET_64BIT, don't return true if UINTVAL (mask) << (63 - nb)
	is larger than signed int maximum.

	* gcc.target/powerpc/pr109566.c: New test.