Skip to content
Snippets Groups Projects
  • Richard Sandiford's avatar
    30213ae9
    vect: Make reduction code handle calls · 30213ae9
    Richard Sandiford authored
    This patch extends the reduction code to handle calls.  So far
    it's a structural change only; a later patch adds support for
    specific function reductions.
    
    Most of the patch consists of using code_helper and gimple_match_op
    to describe the reduction operations.  The other main change is that
    vectorizable_call now needs to handle fully-predicated reductions.
    
    There are some new functions that are provided for ABI completeness
    and aren't currently used:
    
      first_commutative_argument
      commutative_ternary_op_p
      1- and 3-argument forms of gimple_build
    
    gcc/
    	* builtins.h (associated_internal_fn): Declare overload that
    	takes a (combined_cfn, return type) pair.
    	* builtins.c (associated_internal_fn): Split new overload out
    	of original fndecl version.  Also provide an overload that takes
    	a (combined_cfn, return type) pair.
    	* internal-fn.h (commutative_binary_fn_p): Declare.
    	(commutative_ternary_fn_p): Likewise.
    	(associative_binary_fn_p): Likewise.
    	* internal-fn.c (commutative_binary_fn_p, commutative_ternary_fn_p):
    	New functions, split out from...
    	(first_commutative_argument): ...here.
    	(associative_binary_fn_p): New function.
    	* gimple-match.h (code_helper): Add a constructor that takes
    	internal functions.
    	(commutative_binary_op_p): Declare.
    	(commutative_ternary_op_p): Likewise.
    	(first_commutative_argument): Likewise.
    	(associative_binary_op_p): Likewise.
    	(canonicalize_code): Likewise.
    	(directly_supported_p): Likewise.
    	(get_conditional_internal_fn): Likewise.
    	(gimple_build): New overloads that takes a code_helper.
    	* gimple-fold.c (gimple_build): Likewise.
    	* gimple-match-head.c (commutative_binary_op_p): New function.
    	(commutative_ternary_op_p): Likewise.
    	(first_commutative_argument): Likewise.
    	(associative_binary_op_p): Likewise.
    	(canonicalize_code): Likewise.
    	(directly_supported_p): Likewise.
    	(get_conditional_internal_fn): Likewise.
    	* tree-vectorizer.h: Include gimple-match.h.
    	(neutral_op_for_reduction): Take a code_helper instead of a tree_code.
    	(needs_fold_left_reduction_p): Likewise.
    	(reduction_fn_for_scalar_code): Likewise.
    	(vect_can_vectorize_without_simd_p): Declare a nNew overload that takes
    	a code_helper.
    	* tree-vect-loop.c: Include case-cfn-macros.h.
    	(fold_left_reduction_fn): Take a code_helper instead of a tree_code.
    	(reduction_fn_for_scalar_code): Likewise.
    	(neutral_op_for_reduction): Likewise.
    	(needs_fold_left_reduction_p): Likewise.
    	(use_mask_by_cond_expr_p): Likewise.
    	(build_vect_cond_expr): Likewise.
    	(vect_create_partial_epilog): Likewise.  Use gimple_build rather
    	than gimple_build_assign.
    	(check_reduction_path): Handle calls and operate on code_helpers
    	rather than tree_codes.
    	(vect_is_simple_reduction): Likewise.
    	(vect_model_reduction_cost): Likewise.
    	(vect_find_reusable_accumulator): Likewise.
    	(vect_create_epilog_for_reduction): Likewise.
    	(vect_transform_cycle_phi): Likewise.
    	(vectorizable_reduction): Likewise.  Make more use of
    	lane_reduc_code_p.
    	(vect_transform_reduction): Use gimple_extract_op but expect
    	a tree_code for now.
    	(vect_can_vectorize_without_simd_p): New overload that takes
    	a code_helper.
    	* tree-vect-stmts.c (vectorizable_call): Handle reductions in
    	fully-masked loops.
    	* tree-vect-patterns.c (vect_mark_pattern_stmts): Use
    	gimple_extract_op when updating STMT_VINFO_REDUC_IDX.
    30213ae9
    History
    vect: Make reduction code handle calls
    Richard Sandiford authored
    This patch extends the reduction code to handle calls.  So far
    it's a structural change only; a later patch adds support for
    specific function reductions.
    
    Most of the patch consists of using code_helper and gimple_match_op
    to describe the reduction operations.  The other main change is that
    vectorizable_call now needs to handle fully-predicated reductions.
    
    There are some new functions that are provided for ABI completeness
    and aren't currently used:
    
      first_commutative_argument
      commutative_ternary_op_p
      1- and 3-argument forms of gimple_build
    
    gcc/
    	* builtins.h (associated_internal_fn): Declare overload that
    	takes a (combined_cfn, return type) pair.
    	* builtins.c (associated_internal_fn): Split new overload out
    	of original fndecl version.  Also provide an overload that takes
    	a (combined_cfn, return type) pair.
    	* internal-fn.h (commutative_binary_fn_p): Declare.
    	(commutative_ternary_fn_p): Likewise.
    	(associative_binary_fn_p): Likewise.
    	* internal-fn.c (commutative_binary_fn_p, commutative_ternary_fn_p):
    	New functions, split out from...
    	(first_commutative_argument): ...here.
    	(associative_binary_fn_p): New function.
    	* gimple-match.h (code_helper): Add a constructor that takes
    	internal functions.
    	(commutative_binary_op_p): Declare.
    	(commutative_ternary_op_p): Likewise.
    	(first_commutative_argument): Likewise.
    	(associative_binary_op_p): Likewise.
    	(canonicalize_code): Likewise.
    	(directly_supported_p): Likewise.
    	(get_conditional_internal_fn): Likewise.
    	(gimple_build): New overloads that takes a code_helper.
    	* gimple-fold.c (gimple_build): Likewise.
    	* gimple-match-head.c (commutative_binary_op_p): New function.
    	(commutative_ternary_op_p): Likewise.
    	(first_commutative_argument): Likewise.
    	(associative_binary_op_p): Likewise.
    	(canonicalize_code): Likewise.
    	(directly_supported_p): Likewise.
    	(get_conditional_internal_fn): Likewise.
    	* tree-vectorizer.h: Include gimple-match.h.
    	(neutral_op_for_reduction): Take a code_helper instead of a tree_code.
    	(needs_fold_left_reduction_p): Likewise.
    	(reduction_fn_for_scalar_code): Likewise.
    	(vect_can_vectorize_without_simd_p): Declare a nNew overload that takes
    	a code_helper.
    	* tree-vect-loop.c: Include case-cfn-macros.h.
    	(fold_left_reduction_fn): Take a code_helper instead of a tree_code.
    	(reduction_fn_for_scalar_code): Likewise.
    	(neutral_op_for_reduction): Likewise.
    	(needs_fold_left_reduction_p): Likewise.
    	(use_mask_by_cond_expr_p): Likewise.
    	(build_vect_cond_expr): Likewise.
    	(vect_create_partial_epilog): Likewise.  Use gimple_build rather
    	than gimple_build_assign.
    	(check_reduction_path): Handle calls and operate on code_helpers
    	rather than tree_codes.
    	(vect_is_simple_reduction): Likewise.
    	(vect_model_reduction_cost): Likewise.
    	(vect_find_reusable_accumulator): Likewise.
    	(vect_create_epilog_for_reduction): Likewise.
    	(vect_transform_cycle_phi): Likewise.
    	(vectorizable_reduction): Likewise.  Make more use of
    	lane_reduc_code_p.
    	(vect_transform_reduction): Use gimple_extract_op but expect
    	a tree_code for now.
    	(vect_can_vectorize_without_simd_p): New overload that takes
    	a code_helper.
    	* tree-vect-stmts.c (vectorizable_call): Handle reductions in
    	fully-masked loops.
    	* tree-vect-patterns.c (vect_mark_pattern_stmts): Use
    	gimple_extract_op when updating STMT_VINFO_REDUC_IDX.