On Thu, Jun 09, 2022 at 12:11:28PM +0200, Thomas Schwinge wrote:
> > This patch adds support for dlopening libmemkind.so
>
> Instead of 'dlopen'ing literally 'libmemkind.so':
> ..., shouldn't this instead 'dlopen' 'libmemkind.so.0'?  At least for
> Debian/Ubuntu, the latter ('libmemkind.so.0') is shipped in the "library"
> package:

I agree and I've actually noticed it too right before committing, but I thought
I'll investigate and tweak incrementally because "libmemkind.so"
is what I've actually tested (it is what llvm libomp uses).

Here is the now tested incremental fix.

2022-06-10  Jakub Jelinek  <jakub@redhat.com>

	* allocator.c (gomp_init_memkind): Call dlopen with "libmemkind.so.0"
	rather than "libmemkind.so".

openmp: Add support for HBW or large capacity or interleaved memory through the libmemkind.so library

This patch adds support for dlopening libmemkind.so on Linux and uses it
for some kinds of allocations (but not yet e.g. pinned memory).

2022-06-09  Jakub Jelinek  <jakub@redhat.com>

	* allocator.c: Include dlfcn.h if LIBGOMP_USE_MEMKIND is defined.
	(enum gomp_memkind_kind): New type.
	(struct omp_allocator_data): Add memkind field if LIBGOMP_USE_MEMKIND
	is defined.
	(struct gomp_memkind_data): New type.
	(memkind_data, memkind_data_once): New variables.
	(gomp_init_memkind, gomp_get_memkind): New functions.
	(omp_init_allocator): Initialize data.memkind, don't fail for
	omp_high_bw_mem_space if libmemkind supports it.
	(omp_aligned_alloc, omp_free, omp_aligned_calloc, omp_realloc): Add
	memkind support of LIBGOMP_USE_MEMKIND is defined.
	* config/linux/allocator.c: New file.

The variable omp_atv_sequential was replaced by omp_atv_serialized in OpenMP
5.1. This was already implemented by Jakub (C/C++, commit ea82325a) and
Tobias (Fortran, commit fff15bad).

This patch adds two tests to check if omp_atv_serialized is available (one test
for C/C++ and one for Fortran). Besides that omp_atv_sequential is marked as
deprecated in C/C++ and Fortran for OpenMP 5.1.

libgomp/ChangeLog:

	* allocator.c (omp_init_allocator): Replace omp_atv_sequential with
	omp_atv_serialized.
	* omp.h.in: Add deprecated flag for omp_atv_sequential.
	* omp_lib.f90.in: Add deprecated flag for omp_atv_sequential.
	* testsuite/libgomp.c-c++-common/alloc-10.c: New test.
	* testsuite/libgomp.fortran/alloc-12.f90: New test.

This patch adds new OpenMP 5.1 allocator entrypoints and in addition to that
fixes an omp_alloc bug which is hard to test for - if the first allocator
fails but has a larger alignment trait and has a fallback allocator, either
the default behavior or a user fallback, then the extra alignment will be used
even in the fallback allocation, rather than just starting with whatever
alignment has been requested (in GOMP_alloc or the minimum one in omp_alloc).

Jonathan's comment on IRC this morning made me realize that I should add
alloc_align attributes to 2 of the prototypes and I still need to add testsuite
coverage for omp_realloc, will do that in a follow-up.

2021-09-30  Jakub Jelinek  <jakub@redhat.com>

	* omp.h.in (omp_aligned_alloc, omp_calloc, omp_aligned_calloc,
	omp_realloc): New prototypes.
	(omp_alloc): Move after omp_free prototype, add __malloc__ (omp_free)
	attribute.
	* allocator.c: Include string.h.
	(omp_aligned_alloc): No longer static, add ialias.  Add new_alignment
	variable and use it instead of alignment so that when retrying the old
	alignment is used again.  Don't retry if new alignment is the same
	as old alignment, unless allocator had pool size.
	(omp_alloc, GOMP_alloc, GOMP_free): Use ialias_call.
	(omp_aligned_calloc, omp_calloc, omp_realloc): New functions.
	* libgomp.map (OMP_5.0.2): Export omp_aligned_alloc, omp_calloc,
	omp_aligned_calloc and omp_realloc.
	* testsuite/libgomp.c-c++-common/alloc-4.c (main): Add
	omp_aligned_alloc, omp_calloc and omp_aligned_calloc tests.
	* testsuite/libgomp.c-c++-common/alloc-5.c: New test.
	* testsuite/libgomp.c-c++-common/alloc-6.c: New test.
	* testsuite/libgomp.c-c++-common/alloc-7.c: New test.
	* testsuite/libgomp.c-c++-common/alloc-8.c: New test.

For now, task/taskloop constructs aren't handled and C/C++ array reductions
and reductions with task or inscan modifiers need further work.
Instead of calling omp_alloc/omp_free (where the former doesn't have
alignment argument and omp_aligned_alloc is 5.1 only feature), this calls
GOMP_alloc/GOMP_free, so that the library can fail if it would fall back
into NULL (exception is zero length allocations).

2020-11-12  Jakub Jelinek  <jakub@redhat.com>

gcc/
	* builtin-types.def (BT_FN_PTR_SIZE_SIZE_PTRMODE): New function type.
	* omp-builtins.def (BUILT_IN_GOACC_DECLARE): Move earlier.
	(BUILT_IN_GOMP_ALLOC, BUILT_IN_GOMP_FREE): New builtins.
	* gimplify.c (gimplify_scan_omp_clauses): Force allocator into a
	decl if it is not NULL, INTEGER_CST or decl.
	(gimplify_adjust_omp_clauses): Clear GOVD_EXPLICIT on explicit clauses
	which are being removed.  Remove allocate clauses for variables not seen
	if they are private, firstprivate or linear too.  Call
	omp_notice_variable on the allocator otherwise.
	(gimplify_omp_for): Handle iterator vars mentioned in allocate clauses
	similarly to non-is_gimple_reg iterators.
	* omp-low.c (struct omp_context): Add allocate_map field.
	(delete_omp_context): Delete it.
	(scan_sharing_clauses): Fill it from allocate clauses.  Remove it
	if mentioned also in shared clause.
	(lower_private_allocate): New function.
	(lower_rec_input_clauses): Handle allocate clause for privatized
	variables, except for task/taskloop, C/C++ array reductions for now
	and task/inscan variables.
	(lower_send_shared_vars): Don't consider variables in allocate_map
	as shared.
	* omp-expand.c (expand_omp_for_generic, expand_omp_for_static_nochunk,
	expand_omp_for_static_chunk): Use expand_omp_build_assign instead of
	gimple_build_assign + gsi_insert_after.
	* builtins.c (builtin_fnspec): Handle BUILTIN_GOMP_ALLOC and
	BUILTIN_GOMP_FREE.
	* tree-ssa-ccp.c (evaluate_stmt): Handle BUILTIN_GOMP_ALLOC.
	* tree-ssa-dce.c (mark_stmt_if_obviously_necessary): Handle
	BUILTIN_GOMP_ALLOC.
	(mark_all_reaching_defs_necessary_1): Handle BUILTIN_GOMP_ALLOC
	and BUILTIN_GOMP_FREE.
	(propagate_necessity): Likewise.
gcc/fortran/
	* f95-lang.c (ATTR_ALLOC_WARN_UNUSED_RESULT_SIZE_2_NOTHROW_LIST):
	Define.
	(gfc_init_builtin_functions): Add alloc_size and warn_unused_result
	attributes to __builtin_GOMP_alloc.
	* types.def (BT_PTRMODE): New primitive type.
	(BT_FN_VOID_PTR_PTRMODE, BT_FN_PTR_SIZE_SIZE_PTRMODE): New function
	types.
libgomp/
	* libgomp.map (GOMP_alloc, GOMP_free): Export at GOMP_5.0.1.
	* omp.h.in (omp_alloc): Add malloc and alloc_size attributes.
	* libgomp_g.h (GOMP_alloc, GOMP_free): Declare.
	* allocator.c (omp_aligned_alloc): New for now static function,
	add alignment argument and handle it.
	(omp_alloc): Reimplement using omp_aligned_alloc.
	(GOMP_alloc, GOMP_free): New functions.
	(omp_free): Add ialias.
	* testsuite/libgomp.c-c++-common/allocate-1.c: New test.
	* testsuite/libgomp.c++/allocate-1.C: New test.

libgomp/ChangeLog:

	* allocator.c: Add ialias for omp_init_allocator and
	omp_destroy_allocator.
	* configure.ac: Set INTPTR_T_KIND.
	* configure: Regenerate.
	* Makefile.in: Regenerate.
	* testsuite/Makefile.in: Regenerate.
	* fortran.c (omp_init_allocator_, omp_destroy_allocator_,
	omp_set_default_allocator_, omp_get_default_allocator_): New
	functions and ialias_redirect.
	* icv.c: Add ialias for omp_set_default_allocator and
	omp_get_default_allocator.
	* libgomp.map (OMP_5.0.1): Add omp_init_allocator_,
	omp_destroy_allocator_, omp_set_default_allocator_ and
	omp_get_default_allocator_.
	* omp_lib.f90.in: Add allocator traits parameters, declare
	allocator routines and add related kind parameters.
	* omp_lib.h.in: Likewise.
	* testsuite/libgomp.c-c++-common/alloc-2.c: Fix sizeof.
	* testsuite/libgomp.fortran/alloc-1.F90: New test.
	* testsuite/libgomp.fortran/alloc-2.F90: New test.
	* testsuite/libgomp.fortran/alloc-3.F: New test.
	* testsuite/libgomp.fortran/alloc-4.f90: New test.
	* testsuite/libgomp.fortran/alloc-5.f90: New test.

While this is an OpenMP 5.1 change, it is undesirable to let people use different
values and then deal with ABI backwards compatibility in a year or two.

2020-07-09  Jakub Jelinek  <jakub@redhat.com>

	* omp.h.in (omp_alloctrait_value_t): Change omp_atv_default from
	2 to -1.  Add omp_atv_serialized and define omp_atv_sequential using
	it.  Remove __omp_alloctrait_value_max__.
	* allocator.c (omp_init_allocator): Handle omp_atv_default for
	omp_atk_alignment and omp_atk_pool_size.

2020-06-02  Jakub Jelinek  <jakub@redhat.com>

	* allocator.c (omp_free): Fix up build if HAVE_SYNC_BUILTINS is not
	defined.

2020-05-30  Jakub Jelinek  <jakub@redhat.com>

	* allocator.c (omp_alloc): For size == 0, return NULL early.

	* testsuite/libgomp.c-c++-common/alloc-4.c: New test.

This patch adds very basic allocator support (omp_{init,destroy}_allocator,
omp_{alloc,free}, omp_[sg]et_default_allocator).
The plan is to use memkind (likely dlopened) for high bandwidth memory, but
that part isn't implemented yet, probably mlock for pinned memory and see
what other options there are for other kinds of memory.
For offloading targets, we need to decide if we want to support the
dynamic allocators (and on which targets), or if e.g. all we do is at compile
time replace omp_alloc/omp_free calls with constexpr predefined allocators
with something special.

And allocate directive and allocator/uses_allocators clauses are future work
too.

2020-05-19  Jakub Jelinek  <jakub@redhat.com>

	* allocator.c: New file.