From e0ace4c2f6398f2ee653bc0eb9c95423027efb3d Mon Sep 17 00:00:00 2001 From: Ashesh Sharma Date: Wed, 20 May 2026 17:57:42 -0500 Subject: [PATCH 1/6] Add details surrounding IPC cache --- systems/frontier_user_guide.rst | 48 +++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/systems/frontier_user_guide.rst b/systems/frontier_user_guide.rst index b47bb2d6..ffd718f7 100644 --- a/systems/frontier_user_guide.rst +++ b/systems/frontier_user_guide.rst @@ -670,10 +670,15 @@ To use GPU-aware Cray MPICH with Frontier's PrgEnv modules, users must set the f module load craype-accel-amd-gfx90a module load rocm - export MPICH_GPU_SUPPORT_ENABLED=1 - + export MPICH_GPU_SUPPORT_ENABLED=1 + export MPICH_GPU_IPC_ENABLED=1 # enabled by default if MPICH_GPU_SUPPORT_ENABLED=1 + export GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1 + export HSA_ENABLE_IPC_MODE_LEGACY=1 # Cray MPICH 9.1.0 onwards + export HSA_DISABLE_IPC_MODE_LEGACY=0 # Cray MPICH 9.0.1 and before .. note:: + + For known issues with IPC enabled and suggested workarounds, please see :ref:`gpu-ipc-tips-tricks`. There are extra steps needed to enable GPU-aware MPI on Frontier, which depend on the compiler that is used (see 1. and 2. below). @@ -4015,6 +4020,21 @@ If it is necessary to have bit-wise reproducible results from these libraries, i ------ +.. _gpu-ipc-tips-tricks: + +GPU inter-process communication (IPC) considerations +---------------------------------------------------- + +Cray MPICH uses IPC by default for intra-node inter-process MPI data movement operations that involve GPU buffers, i.e., enabling GPU-aware MPI (``MPICH_GPU_SUPPORT_ENABLED=1``) also sets ``MPICH_GPU_IPC_ENABLED=1``. Disabling IPC is known to noticeably impact intra-node MPI performance. + +Prior to Cray MPICH 8.1.33, a memory leak was observed in edge cases involving repeated creation and freeing of GPU-buffers used in MPI communication. This issue was prevelant when GPU buffers continuously increased in size or occupied new memory regions (e.g., due to defragmentation). The root cause was traced to IPC cache handles not being properly released. AMD addressed this by introducing an API that allows remote processes to free IPC cache handles. For Cray MPICH 9.0.1 and prior, the updated IPC mechanism is enabled by setting ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1`` and ``HSA_DISABLE_IPC_MODE_LEGACY=0``. Starting with Cray MPICH 9.1.0, the latter variable was renamed to ``HSA_ENABLE_IPC_MODE_LEGACY=1`` for improved clarity. + +A separate issue remains where enabling IPC can still lead to memory leaks for message sizes less than or equal to 1 MB. A workaround is to set ``HSA_ENABLE_IPC_MODE_LEGACY=0`` and ``GTL_HSA_VSMSG_CUTOFF_SIZE=0``. When message sizes fall at or below ``GTL_HSA_VSMSG_CUTOFF_SIZE``, the GPU Transport Layer (GTL) uses a blocking memory copy driven by the CPU instead of an HSA memcpy. For context, disabling legacy IPC mode (``HSA_ENABLE_IPC_MODE_LEGACY=0``) switches to AMD's newer IPC implementation based on the Linux dma_buf mechanism. + +Another enhancement in GPU-Aware MPI that relates to IPC, has to do with multiple messages in flight. Prior to Cray MPICH 8.1.33, if the maximum messages in flight at any given time exceeded ``MPICH_GPU_IPC_CACHE_MAX_SIZE``, the program would abort immediately. This has been resolved, and IPC cache handles are evicted as the number of messages in flight increase beyond ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. However, a new issue exists where memory leaks can occur if GPU buffers are repeatedly created and destroyed while the number of messages in flight exceed ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. To mitigate this, it is recommended to set ``MPICH_GPU_IPC_CACHE_MAX_SIZE`` to a value larger than the maximum anticipated messages in flight. The default value is ``50``. If setting ``MPICH_GPU_IPC_CACHE_MAX_SIZE > 50``, the following parameters should be adjusted accordingly to maintain a decreasing order of magnitude, i.e., ``GTL_MAX_SIG_HANDLER_LEN`` > ``GTL_IPC_SIGNAL_POOL_TABLE_LEN`` >= ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. + +Environment variables of primary interest to GPU-aware MPI have been described in :ref:`mpi-tuning`. For a detailed list, please refer to the `Cray MPICH User Guide `_. + .. _mpi-tuning: Useful MPI Environment Variables @@ -4059,6 +4079,30 @@ Using GPU-aware MPI is highly recommended on Frontier because the HPE Slingshot There is a small but measurable latency impact for enabling GPU-aware MPI with CPU buffers. Applications that do not use GPU buffers in MPI calls may want to leave this variable unset or set to ``0``, especially if they are sensitive to small message latency. +``MPICH_GPU_IPC_ENABLED`` +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Set to ``1`` by default if ``MPICH_GPU_SUPPORT_ENABLED=1``. This enables GPU IPC support for intra-node GPU-GPU communication operations. This variable has no effect if MPICH_GPU_SUPPORT_ENABLED is set to 0. Disabling IPC is known to noticeably impact intra-node MPI performance. For known issues with IPC enabled and suggested workarounds, please see :ref:`gpu-ipc-tips-tricks`. + +``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This option is unset by default, and is advised to be set to ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1`` when ``MPICH_GPU_IPC_ENABLED=1``. + +Registering and unregistering GPU buffers for inter-process communication takes a relatively long time to do. The MPI library caches remote buffer attachments in order to reuse them, instead of detaching after every transfer. One limitation has been that when a process stops using a GPU buffer, there was no easy way to let the processes that are caching attachments know about this, and so they continued to cache the buffer. This consumes some of the limited GPU memory. A new feature since ROCM 6.4 enables the MPI library to alert the other processes through a signalling mechanism. If this environment variable is set, then this feature will be used to help reduce the amount of memory that the cache is consuming. This may help prevent out of memory situations. When enabling this feature, we do not recommend using the memhooks ``FI_MR_CACHE_MONITOR``, as it has been shown to hang in certain situations. + +``HSA_ENABLE_IPC_MODE_LEGACY`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This specifies AMD's IPC implementation, and should be set to ``1`` when ``MPICH_GPU_IPC_ENABLED=1``. Cray MPICH 9.0.1 and before should use ``HSA_DISABLE_IPC_MODE_LEGACY=0``. + +AMD has a newer version that uses the Linux dma-buf subsytem which can be accessed by setting ``HSA_ENABLE_IPC_MODE_LEGACY=0``. However, the newer version is not natively supported by Cray MPICH. For workaround with ``HSA_ENABLE_IPC_MODE_LEGACY=0``, please see :ref:`gpu-ipc-tips-tricks`. + +``MPICH_GPU_IPC_CACHE_MAX_SIZE`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Specifies the maximum number of GPU IPC cache handles a process can retain in its IPC cache at a given point. This value currently defaults to 50. If a process has more entries in its IPC cache than ``MPICH_GPU_IPC_CACHE_MAX_SIZE``, the oldest entry is evicted. Frequent cache evection hurts performance. For known issues with number of messages in flight relative to ``MPICH_GPU_IPC_CACHE_MAX_SIZE``, please see :ref:`gpu-ipc-tips-tricks`. + ``MPICH_ASYNC_PROGRESS`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From f505a75978a5b0c4db36390305c33cf1c167a2c2 Mon Sep 17 00:00:00 2001 From: Ashesh Sharma <36968394+ashesh2512@users.noreply.github.com> Date: Wed, 1 Jul 2026 10:04:18 -0500 Subject: [PATCH 2/6] remove remark on FI_MR_CACHE_MONITOR --- systems/frontier_user_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/systems/frontier_user_guide.rst b/systems/frontier_user_guide.rst index ffd718f7..a542d922 100644 --- a/systems/frontier_user_guide.rst +++ b/systems/frontier_user_guide.rst @@ -4089,7 +4089,7 @@ Set to ``1`` by default if ``MPICH_GPU_SUPPORT_ENABLED=1``. This enables GPU IPC This option is unset by default, and is advised to be set to ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1`` when ``MPICH_GPU_IPC_ENABLED=1``. -Registering and unregistering GPU buffers for inter-process communication takes a relatively long time to do. The MPI library caches remote buffer attachments in order to reuse them, instead of detaching after every transfer. One limitation has been that when a process stops using a GPU buffer, there was no easy way to let the processes that are caching attachments know about this, and so they continued to cache the buffer. This consumes some of the limited GPU memory. A new feature since ROCM 6.4 enables the MPI library to alert the other processes through a signalling mechanism. If this environment variable is set, then this feature will be used to help reduce the amount of memory that the cache is consuming. This may help prevent out of memory situations. When enabling this feature, we do not recommend using the memhooks ``FI_MR_CACHE_MONITOR``, as it has been shown to hang in certain situations. +Registering and unregistering GPU buffers for inter-process communication takes a relatively long time to do. The MPI library caches remote buffer attachments in order to reuse them, instead of detaching after every transfer. One limitation has been that when a process stops using a GPU buffer, there was no easy way to let the processes that are caching attachments know about this, and so they continued to cache the buffer. This consumes some of the limited GPU memory. A new feature since ROCM 6.4 enables the MPI library to alert the other processes through a signalling mechanism. If this environment variable is set, then this feature will be used to help reduce the amount of memory that the cache is consuming. This may help prevent out of memory situations. ``HSA_ENABLE_IPC_MODE_LEGACY`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From 6a53cbe0274fb824b718c0eeb03abc193f625416 Mon Sep 17 00:00:00 2001 From: Ashesh Sharma <36968394+ashesh2512@users.noreply.github.com> Date: Wed, 1 Jul 2026 11:49:07 -0500 Subject: [PATCH 3/6] fix bug --- systems/frontier_user_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/systems/frontier_user_guide.rst b/systems/frontier_user_guide.rst index a542d922..c2b4ae61 100644 --- a/systems/frontier_user_guide.rst +++ b/systems/frontier_user_guide.rst @@ -4094,7 +4094,7 @@ Registering and unregistering GPU buffers for inter-process communication takes ``HSA_ENABLE_IPC_MODE_LEGACY`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This specifies AMD's IPC implementation, and should be set to ``1`` when ``MPICH_GPU_IPC_ENABLED=1``. Cray MPICH 9.0.1 and before should use ``HSA_DISABLE_IPC_MODE_LEGACY=0``. +This specifies AMD's IPC implementation, and is set to ``1`` by default when ``MPICH_GPU_IPC_ENABLED=1``. AMD has a newer version that uses the Linux dma-buf subsytem which can be accessed by setting ``HSA_ENABLE_IPC_MODE_LEGACY=0``. However, the newer version is not natively supported by Cray MPICH. For workaround with ``HSA_ENABLE_IPC_MODE_LEGACY=0``, please see :ref:`gpu-ipc-tips-tricks`. From bfe159e7419083112f763af5af81ec746f395ba0 Mon Sep 17 00:00:00 2001 From: Ashesh Sharma <36968394+ashesh2512@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:45:39 -0500 Subject: [PATCH 4/6] Update frontier_user_guide.rst --- systems/frontier_user_guide.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/systems/frontier_user_guide.rst b/systems/frontier_user_guide.rst index c2b4ae61..fa11a4da 100644 --- a/systems/frontier_user_guide.rst +++ b/systems/frontier_user_guide.rst @@ -671,7 +671,6 @@ To use GPU-aware Cray MPICH with Frontier's PrgEnv modules, users must set the f module load rocm export MPICH_GPU_SUPPORT_ENABLED=1 - export MPICH_GPU_IPC_ENABLED=1 # enabled by default if MPICH_GPU_SUPPORT_ENABLED=1 export GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1 export HSA_ENABLE_IPC_MODE_LEGACY=1 # Cray MPICH 9.1.0 onwards export HSA_DISABLE_IPC_MODE_LEGACY=0 # Cray MPICH 9.0.1 and before From ebc0eac3ca51af8fd2956f964f6a2129e1a92794 Mon Sep 17 00:00:00 2001 From: Ashesh Sharma <36968394+ashesh2512@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:49:59 -0500 Subject: [PATCH 5/6] Update frontier_user_guide.rst --- systems/frontier_user_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/systems/frontier_user_guide.rst b/systems/frontier_user_guide.rst index fa11a4da..09018af9 100644 --- a/systems/frontier_user_guide.rst +++ b/systems/frontier_user_guide.rst @@ -4030,7 +4030,7 @@ Prior to Cray MPICH 8.1.33, a memory leak was observed in edge cases involving r A separate issue remains where enabling IPC can still lead to memory leaks for message sizes less than or equal to 1 MB. A workaround is to set ``HSA_ENABLE_IPC_MODE_LEGACY=0`` and ``GTL_HSA_VSMSG_CUTOFF_SIZE=0``. When message sizes fall at or below ``GTL_HSA_VSMSG_CUTOFF_SIZE``, the GPU Transport Layer (GTL) uses a blocking memory copy driven by the CPU instead of an HSA memcpy. For context, disabling legacy IPC mode (``HSA_ENABLE_IPC_MODE_LEGACY=0``) switches to AMD's newer IPC implementation based on the Linux dma_buf mechanism. -Another enhancement in GPU-Aware MPI that relates to IPC, has to do with multiple messages in flight. Prior to Cray MPICH 8.1.33, if the maximum messages in flight at any given time exceeded ``MPICH_GPU_IPC_CACHE_MAX_SIZE``, the program would abort immediately. This has been resolved, and IPC cache handles are evicted as the number of messages in flight increase beyond ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. However, a new issue exists where memory leaks can occur if GPU buffers are repeatedly created and destroyed while the number of messages in flight exceed ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. To mitigate this, it is recommended to set ``MPICH_GPU_IPC_CACHE_MAX_SIZE`` to a value larger than the maximum anticipated messages in flight. The default value is ``50``. If setting ``MPICH_GPU_IPC_CACHE_MAX_SIZE > 50``, the following parameters should be adjusted accordingly to maintain a decreasing order of magnitude, i.e., ``GTL_MAX_SIG_HANDLER_LEN`` > ``GTL_IPC_SIGNAL_POOL_TABLE_LEN`` >= ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. +Another enhancement in GPU-Aware MPI that relates to IPC, has to do with multiple messages in flight. Prior to Cray MPICH 8.1.33, if the maximum messages in flight at any given time exceeded ``MPICH_GPU_IPC_CACHE_MAX_SIZE``, the program would abort immediately. This has been resolved, and IPC cache handles are evicted as the number of messages in flight increase beyond ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. However, a new issue exists where memory leaks can occur if GPU buffers are repeatedly created and destroyed while the number of messages in flight exceed ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. To mitigate this, it is recommended to set ``MPICH_GPU_IPC_CACHE_MAX_SIZE`` to a value larger than the maximum anticipated messages in flight. The default value is ``50``. If setting ``MPICH_GPU_IPC_CACHE_MAX_SIZE > 50``, the following parameters should be adjusted accordingly to maintain a decreasing order of magnitude: ``GTL_MAX_SIG_HANDLER_LEN`` > ``GTL_IPC_SIGNAL_POOL_TABLE_LEN`` >= ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. For example: GTL_MAX_SIG_HANDLER_LEN=1024, GTL_IPC_SIGNAL_POOL_TABLE_LEN=50, MPICH_GPU_IPC_CACHE_MAX_SIZE=50. Environment variables of primary interest to GPU-aware MPI have been described in :ref:`mpi-tuning`. For a detailed list, please refer to the `Cray MPICH User Guide `_. From f9f6d680baed492ce3d5ff0173a2fa46b2ae5a11 Mon Sep 17 00:00:00 2001 From: Ashesh Sharma <36968394+ashesh2512@users.noreply.github.com> Date: Wed, 22 Jul 2026 13:18:13 -0500 Subject: [PATCH 6/6] Update frontier_user_guide.rst --- systems/frontier_user_guide.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/systems/frontier_user_guide.rst b/systems/frontier_user_guide.rst index 09018af9..856bc2c4 100644 --- a/systems/frontier_user_guide.rst +++ b/systems/frontier_user_guide.rst @@ -4028,7 +4028,7 @@ Cray MPICH uses IPC by default for intra-node inter-process MPI data movement op Prior to Cray MPICH 8.1.33, a memory leak was observed in edge cases involving repeated creation and freeing of GPU-buffers used in MPI communication. This issue was prevelant when GPU buffers continuously increased in size or occupied new memory regions (e.g., due to defragmentation). The root cause was traced to IPC cache handles not being properly released. AMD addressed this by introducing an API that allows remote processes to free IPC cache handles. For Cray MPICH 9.0.1 and prior, the updated IPC mechanism is enabled by setting ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE=1`` and ``HSA_DISABLE_IPC_MODE_LEGACY=0``. Starting with Cray MPICH 9.1.0, the latter variable was renamed to ``HSA_ENABLE_IPC_MODE_LEGACY=1`` for improved clarity. -A separate issue remains where enabling IPC can still lead to memory leaks for message sizes less than or equal to 1 MB. A workaround is to set ``HSA_ENABLE_IPC_MODE_LEGACY=0`` and ``GTL_HSA_VSMSG_CUTOFF_SIZE=0``. When message sizes fall at or below ``GTL_HSA_VSMSG_CUTOFF_SIZE``, the GPU Transport Layer (GTL) uses a blocking memory copy driven by the CPU instead of an HSA memcpy. For context, disabling legacy IPC mode (``HSA_ENABLE_IPC_MODE_LEGACY=0``) switches to AMD's newer IPC implementation based on the Linux dma_buf mechanism. +A separate issue remains where enabling IPC can still lead to memory leaks for message sizes less than or equal to 1 MB. A workaround is to set ``HSA_ENABLE_IPC_MODE_LEGACY=0`` and ``GTL_HSA_VSMSG_CUTOFF_SIZE=0``. When message sizes fall at or below ``GTL_HSA_VSMSG_CUTOFF_SIZE``, the GPU Transport Layer (GTL) uses a blocking memory copy driven by the CPU instead of an HSA memcpy. For context, disabling legacy IPC mode (``HSA_ENABLE_IPC_MODE_LEGACY=0``/``HSA_DISABLE_IPC_MODE_LEGACY=1``) switches to AMD's newer IPC implementation based on the Linux dma_buf mechanism. Another enhancement in GPU-Aware MPI that relates to IPC, has to do with multiple messages in flight. Prior to Cray MPICH 8.1.33, if the maximum messages in flight at any given time exceeded ``MPICH_GPU_IPC_CACHE_MAX_SIZE``, the program would abort immediately. This has been resolved, and IPC cache handles are evicted as the number of messages in flight increase beyond ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. However, a new issue exists where memory leaks can occur if GPU buffers are repeatedly created and destroyed while the number of messages in flight exceed ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. To mitigate this, it is recommended to set ``MPICH_GPU_IPC_CACHE_MAX_SIZE`` to a value larger than the maximum anticipated messages in flight. The default value is ``50``. If setting ``MPICH_GPU_IPC_CACHE_MAX_SIZE > 50``, the following parameters should be adjusted accordingly to maintain a decreasing order of magnitude: ``GTL_MAX_SIG_HANDLER_LEN`` > ``GTL_IPC_SIGNAL_POOL_TABLE_LEN`` >= ``MPICH_GPU_IPC_CACHE_MAX_SIZE``. For example: GTL_MAX_SIG_HANDLER_LEN=1024, GTL_IPC_SIGNAL_POOL_TABLE_LEN=50, MPICH_GPU_IPC_CACHE_MAX_SIZE=50.