===============================================================================
				Changes in 2.2.0
===============================================================================

# Implemented dynamic heap memory allocation (requires CUDA version >= 11.3) for
runs with P2P GPUs. It can be enabled using NVSHMEM_DISABLE_CUDA_VMM=0. Support
for IB runs will be added in the next release.
# Improved UCX transport performance for AMO and RMA operations
# Improved performance for warp and block put/get operations
# Added atomic support for PCIe connected GPUs over the UCX transport
# UCX transport now supports non-symmetric buffers for use as local buffers
in RMA and AMO operations
# Added support for initializing NVSHMEM in CUmodule
# Enabled MPI and PMIx bootstrap modules to be compiled externally from the
NVSHMEM build. This allows multiple builds of these plugins to support various
MPI and PMIx libraries. They can be selected by setting NVSHMEM_BOOTSTRAP="plugin"
and NVSHMEM_BOOTSTRAP_PLUGIN="plugin_name.so". Plugin sources are installed along
with the compiled NVSHMEM library.
# Enabled MPI bootstrap to be used with nvshmem_init by setting
NVSHMEM_BOOTSTRAP=MPI or via the bootstrap plugin method.
# Fixed bugs in nvshmem_<typename>_g and fetch atomics implementation
# Changed nvshmem_<typename>_collect to nvshmem_<typename>_fcollect to match
OpenSHMEM specification
# Fixed type of nreduce argument in reduction API to size_t to match OpenSHMEM
specification
# Improved NVSHMEM build times with multi-threaded option in CUDA compiler
(requires CUDA version >= 11.2)
# Several fixes to address Coverity reports

===============================================================================
				Changes in 2.1.2
===============================================================================

# Added a new, experimental UCX internode communication transport layer
# Added support for automatic warp-level coalescing of nvshmem_g operations
# Added support for put-with-signal operations on CUDA streams
# Added support for mapping the symmetric heap using the cuMem APIs
# Improved performance of single-threaded NVSHMEM put/get device API
# Added the NVSHMEM_MAX_TEAMS environment variable to specify maximum number
of teams that can be created
# Improved the host and on-stream Alltoall performance by using NCCL
# Fixed a bug in the compare-and-swap operation that caused several bytes of the
compare operand to be lost
# Added CPU core affinity to debugging output
# Added support for the CUDA 11.3 cudaDeviceFlushGPUDirectRDMAWrites API for consistency
# Improved support for the NVIDIA Tools Extension (NVTX) to enable performance
analysis through NVIDIA NSight
# Removed support for nvshmem_wait API that has been deprecated in OpenSHMEM 1.5
# Removed NVSHMEM_IS_P2P_RUN environment variable, runtime automatically determines it
# Made improvements to NVSHMEM example codes
# Added NVSHMEM_REMOTE_TRANSPORT environment variable for selecting the networking
layer used for communication between nodes
# Set maxrregcount to 32 for non-inlined device functions to ensure that calling
these NVSHMEM functions does not negatively affect kernel occupancy

===============================================================================
				Changes in 2.0.3
===============================================================================

# Added work-around to avoid deadlocks due to CUDA context resource reconfiguration
on Power systems
# Added environment variable NVSHMEM_CUDA_LIMIT_STACK_SIZE to set GPU thread stack size
on Power systems
# Use of NCCL for stream and host NVSHMEM collectives is now supported on Power systems
# Updated threading level support reported for host and stream-based APIs
to NVSHMEM_THREAD_SERIALIZED. Device-side APIs support NVSHMEM_THREAD_MULTIPLE
# Fixed a bug that could lead to incorrect behavior for atomic compare-and-swap
# Fixed an issue that was observed to lead to incorrect results when using GDRCopy

===============================================================================
				Changes in 2.0.2 EA
===============================================================================

# Added the teams and team-based collectives APIs from OpenSHMEM 1.5.
# Added support to use the NVIDIA Collective Communication Library (NCCL) for
optimized NVSHMEM host and on-stream collectives.
# Added support for RDMA over Converged Ethernet (RoCE) networks.
# Added support for PMI-2 to enable an NVSHMEM job launch with srun/SLURM. 
# Added support for PMIx to enable an NVSHMEM job launch with PMIx-compatible
launchers, such as Slurm and Open MPI.
# Uniformly reformatted the perftest benchmark output.
# Added support for the putmem_signal and signal_wait_until APIs.
# Improved support for single-node environments without InfiniBand.
# Fixed a bug that occurred when large numbers of fetch atomic operations were
performed on InfiniBand.
# Improved topology awareness in NIC-to-GPU assignments for DGX A100 systems.

===============================================================================
				Changes in 1.1.3
===============================================================================

# Implements nvshmem_<type>_put_signal API from OpenSHMEM 1.5
# Adds nvshmemx_signal_op API
# Optimizes implementation of signal set operation over P2P connected GPUs
# Optimizes performance of nvshmem_fence() function
# Optimizes latency of NVSHMEM atomics API
# Fixes bug in nvshmem_ptr API
# Fixes bug in implementation of host-side strided transfer (iput, iget, etc.) API
# Fixes bug in on-stream reduction for `long long` datatype
# Fixes hang during nvshmem barrier collective operation
# Fixes __device__ nvshmem_quiet() to also do quiet on IB ops to self
# Fixes bug in fetch atomic and g implementation 

===============================================================================
				Changes in 1.0.1
===============================================================================

# Combines the memory of multiple GPUs into a partitioned global address space 
that’s accessed through NVSHMEM APIs.
# Includes a low-overhead, in-kernel communication API for use by GPU threads.
# Includes stream-based and CPU-initiated communication APIs.  
# Supports peer-to-peer communication using NVIDIA NVLink and PCI Express and for 
GPU clusters using NVIDIA Mellanox® InfiniBand. 
# Supports x86 and POWER9 processors.  
# Is interoperable with MPI and other OpenSHMEM implementations.
