diff --git a/man/fabric.7.md b/man/fabric.7.md index a25190a46b0..5dedea377d3 100644 --- a/man/fabric.7.md +++ b/man/fabric.7.md @@ -455,6 +455,11 @@ Added new fields to the following attributes: *fi_domain_attr* : Added max_group_id +*fi_info* +: The fi_info structure was expanded to reference a new fabric object, + fi_hmem_attr. When available, the fi_hmem_attr references a new set of + attributes related to heterogeneous memory. + # SEE ALSO [`fi_info`(1)](fi_info.1.html), diff --git a/man/fi_fabric.3.md b/man/fi_fabric.3.md index 0d899ff9137..419cb68261f 100644 --- a/man/fi_fabric.3.md +++ b/man/fi_fabric.3.md @@ -177,6 +177,9 @@ datatype or field value. *FI_TYPE_LOG_SUBSYS* : enum fi_log_subsys +*FI_TYPE_HMEM_ATTR* +: struct fi_hmem_attr + fi_tostr() will return a pointer to an internal libfabric buffer that should not be modified, and will be overwritten the next time fi_tostr() is invoked. fi_tostr() is not thread safe. diff --git a/man/fi_getinfo.3.md b/man/fi_getinfo.3.md index 6219792257e..b4099278e78 100644 --- a/man/fi_getinfo.3.md +++ b/man/fi_getinfo.3.md @@ -143,6 +143,7 @@ struct fi_info { struct fi_domain_attr *domain_attr; struct fi_fabric_attr *fabric_attr; struct fid_nic *nic; + struct fi_hmem_attr *hmem_attr; }; ``` @@ -249,6 +250,73 @@ struct fi_info { closely associated with a hardware NIC. See [`fi_nic`(3)](fi_nic.3.html) for details. +*hmem_attr - heterogeneous memory attributes* +: Optionally supplied HMEM attributes. HMEM attributes may be + specified and returned as part of fi_getinfo. When provided as + hints, requested values of struct fi_hmem_attr should be set. On + output, the actual HMEM attributes that can be provided will be + returned. + +## HMEM ATTRIBUTES + +```c +enum fi_hmem_attr_opt { + FI_HMEM_ATTR_UNSPEC, + FI_HMEM_ATTR_REQUIRED, + FI_HMEM_ATTR_PREFERRED, + FI_HMEM_ATTR_DISABLED +}; + +struct fi_hmem_attr { + enum fi_hmem_iface iface; + enum fi_hmem_attr_opt api_permitted; + enum fi_hmem_attr_opt use_p2p; + enum fi_hmem_attr_opt use_dev_reg_copy; + struct fi_hmem_attr *next; +}; +``` +- *fi_hmem_attr_opt - int* +: Defines how the provider should handle HMEM attributes for an interface. + By default, the provider will chose whether to use the attributes + (FI_HMEM_ATTR_UNSPEC). + Valid values defined in fabric.h are: + * FI_HMEM_ATTR_UNSPEC: The attribute may be used by the provider + and is subject to the provider implementation. + * FI_HMEM_ATTR_REQUIRED: The attribute must be used for this interface, + operations that cannot be performed will be reported as failing. + * FI_HMEM_ATTR_PREFERRED: The attribute should be used by the + provider if available, but the provider may choose other implementation + if it is unavailable. + * FI_HMEM_ATTR_DISABLED: The attribute should not be used. + +- *iface* + +Indicates the software interfaces used by the application, details in +[`fi_mr`(3)](fi_mr.3.html) + +- *api_permitted* + +Controls whether libfabric is allowed to make device-specific API calls. +By default, libfabric is permitted to call device-specific API(e.g. CUDA API). +If user wish to prohibit libfabric from making such calls, user can achieve +that by set this field to FI_HMEM_ATTR_DISABLED. +The setopt option FI_OPT_CUDA_API_PERMITTED for endpoint takes precedence +over this attribute when api_permitted is not disabled. + +- *use_p2p* + +Controls whether peer to peer FI_HMEM transfers should be used. +The FI_OPT_FI_HMEM_P2P setopt option discussed in +[`fi_endpoint`(3)](fi_endpoint.3.html) takes precedence over this attribute. + +- *use_dev_reg_copy* + +Controls whether optimized memcpy for device memory is used, e.g. GDR copy. + +- *next* + +Pointer to the next fi_hmem_attr if using multiple non-system iface. + # CAPABILITIES Interface capabilities are obtained by OR-ing the following flags diff --git a/man/fi_info.1.md b/man/fi_info.1.md index 36bf07b3bc8..1143770a53b 100644 --- a/man/fi_info.1.md +++ b/man/fi_info.1.md @@ -216,6 +216,11 @@ fi_info: speed: 0 state: FI_LINK_UP network_type: InfiniBand + fi_hmem_attr: + iface: FI_HMEM_SYSTEM + api_permitted: FI_HMEM_ATTR_UNSPEC + use_p2p: FI_HMEM_ATTR_UNSPEC + use_dev_reg_copy: FI_HMEM_ATTR_UNSPEC ``` To see libfabric related environment variables `-e` option.