How to test UCT 'intra-node' components? #7551

Honggang-LI · 2021-10-18T12:32:53Z

Honggang-LI
Oct 18, 2021

How can I test posix, sysv and cma components?

thanks

Oct 18, 2021

gtest will test all components including intra-node, also can run ucx_perftest (either UCP or UCT layer) and specify the transport to use. When using UCT layer test it's with "-x" parameter, and when using UCP test it's with UCX_TLS env var

View full answer

yosefe · 2021-10-18T12:43:23Z

yosefe
Oct 18, 2021
Maintainer

gtest will test all components including intra-node, also can run ucx_perftest (either UCP or UCT layer) and specify the transport to use. When using UCT layer test it's with "-x" parameter, and when using UCP test it's with UCX_TLS env var

3 replies

Honggang-LI Oct 18, 2021
Author

also can run ucx_perftest (either UCP or UCT layer) and specify the transport to use.

Can you please provide an example? It seems does not work for me.

$ /tmp/ucx/bin/ucx_perftest -t am_lat -x cma -d memory
[1634561332.151900] [graph:176148:0]        perftest.c:881  UCX  WARN  CPU affinity is not set (bound to 8 cpus). Performance may be impacted.
Waiting for connection...
Accepted connection from 127.0.0.1:58980
+----------------------------------------------------------------------------------------------------------+
| API:          transport layer                                                                            |
| Test:         active message latency                                                                     |
| Data layout:  short                                                                                      |
| Send memory:  host                                                                                       |
| Recv memory:  host                                                                                       |
| Message size: 8                                                                                          |
+----------------------------------------------------------------------------------------------------------+
[1634561340.659782] [graph:176148:0]         libperf.c:520  UCX  ERROR cma/memory does not support operation am short
[1634561340.672145] [graph:176148:0]    perftest_run.c:315  UCX  ERROR Failed to run test: Unsupported operation

$  /tmp/ucx/bin/ucx_perftest -t am_lat -x cma -d memory localhost
[1634561340.656880] [graph:176149:0]        perftest.c:881  UCX  WARN  CPU affinity is not set (bound to 8 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |        latency (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1634561340.672010] [graph:176149:0]         libperf.c:520  UCX  ERROR cma/memory does not support operation am short
[1634561340.672130] [graph:176149:0]    perftest_run.c:315  UCX  ERROR Failed to run test: Unsupported operation

yosefe Oct 18, 2021
Maintainer

"cma" transport does not support "am_lat" operation. it supports only put/get zcopy.
pls use "posix" or "sysv" instead of "cma"

Honggang-LI Oct 18, 2021
Author

"cma" transport does not support "am_lat" operation. it supports only put/get zcopy. pls use "posix" or "sysv" instead of "cma"

It works. thanks

Honggang-LI · 2021-10-18T13:06:01Z

Honggang-LI
Oct 18, 2021
Author

BTW, the examples/uct_hello_world only supports network component. It does not support intra-node nor loopback components. Is it right?

0 replies

yosefe · 2021-10-18T17:58:04Z

yosefe
Oct 18, 2021
Maintainer

examples/uct_hello_world supports network and intra-node , but not loopback components.

5 replies

Honggang-LI Oct 19, 2021
Author

examples/uct_hello_world supports network and intra-node , but not loopback components.

Please provide an example, as it does not work for me.

 ucx (master *)]$ ./src/tools/info/ucx_info -d | grep -A 2 Transport
#      Transport: posix
#         Device: memory
#           Type: intra-node
--
#      Transport: sysv
#         Device: memory
#           Type: intra-node
--
#      Transport: self
#         Device: memory0
#           Type: loopback
--
#      Transport: tcp
#         Device: eno1
#           Type: network
--
#      Transport: tcp
#         Device: lo
#           Type: network
--
#      Transport: cma
#         Device: memory
#           Type: intra-node

 ucx (master *)]$ ./examples/uct_hello_world -d memory -t posix
INFO: UCT_HELLO_WORLD AM function = uct_ep_am_short server = (null) port = 13337
Unsupported memory type host by posix/memory on posix MD
No supported (dev/tl) found (memory/posix)
Failed to find supported device and transport

 ucx (master *)]$ ./examples/uct_hello_world -d memory -t sysv
INFO: UCT_HELLO_WORLD AM function = uct_ep_am_short server = (null) port = 13337
Unsupported memory type host by sysv/memory on sysv MD
No supported (dev/tl) found (memory/sysv)
Failed to find supported device and transport

yosefe Oct 19, 2021
Maintainer

seems there is a bug in the example: it requires the UCT MD to support registration, but it's actually needed only if cmd_args.func_am_type == FUNC_AM_ZCOPY. Need to check

ucx/examples/uct_hello_world.c

Line 370 in f7a4757

if (!(iface_p->md_attr.cap.reg_mem_types & UCS_BIT(test_mem_type))) {

only for ZCOPY case

Honggang-LI Oct 19, 2021
Author

ucx/examples/uct_hello_world.c

Lines 370 to 378 in f7a4757

    
           if (!(iface_p->md_attr.cap.reg_mem_types & UCS_BIT(test_mem_type))) { 
        
               fprintf(stderr, "Unsupported memory type %s by " 
        
                       UCT_TL_RESOURCE_DESC_FMT" on %s MD\n", 
        
                       ucs_memory_type_names[test_mem_type], 
        
                       UCT_TL_RESOURCE_DESC_ARG(&tl_resources[tl_index]), 
        
                       component_attr.md_resources[md_index].md_name); 
        
               status = UCS_ERR_UNSUPPORTED; 
        
               break; 
        
           }

It does not help that comment out those lines. I get segfault.

Honggang-LI Oct 20, 2021
Author

ucx/src/uct/api/uct.h

Lines 1963 to 1970 in 59998e0

    
            * @param [in]  iface_addr Interface address to check reachability to. It is 
        
            *                         NULL if iface_attr.iface_addr_len == 0, and must 
        
            *                         be non-NULL otherwise. 
        
            * 
        
            * @return Nonzero if reachable, 0 if not. 
        
            */ 
        
           int uct_iface_is_reachable(const uct_iface_h iface, const uct_device_addr_t *dev_addr, 
        
                                      const uct_iface_addr_t *iface_addr);

ucx/examples/uct_hello_world.c

Line 638 in 59998e0

status = (ucs_status_t)uct_iface_is_reachable(if_info.iface, peer_dev, NULL);

I always get segfault because the third parameter of uct_iface_is_reachable is NULL when run uct_hello_world over intro-node components.

It seems the API document and usage is conflicting?

Honggang-LI Oct 21, 2021
Author

It seems the API document and usage is conflicting?

I confirmed intra-node components work for me with this patch.

ucx (master +)]$ git diff --cached
diff --git a/examples/uct_hello_world.c b/examples/uct_hello_world.c
index f97545f04580..6537bb1e19c5 100644
--- a/examples/uct_hello_world.c
+++ b/examples/uct_hello_world.c
@@ -367,7 +367,9 @@ static ucs_status_t dev_tl_lookup(const cmd_args_t *cmd_args,
             for (tl_index = 0; tl_index < num_tl_resources; ++tl_index) {
                 if (!strcmp(cmd_args->dev_name, tl_resources[tl_index].dev_name) &&
                     !strcmp(cmd_args->tl_name, tl_resources[tl_index].tl_name)) {
-                    if (!(iface_p->md_attr.cap.reg_mem_types & UCS_BIT(test_mem_type))) {
+                    if ((cmd_args->func_am_type == FUNC_AM_ZCOPY) &&
+                        !(iface_p->md_attr.cap.reg_mem_types &
+                          UCS_BIT(test_mem_type))) {
                         fprintf(stderr, "Unsupported memory type %s by "
                                 UCT_TL_RESOURCE_DESC_FMT" on %s MD\n",
                                 ucs_memory_type_names[test_mem_type],
@@ -635,9 +637,6 @@ int main(int argc, char **argv)
     CHKERR_ACTION(0 != res, "device exchange",
                   status = UCS_ERR_NO_MESSAGE; goto out_close_oob_sock);
 
-    status = (ucs_status_t)uct_iface_is_reachable(if_info.iface, peer_dev, NULL);
-    CHKERR_JUMP(0 == status, "reach the peer", out_close_oob_sock);
-
     /* Get interface address */
     if (if_info.iface_attr.cap.flags & UCT_IFACE_FLAG_CONNECT_TO_IFACE) {
         status = uct_iface_get_address(if_info.iface, own_iface);
@@ -649,6 +648,10 @@ int main(int argc, char **argv)
         CHKERR_JUMP(0 != status, "ifaces exchange", out_close_oob_sock);
     }
 
+    status = (ucs_status_t)uct_iface_is_reachable(if_info.iface, peer_dev,
+                                                  peer_iface);
+    CHKERR_JUMP(0 == status, "reach the peer", out_close_oob_sock);
+
     ep_params.field_mask = UCT_EP_PARAM_FIELD_IFACE;
     ep_params.iface      = if_info.iface;
     if (if_info.iface_attr.cap.flags & UCT_IFACE_FLAG_CONNECT_TO_EP) {

ucx (master +)]$ ./examples/uct_hello_world -d memory -t sysv -n localhost
INFO: UCT_HELLO_WORLD AM function = uct_ep_am_short server = localhost port = 13337
Using sysv/memory

ucx (master +)]$ ./examples/uct_hello_world -d memory -t sysv
INFO: UCT_HELLO_WORLD AM function = uct_ep_am_short server = (null) port = 13337
Using sysv/memory
Waiting for connection...


----- UCT TEST SUCCESS ----

[callback] uct_ep_am_short sent ABCDEFGHIJKLMNO (16 bytes)

---------------------------


----- UCT TEST SUCCESS ----

[main] uct_ep_am_short sent ABCDEFGHIJKLMNO (16 bytes)

---------------------------

Honggang-LI · 2021-10-21T12:45:26Z

Honggang-LI
Oct 21, 2021
Author

#7569

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to test UCT 'intra-node' components? #7551

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to test UCT 'intra-node' components? #7551

Honggang-LI Oct 18, 2021

Replies: 4 comments · 8 replies

yosefe Oct 18, 2021 Maintainer

Honggang-LI Oct 18, 2021 Author

yosefe Oct 18, 2021 Maintainer

Honggang-LI Oct 18, 2021 Author

Honggang-LI Oct 18, 2021 Author

yosefe Oct 18, 2021 Maintainer

Honggang-LI Oct 19, 2021 Author

yosefe Oct 19, 2021 Maintainer

Honggang-LI Oct 19, 2021 Author

Honggang-LI Oct 20, 2021 Author

Honggang-LI Oct 21, 2021 Author

Honggang-LI Oct 21, 2021 Author

Honggang-LI
Oct 18, 2021

Replies: 4 comments 8 replies

yosefe
Oct 18, 2021
Maintainer

Honggang-LI Oct 18, 2021
Author

yosefe Oct 18, 2021
Maintainer

Honggang-LI Oct 18, 2021
Author

Honggang-LI
Oct 18, 2021
Author

yosefe
Oct 18, 2021
Maintainer

Honggang-LI Oct 19, 2021
Author

yosefe Oct 19, 2021
Maintainer

Honggang-LI Oct 19, 2021
Author

Honggang-LI Oct 20, 2021
Author

Honggang-LI Oct 21, 2021
Author

Honggang-LI
Oct 21, 2021
Author