Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2405][BRCM-DNX]: 'sonic-clear queuecounter' CLI does not clear the queue counters. #20913

Closed
amitpawar12 opened this issue Nov 25, 2024 · 6 comments
Labels
Chassis 🤖 Modular chassis support

Comments

@amitpawar12
Copy link

amitpawar12 commented Nov 25, 2024

Description

This is regression in 2405 as compared to release 2205.

We are using CLI - 'sudo ip netns exec asic<0/1> sonic-clear queuecounter' to clear the queue counter. The same CLI was used in 2205 to clear the counters.

However, in 2405, this CLI does not clear the counter.

Note - there is an on-going discussion about 'show ...' CLIs involved in multi-asic environment. However, the clear commands should also be addressed. This issue is a placeholder to ensure that we address this issue.

@saksarav-nokia.

Steps to reproduce the issue:

  1. Send traffic to one of the ports.
  2. Check the queue counters on the egress ASIC.
  3. Ensure that counters are non-zero.
admin@ixre-egl-board72:~$ sudo ip netns exec asic0 show queue counter --nonzero
Ethernet0 Last cached time was 2024-11-25T16:36:01.921665
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC0       199028410     195047841800            0             0
Ethernet0    UC2       199028410     195047841800            0             0
Ethernet0    UC3       198293421     194327552580    198233784  194269108320
Ethernet0    UC4       198547418     194576469640    198259271  194294085580
Ethernet0    UC5       199028915     195048336700            0             0
Ethernet0    UC7              36            4,557            0             0
  1. Clear the counters and check the counters.
admin@ixre-egl-board72:~$ sudo ip netns exec asic0 sonic-clear queuecounter
Clear and update saved counters for Ethernet0
Clear and update saved counters for Ethernet8
Clear and update saved counters for Ethernet16
Clear and update saved counters for Ethernet24
Clear and update saved counters for Ethernet32
Clear and update saved counters for Ethernet40
------------- curtailed output ------------
  1. In 2205, the counters will be reset to zero. However, in 2405, this does not happen.
admin@ixre-egl-board72:~$ sudo ip netns exec asic0 show queue counter --nonzero
Ethernet0 Last cached time was 2024-11-25T16:36:42.270594
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC0       199028410     195047841800            0             0
Ethernet0    UC2       199028410     195047841800            0             0
Ethernet0    UC3       198293421     194327552580    198233784  194269108320
Ethernet0    UC4       198547418     194576469640    198259271  194294085580
Ethernet0    UC5       199028915     195048336700            0             0
Ethernet0    UC7            5265           480007         1283        103768

  1. Tried this with different CLI to view the counters, and that counter is also showing old values.
admin@ixre-egl-board71:~$ show queue counters -n asic0  --nonzero
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC3       603846010     608676778080    603717878  608547621024
Ethernet0    UC4       603898579     608729767632    603780253  608610495024
Ethernet0    UC7            3731           298476            0             0

     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC7            9445           755655            0             0

        Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
------------  -----  --------------  ---------------  -----------  ------------
Ethernet-IB0    UC0          176678         12995171            0             0
Ethernet-IB0    UC7          178899         15718445            0             0

         Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
-------------  -----  --------------  ---------------  -----------  ------------
Ethernet-Rec0    UC7               4              528            0             0

admin@ixre-egl-board71:~$ sudo ip netns exec asic0 sonic-clear queuecounters
Clear and update saved counters for Ethernet0
Clear and update saved counters for Ethernet8
Clear and update saved counters for Ethernet16
---------- curtailed output --------------

admin@ixre-egl-board71:~$ show queue counters -n asic0  --nonzero
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC3       603846010     608676778080    603717878  608547621024
Ethernet0    UC4       603898579     608729767632    603780253  608610495024
Ethernet0    UC7            3734           298662            0             0

     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC7            9448           755841            0             0

        Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
------------  -----  --------------  ---------------  -----------  ------------
Ethernet-IB0    UC0          176822         13005775            0             0
Ethernet-IB0    UC7          179044         15731133            0             0

         Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
-------------  -----  --------------  ---------------  -----------  ------------
Ethernet-Rec0    UC7               4              528            0             0



Describe the results you received:

In 2405 release, these counters are not reset to zero.

Describe the results you expected:

The counters should reset to zero.

Output of show version:

admin@ixre-egl-board72:~$ show version 

SONiC Software Version: SONiC.HEAD.884690-202405-281b4a58
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: XXXXXXX
Build commit: 281b4a58
Build date: Fri Nov 22 08:05:41 UTC 2024
Built by: gitlab-runner@sonic-build-server03

Platform: x86_64-nokia_ixr7250e_36x400g-r0
HwSKU: Nokia-IXR7250E-36x100G
ASIC: broadcom
ASIC Count: 2
Serial Number: XXXXXXX
Model Number: XXXXXXX
Hardware Revision: 56
Uptime: 16:58:34 up  1:53,  1 user,  load average: 1.24, 1.35, 1.42
Date: Mon 25 Nov 2024 16:58:34

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

@prabhataravind prabhataravind added the Chassis 🤖 Modular chassis support label Dec 4, 2024
@prabhataravind
Copy link
Contributor

@rlhui for viz

@kenneth-arista
Copy link
Contributor

This should be addressed by sonic-net/sonic-utilities#3635 which merged to 202405 a fews days ago. However, I haven't yet had a chance to test the latest upstream image to confirm.

@amitpawar12
Copy link
Author

Logs for 'sonic-clear queuecounter':

admin@ixre-egl-board73:~$ show queue counters -n asic0 --nonzero
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC7            1064           290162            0             0

     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC0      2329501912    2348137927296            0             0
Ethernet8    UC1      1164750956    1174068963648            0             0
Ethernet8    UC2      3494252868    3522206890944            0             0
Ethernet8    UC3     16952485537   17088105421296            0             0
Ethernet8    UC4     16952483980   17088103851840            0             0
Ethernet8    UC5      2329501912    2348137927296            0             0
Ethernet8    UC6      1164750956    1174068963648            0             0
Ethernet8    UC7            1060           289385            0             0

admin@ixre-egl-board73:~$ 
admin@ixre-egl-board73:~$ 
admin@ixre-egl-board73:~$ 
admin@ixre-egl-board73:~$ sonic-clear queuecounters 


admin@ixre-egl-board73:~$ show queue counters -n asic0 --nonzero
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC7            1065           290436            0             0

     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC0      2329501912    2348137927296            0             0
Ethernet8    UC1      1164750956    1174068963648            0             0
Ethernet8    UC2      3494252868    3522206890944            0             0
Ethernet8    UC3     16952485537   17088105421296            0             0
Ethernet8    UC4     16952483980   17088103851840            0             0
Ethernet8    UC5      2329501912    2348137927296            0             0
Ethernet8    UC6      1164750956    1174068963648            0             0
Ethernet8    UC7            1061           289659            0             0

@kenneth-arista
Copy link
Contributor

I confirmed that the cherry-picked PR (sonic-net/sonic-utilities#3635) addresses this issue. The issue now is that we need to bump up sonic-utilities to pick up the change.

@amitpawar12
Copy link
Author

@kenneth-arista , @vmittal-msft, @saksarav-nokia

Sakthi helped to patch the queue counter changes. We tested the patch and found that 'Ethernet0' egress stats are not cleared with 'sonic-clear queuecounters' CLI.

Output with patch:

admin@ixre-egl-board72:~$ show queue counter -n asic0 --nonzero
Ethernet0 Last cached time for asic0 was 2024-12-05T21:43:32.337031
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC0      3119801769    3144760183152            0             0
Ethernet0    UC1      2944297818    2967852200544            0             0
Ethernet0    UC2      1293807557    1304158017456            0             0
Ethernet0    UC3      3609147049    3638020225392    830694313  837339867504
Ethernet0    UC4      3609137405    3638010504240    830694125  837339678000
Ethernet0    UC5      1777331998    1791550653984            0             0
Ethernet0    UC6      1971429416    1987200851328            0             0
Ethernet0    UC7             215           17,244            0             0

Ethernet8 Last cached time for asic0 was 2024-12-05T21:43:32.343546
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC7             217           17,368            0             0

Ethernet16 Last cached time for asic0 was 2024-12-05T21:43:32.351116
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet16    UC7             218           17,430            0             0

Ethernet24 Last cached time for asic0 was 2024-12-05T21:43:32.356603
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet24    UC7             218           17,430            0             0

admin@ixre-egl-board72:~$ sonic-clear queuecounters 
Clear and update saved counters for Ethernet0
Clear and update saved counters for Ethernet8
Clear and update saved counters for Ethernet16
Clear and update saved counters for Ethernet24
Clear and update saved counters for Ethernet32
Clear and update saved counters for Ethernet40
---- curtailed output -------------

admin@ixre-egl-board72:~$ show queue counter -n asic0 --nonzero
Ethernet0 Last cached time for asic0 was 2024-12-05T21:48:52.341509
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC0      3119801769    3144760183152            0             0
Ethernet0    UC1      2944297818    2967852200544            0             0
Ethernet0    UC2      1293807557    1304158017456            0             0
Ethernet0    UC3      3609147049    3638020225392    830694313  837339867504
Ethernet0    UC4      3609137405    3638010504240    830694125  837339678000
Ethernet0    UC5      1777331998    1791550653984            0             0
Ethernet0    UC6      1971429416    1987200851328            0             0
Ethernet0    UC7              10              910            0             0

Ethernet8 Last cached time for asic0 was 2024-12-05T21:48:52.345969
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC7               7              724            0             0                        <<<< This stats are cleared

Ethernet16 Last cached time for asic0 was 2024-12-05T21:48:52.350290
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet16    UC7               9              636            0             0                        <<<< This stats are cleared

Ethernet24 Last cached time for asic0 was 2024-12-05T21:48:52.354574
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet24    UC7               9              636            0             0                        <<<< This stats are cleared

As seen above, Ethernet0 stats are not cleared, while the other interfaces have their counters cleared. It is possible that maybe if there are multiple priority entries for a given interface, the CLI is unable to clear it. Will test and update.

Thanks,
-A

rlhui pushed a commit to sonic-net/sonic-utilities that referenced this issue Dec 12, 2024
…oq (#3671)

Fixed the sonic-clear queuecounter issues reported in sonic-net/sonic-buildimage#20913

For the egress queue counters, after the queue counter is cleared and show is issued with --nonzero option, the code takes the elif path at line 323 (old code) even if the diff between cached counter and current counter is 0, prints the current counter values from counter_db . This issue was for both egress queue counter and voq counter.
When the sonic-clear queuecounter is issued , the queuestat is called first without --voq option and this gets the port names and queue ids for each port in each asic , reads the egress queue counters and cache the values in /tmp/cache/queuestat/. Then queuestat is called with --voq option and this gets all the system ports names and voq id's for each asic , reads the voq counters and cache the values in /tmp/cache/queuestat. Since each asic has the all the system ports and all the voqs, and since the cache file name is queuestat+system_port_name with out asic namespace, caching asic1's voq counters overwrites the asic0's voq counters .
How I did it
1)Corrected the logic mistake for issue 1. Since cnstat_diff_print is called only if cache file is present and this should print only if non-zero & valid diff or (not non-zero )
2) Added asic namespace with the cache file name for vow counters.

Signed-off-by: saksarav <[email protected]>
mssonicbld pushed a commit to mssonicbld/sonic-utilities that referenced this issue Dec 24, 2024
…oq (sonic-net#3671)

Fixed the sonic-clear queuecounter issues reported in sonic-net/sonic-buildimage#20913

For the egress queue counters, after the queue counter is cleared and show is issued with --nonzero option, the code takes the elif path at line 323 (old code) even if the diff between cached counter and current counter is 0, prints the current counter values from counter_db . This issue was for both egress queue counter and voq counter.
When the sonic-clear queuecounter is issued , the queuestat is called first without --voq option and this gets the port names and queue ids for each port in each asic , reads the egress queue counters and cache the values in /tmp/cache/queuestat/. Then queuestat is called with --voq option and this gets all the system ports names and voq id's for each asic , reads the voq counters and cache the values in /tmp/cache/queuestat. Since each asic has the all the system ports and all the voqs, and since the cache file name is queuestat+system_port_name with out asic namespace, caching asic1's voq counters overwrites the asic0's voq counters .
How I did it
1)Corrected the logic mistake for issue 1. Since cnstat_diff_print is called only if cache file is present and this should print only if non-zero & valid diff or (not non-zero )
2) Added asic namespace with the cache file name for vow counters.

Signed-off-by: saksarav <[email protected]>
mssonicbld pushed a commit to sonic-net/sonic-utilities that referenced this issue Dec 24, 2024
…oq (#3671)

Fixed the sonic-clear queuecounter issues reported in sonic-net/sonic-buildimage#20913

For the egress queue counters, after the queue counter is cleared and show is issued with --nonzero option, the code takes the elif path at line 323 (old code) even if the diff between cached counter and current counter is 0, prints the current counter values from counter_db . This issue was for both egress queue counter and voq counter.
When the sonic-clear queuecounter is issued , the queuestat is called first without --voq option and this gets the port names and queue ids for each port in each asic , reads the egress queue counters and cache the values in /tmp/cache/queuestat/. Then queuestat is called with --voq option and this gets all the system ports names and voq id's for each asic , reads the voq counters and cache the values in /tmp/cache/queuestat. Since each asic has the all the system ports and all the voqs, and since the cache file name is queuestat+system_port_name with out asic namespace, caching asic1's voq counters overwrites the asic0's voq counters .
How I did it
1)Corrected the logic mistake for issue 1. Since cnstat_diff_print is called only if cache file is present and this should print only if non-zero & valid diff or (not non-zero )
2) Added asic namespace with the cache file name for vow counters.

Signed-off-by: saksarav <[email protected]>
@amitpawar12
Copy link
Author

Verification logs:

  1. Before clearing the counters:
admin@ixre-egl-board72:~$ show queue counter -n asic0 --nonzero
Ethernet0 Last cached time for asic0 was 2025-01-08T18:31:12.675517
     Port    TxQ    Counter/pkts      Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  -----------------  -----------  ------------
Ethernet0    UC0   1,452,458,737  1,423,409,562,260            0             0
Ethernet0    UC2   1,452,458,735  1,423,409,560,300            0             0
Ethernet0    UC3          45,050         44,149,000            0             0
Ethernet0    UC4          45,038         44,137,240            0             0
Ethernet0    UC5   1,452,458,737  1,423,409,562,260            0             0
Ethernet0    UC7           1,669            210,732            0             0

Ethernet8 Last cached time for asic0 was 2025-01-08T18:31:12.681983
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC7           1,035           82,926            0             0

Ethernet16 Last cached time for asic0 was 2025-01-08T18:31:12.688312
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet16    UC7           1,030           82,538            0             0

Ethernet24 Last cached time for asic0 was 2025-01-08T18:31:12.694611
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet24    UC7           1,032           82,688            0             0
  1. Clearing the counters:
Clear and update saved counters for Ethernet0
Clear and update saved counters for Ethernet8
Clear and update saved counters for Ethernet16
Clear and update saved counters for Ethernet24
---- curtailed output ---
  1. Checking the counter again:
admin@ixre-egl-board72:~$ show queue counter -n asic0 --nonzero
admin@ixre-egl-board72:~$ show queue counter -n asic0 --nonzero
Ethernet0 Last cached time for asic0 was 2025-01-08T18:55:10.509477
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet0    UC7              13            1,714            0             0

Ethernet8 Last cached time for asic0 was 2025-01-08T18:55:10.518857
     Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
---------  -----  --------------  ---------------  -----------  ------------
Ethernet8    UC7               8              734            0             0

Ethernet16 Last cached time for asic0 was 2025-01-08T18:55:10.528296
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet16    UC7              10              910            0             0

Ethernet24 Last cached time for asic0 was 2025-01-08T18:55:10.539009
      Port    TxQ    Counter/pkts    Counter/bytes    Drop/pkts    Drop/bytes
----------  -----  --------------  ---------------  -----------  ------------
Ethernet24    UC7              10              910            0             0

  1. show version output:
dmin@ixre-egl-board72:~$ show version 

SONiC Software Version: SONiC.HEAD.911514-202405-eded7e52f
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: 6.1.0-22-2-amd64
Build commit: xxxxxxxx
Build date: Fri Jan  3 22:41:31 UTC 2025
Built by: gitlab-runner@sonic-bld2

Closing this issue.

Thanks,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Chassis 🤖 Modular chassis support
Projects
Status: Done
Development

No branches or pull requests

3 participants