[T2] [Chassis] - Zebra process crashed during line card config-reload

[sanjair-git]

Description

On a T2 chassis line card, we see 'zebra' process got crashed and generated the following core during OC test run when the test did a 'config reload'

2024 Nov 24 06:26:29.856703 ixre-egl-board27 NOTICE swss1#orchagent: :- addRoutePost: Update Nexthop Group 10.0.0.165@Ethernet144,10.0.0.167@Ethernet152,10.0.0.169@Ethernet160,10.0.0.171@Ethernet168,10.0.0.173@Ethernet176,10.0.0.175@Ethernet184,10.0.0.177@Ethernet192,10.0.0.179@Ethernet200,10.0.0.181@Ethernet208,10.0.0.183@Ethernet216,10.0.0.185@Ethernet224,10.0.0.187@Ethernet232,10.0.0.189@Ethernet240
2024 Nov 24 06:26:29.864921 ixre-egl-board27 NOTICE bgp0#fpmsyncd: message repeated 8 times: [ :- onRouteMsg: RouteTable del msg for route with only one nh on eth0/docker0: fe80::/64 :: eth0 ]
2024 Nov 24 06:26:29.864955 ixre-egl-board27 ERR bgp0#fpmsyncd: :- poll_descriptors: readData error: Connection reset by peer
2024 Nov 24 06:26:29.865190 ixre-egl-board27 INFO bgp0#supervisord: fpmsyncd Connection lost, reconnecting...
2024 Nov 24 06:26:29.865559 ixre-egl-board27 INFO bgp0#supervisord: fpmsyncd Waiting for fpm-client connection...
2024 Nov 24 06:26:29.865639 ixre-egl-board27 INFO bgp0#supervisord 2024-11-24 06:26:29,865 WARN exited: zebra (terminated by SIGABRT (core dumped); not expected)
2024 Nov 24 06:26:29.868045 ixre-egl-board27 WARNING bgp0#bgpmon: *WARNING* JSONDecodeError: Expecting value: line 1 column 1 (char 0) when execute: ['vtysh', '-c', 'show bgp summary json'] Retry attempt: 1
2024 Nov 24 06:26:29.900148 ixre-egl-board27 NOTICE swss1#orchagent: :- addRoutePost: Update Nexthop Group 

Attaching core, syslog and zebra logs for reference,

zebra.1732429588.53.0.core.gz
zebra-logs.zip
zebra-issue-board27.gz

Steps to reproduce the issue:

1. On any T2 chassis line card, run a test which does 'config-reload'. The issue might happen.

Describe the results you received:

• zebra process crashed and didn't come up for next 30 minutes.
• core generated

2024 Nov 24 06:27:29.924944 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (1.0 minutes).
2024 Nov 24 06:28:29.979504 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (2.0 minutes).
2024 Nov 24 06:29:30.034640 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (3.0 minutes).
2024 Nov 24 06:30:30.090872 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (4.0 minutes).
2024 Nov 24 06:31:30.141823 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (5.0 minutes).
2024 Nov 24 06:32:30.196762 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (6.0 minutes).
2024 Nov 24 06:33:30.253843 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (7.0 minutes).
2024 Nov 24 06:34:30.307730 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (8.0 minutes).
2024 Nov 24 06:35:30.362860 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (9.0 minutes).
2024 Nov 24 06:36:30.418050 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (10.0 minutes).
2024 Nov 24 06:37:30.475897 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (11.0 minutes).
2024 Nov 24 06:38:30.534972 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (12.0 minutes).
2024 Nov 24 06:39:30.593887 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not unning in namespace 'asic0' (13.0 minutes).
2024 Nov 24 06:40:30.649885 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (14.0 minutes).
2024 Nov 24 06:41:30.707705 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (15.0 minutes).
2024 Nov 24 06:42:30.765343 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (16.0 minutes).
2024 Nov 24 06:43:30.822839 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (17.0 minutes).
2024 Nov 24 06:44:30.882951 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (18.0 minutes).
2024 Nov 24 06:45:30.936833 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (19.0 minutes).
2024 Nov 24 06:46:30.992880 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (20.0 minutes).
2024 Nov 24 06:47:31.051059 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (21.0 minutes).
2024 Nov 24 06:48:31.106957 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (22.0 minutes).
2024 Nov 24 06:49:31.166279 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (23.0 minutes).
2024 Nov 24 06:50:31.224898 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (24.0 minutes).
2024 Nov 24 06:51:31.280794 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (25.0 minutes).
2024 Nov 24 06:52:31.337925 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (26.0 minutes).
2024 Nov 24 06:53:31.395933 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (27.0 minutes).
2024 Nov 24 06:54:31.451244 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (28.0 minutes).
2024 Nov 24 06:55:31.509811 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (29.0 minutes).
2024 Nov 24 06:56:31.566762 ixre-egl-board27 ERR bgp0#supervisor-proc-exit-listener: Process 'zebra' is not running in namespace 'asic0' (30.0 minutes).

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):