fluentd-kubernetes-daemonset segmentation fault

[rpurdel]

Using fluentd-kubernetes-daemonset:v1.15-debian-cloudwatch-amd64-1 and sometimes getting a segmentation fault in the k8s metadata filter. Any idea why this is?

/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371: [BUG] Segmentation fault at 0x0000000000000030
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0030 p:---- s:0169 e:000168 CFUNC  :dump
c:0029 p:0143 s:0164 e:000162 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_meta
c:0028 p:0010 s:0148 e:000147 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96
c:0027 p:0022 s:0141 e:000140 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0026 p:---- s:0137 e:000136 CFUNC  :each
c:0025 p:0023 s:0133 e:000132 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0024 p:0031 s:0123 e:000122 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94
c:0023 p:0008 s:0116 e:000115 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218 [FINISH]
c:0022 p:---- s:0110 e:000109 IFUNC 
c:0021 p:---- s:0107 e:000106 CFUNC  :each
c:0020 p:---- s:0104 e:000103 CFUNC  :reduce
c:0019 p:0021 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217
c:0018 p:0009 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC 
c:0014 p:0257 s:0066 e:000063 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC  :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `reduce'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `filter'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `dump'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f61e7da0b24 RBP: 0x00007f61da821928 RSP: 0x00007f61dacfb828
 RAX: 0x00007f61da805460 RBX: 0x0000000000000000 RCX: 0x000000000000000d
 RDX: 0x0000000000000000 RDI: 0x00007f61da803bf8 RSI: 0x00007f61da821900
  R8: 0x000000000000000c  R9: 0x00000000000019c8 R10: 0x0000000000000000
 R11: 0x00007f61da821900 R12: 0x000000000000d000 R13: 0x000000000000000a
 R14: 0x0000000000000000 R15: 0x000000000000000a EFL: 0x0000000000010246

-- C level backtrace information -------------------------------------------

[ad-m-ss]

I have the following segmentation fault in Kubernetes pods when using Docker image fluent/fluentd-kubernetes-daemonset:v1.15.3-debian-cloudwatch-1.0:

/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180: [BUG] Segmentation fault at 0x0000000000000048
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0030 p:---- s:0175 e:000174 CFUNC  :write
c:0029 p:0036 s:0170 e:000169 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180
c:0028 p:0010 s:0163 e:000162 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1039
c:0027 p:0022 s:0157 e:000156 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0026 p:---- s:0153 e:000152 CFUNC  :each
c:0025 p:0023 s:0149 e:000148 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0024 p:0058 s:0139 e:000138 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1038
c:0023 p:0014 s:0127 e:000126 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:922
c:0022 p:0020 s:0119 e:000118 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:852
c:0021 p:0011 s:0113 e:000112 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0020 p:0009 s:0105 e:000104 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_relabel.rb:29
c:0019 p:0013 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840
c:0018 p:0020 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC
c:0014 p:0257 s:0066 e:000063 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC  :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840:in `emit_sync'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/out_relabel.rb:29:in `process'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:852:in `emit_buffered'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:922:in `execute_chunking'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1038:in `handle_stream_simple'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1039:in `block in handle_stream_simple'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180:in `format'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-cloudwatch-logs-0.14.3/lib/fluent/plugin/out_cloudwatch_logs.rb:180:in `write'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f4261b69922 RBP: 0x00007f4260e03b18 RSP: 0x00007f42524fb8b0
 RAX: 0x00007f4260eb9101 RBX: 0x00007f4260ea7c00 RCX: 0x0000000000000006
 RDX: 0xffffffffffff9000 RDI: 0xffffffffffffffbf RSI: 0x00007f4260ea7c00
  R8: 0x000000000000000c  R9: 0x0000000000001988 R10: 0x00007f4260eb91a8
 R11: 0x0000000000000000 R12: 0x0000000000007000 R13: 0x0000000000000000
 R14: 0x0000000000000000 R15: 0x0000000000000006 EFL: 0x0000000000010202

-- C level backtrace information -------------------------------------------

It's hard for me to tell if it's related.

[emil2k]

We have also experienced a similar segmentation fault using the image v1.15-debian-cloudwatch-1:

/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371: [BUG] Segmentation fault at 0x0000000000010047
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0045 p:---- s:0266 e:000265 CFUNC  :dump
c:0044 p:0143 s:0261 e:000259 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_meta
c:0043 p:0010 s:0245 e:000244 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96
c:0042 p:0011 s:0238 e:000237 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:110
c:0041 p:0031 s:0231 e:000230 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94
c:0040 p:0008 s:0224 e:000223 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218 [FINISH]
c:0039 p:---- s:0218 e:000217 IFUNC
c:0038 p:---- s:0215 e:000214 CFUNC  :each
c:0037 p:---- s:0212 e:000211 CFUNC  :reduce
c:0036 p:0021 s:0207 e:000206 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217
c:0035 p:0009 s:0201 e:000200 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193
c:0034 p:0011 s:0193 e:000192 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0033 p:0028 s:0185 e:000184 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:106
c:0032 p:0011 s:0178 e:000177 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:103
c:0031 p:0038 s:0173 e:000172 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:338
c:0030 p:0073 s:0164 e:000163 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:288
c:0029 p:0122 s:0156 e:000155 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:107 [FINISH]
c:0028 p:---- s:0151 e:000150 CFUNC  :synchronize
c:0027 p:0014 s:0147 e:000146 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137
c:0026 p:0004 s:0142 e:000141 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:90
c:0025 p:0009 s:0135 e:000134 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:82
c:0024 p:0022 s:0130 e:000129 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196 [FINISH]
c:0023 p:---- s:0126 e:000125 CFUNC  :each
c:0022 p:0023 s:0122 e:000121 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195
c:0021 p:0005 s:0112 e:000111 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:81
c:0020 p:0016 s:0105 e:000104 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/compat/output.rb:164
c:0019 p:0013 s:0099 e:000098 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840
c:0018 p:0020 s:0093 e:000092 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196
c:0017 p:0011 s:0085 e:000084 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115
c:0016 p:0064 s:0077 e:000076 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615 [FINISH]
c:0015 p:---- s:0069 e:000068 IFUNC
c:0014 p:0257 s:0066 e:000063 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104
c:0013 p:0047 s:0056 e:000055 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140
c:0012 p:0026 s:0050 e:000049 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064
c:0011 p:0003 s:0046 e:000045 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006 [FINISH]
c:0010 p:---- s:0043 e:000042 CFUNC  :synchronize
c:0009 p:0006 s:0039 e:000038 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006
c:0008 p:0058 s:0035 e:000034 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817
c:0007 p:0004 s:0030 e:000029 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394
c:0006 p:0005 s:0027 e:000026 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749 [FINISH]
c:0005 p:---- s:0022 e:000021 CFUNC  :run_once
c:0004 p:0041 s:0017 e:000016 METHOD /fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0012 e:000011 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0080 s:0008 e:000007 BLOCK  /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'
/fluentd/vendor/bundle/ruby/3.1.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:749:in `on_timer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:394:in `block in setup_watcher'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:817:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1006:in `block in on_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1064:in `handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1140:in `with_io'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:1104:in `block in handle_notify'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/in_tail.rb:615:in `receive_lines'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:196:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:840:in `emit_sync'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/compat/output.rb:164:in `process'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:81:in `emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:195:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:196:in `block in each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:82:in `block in emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:90:in `process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:137:in `synchronize'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:107:in `block in process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:288:in `push'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/exception_detector.rb:338:in `update_buffer'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-detect-exceptions-0.0.14/lib/fluent/plugin/out_detect_exceptions.rb:103:in `block (2 levels) in process_record'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:106:in `emit'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:115:in `emit_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:193:in `emit_events'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `reduce'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:217:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event_router.rb:218:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:94:in `filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/event.rb:110:in `each'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter.rb:96:in `block in filter_stream'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `filter'
/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-kubernetes_metadata_filter-2.13.0/lib/fluent/plugin/filter_kubernetes_metadata.rb:371:in `dump'

-- Machine register context ------------------------------------------------
 RIP: 0x00007f3651497922 RBP: 0x00007f3637a03a98 RSP: 0x00007f3637efb220
 RAX: 0x00007f3637ab5401 RBX: 0x00007f3637a9bb00 RCX: 0x0000000000000006
 RDX: 0x00007f3637aa7d00 RDI: 0xffffffffffffffbf RSI: 0x00007f3637a9bb00
  R8: 0x0000000000000000  R9: 0x00007f3628752000 R10: 0x00007f3637ab54a8
 R11: 0x0000000000000000 R12: 0x0000000000007000 R13: 0x0000000000000000
 R14: 0x000000000000ffff R15: 0x0000000000000006 EFL: 0x0000000000010206

-- C level backtrace information -------------------------------------------
2023-04-11 18:11:12 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 18:46:53 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 19:30:29 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 20:06:03 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone
2023-04-11 20:49:46 +0000 [info]: [filter_kube_metadata] 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days

[ad-m-ss]

FTR: As problem resolution we migrated from Fluentd to Fluentbit.

[kenhys]

All of reports are against v1.15 series (ruby 3.1)
Are these issue still reproducible in v1.16 or v1.17?