Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. #4462

MaxTownley · 2024-04-11T12:28:39Z

Describe the bug

When creating a source to receive syslog messages using the TLS transport method the client_cert_auth false setting is not applied/ So when a client cert is provided it will still attempt to validate the certificate.

This is because if the insecure setting is set to false the SSLContext:DEFAULT_PARAMS are set. This sets the verify_mode to VERIFY_PEER https://github.com/ruby/openssl/blob/master/lib/openssl/ssl.rb#L25

The verify_mode is only changed in cert_option_create_context if conf.client_cert_auth is set to true.

It seems the if statement here should also have an else that will explicitly set the verify_mode to VERIFY_NONE.

fluentd/lib/fluent/plugin_helper/cert_option.rb

Lines 34 to 36 in 284bf40

    
           if conf.client_cert_auth 
        
             ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER | OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT 
        
           end

if conf.client_cert_auth
    ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER | OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
else
    ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
end

To Reproduce

Set up a source for syslog over TLS.
Set client_cert_auth to false -- this is the default anyway.
Create a syslog client that has client certificates set up.
Attempt to send a log to fluentd.

Client config

global(
DefaultNetstreamDriver="ossl"
DefaultNetstreamDriverCAFile="my_ca_cert.pem"
DefaultNetstreamDriverCertFile="my_client_cer.pem"
DefaultNetstreamDriverKeyFile="my_client_key.pem"
)

Expected behavior

The server will ignore the certificate and allow you to proceed without validation.

Your Environment

- Fluentd version: fluent-package 5.0.3 fluentd 1.16.5 (95d130aaa44fc09b14a4e9686ee8018253185fc5)
- TD Agent version:
- Operating system: RHEL 8.6
- Kernel version: 4.18.0-372.9.1.el8.x86_64

Your Configuration

<source>
  @type syslog
  port 5140
  bind 0.0.0.0
  <transport tls>
    ca_path /etc/syslog_tls/syslog_ca_cert.pem
    cert_path /etc/syslog_tls/certificate.pem
    private_key_path /etc/syslog_tls/requests/server.key
    client_cert_auth false
    min_version TLS1_2
    max_version TLS1_3
  </transport>
  tag system
</source>

<match system.**>
  @type stdout
</match>

Your Error Log

2024-04-11 13:18:52 +0100 [info]: starting fluentd-1.16.5 pid=471933 ruby="3.2.3"
2024-04-11 13:18:52 +0100 [info]: spawn command to main:  cmdline=["/opt/fluent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/fluent/bin/fluentd", "--log", "/var/log/fluent/fluentd.log", "--daemon", "/var/run/fluent/fluentd.pid", "--under-supervisor"]
2024-04-11 13:18:53 +0100 [info]: #0 init worker0 logger path="/var/log/fluent/fluentd.log" rotate_age=nil rotate_size=nil
2024-04-11 13:18:53 +0100 [info]: adding match pattern="td.*.*" type="tdlog"
2024-04-11 13:18:53 +0100 [info]: adding match pattern="debug.**" type="stdout"
2024-04-11 13:18:53 +0100 [info]: adding match pattern="system.**" type="stdout"
2024-04-11 13:18:53 +0100 [info]: adding source type="syslog"
2024-04-11 13:18:53 +0100 [warn]: #0 For security reason, setting private_key_passphrase is recommended when cert_path is specified
2024-04-11 13:18:53 +0100 [info]: adding source type="http"
2024-04-11 13:18:53 +0100 [info]: adding source type="debug_agent"
2024-04-11 13:18:53 +0100 [info]: #0 starting fluentd worker pid=471945 ppid=471942 worker=0
2024-04-11 13:18:54 +0100 [info]: #0 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2024-04-11 13:18:54 +0100 [info]: #0 listening syslog socket on 0.0.0.0:5140 with tls
2024-04-11 13:18:54 +0100 [warn]: #0 For security reason, setting private_key_passphrase is recommended when cert_path is specified
2024-04-11 13:18:54 +0100 [info]: #0 fluentd worker is now running worker=0
2024-04-11 13:18:57 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX" host="XXXX" port=58960 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXX state=error: certificate verify failed (self signed certificate in certificate chain)"
2024-04-11 13:19:39 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX" host="XXXX" port=46584 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXX state=error: certificate verify failed (self signed certificate in certificate chain)"
2024-04-11 13:19:40 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX" host="XXXX" port=46596 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXX state=error: certificate verify failed (self signed certificate in certificate chain)"
2024-04-11 13:19:40 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX" host="XXXX" port=42242 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXX state=error: certificate verify failed (self signed certificate in certificate chain)"
2024-04-11 13:19:41 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX" host="XXXX" port=42246 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXX state=error: certificate verify failed (self signed certificate in certificate chain)"
2024-04-11 13:19:47 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX host="XXXX" port=38338 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXX state=error: certificate verify failed (self signed certificate in certificate chain)"
2024-04-11 13:19:48 +0100 [warn]: #0 unexpected error before accepting TLS connection by OpenSSL addr="XXXX" host="XXXX" port=38340 error_class=OpenSSL::SSL::SSLError error="SSL_accept returned=1 errno=0 peeraddr=XXXXstate=error: certificate verify failed (self signed certificate in certificate chain)"
Waiting for data... (interrupt to abort)

Additional context

If you specify insecure this will work as the DEFAULT_PARAMS are not loaded in and the verify_mode will be set to none which is the default. This doesn't seem like the intention behind the insecure setting though it seems the actual intention is to stop weak ciphers from being used.

The text was updated successfully, but these errors were encountered:

MaxTownley · 2024-04-11T23:16:02Z

I am just trying to work out if on the latest versions of openssl loading in the DEFAULT_PARAMS even makes the system anymore secure by default - as currently in my tests with "OpenSSL 1.1.1k FIPS 25 Mar 2021" it would seem that the list of ciphers with insecure false are weaker than with it set to true.....

daipom · 2024-04-12T02:40:21Z

@MaxTownley Thanks for reporting this in detail!

It seems the if statement here should also have an else that will explicitly set the verify_mode to VERIFY_NONE.

In my first impression, this looks reasonable.
It seems that the current implementation fails to take into account that the default mode is VERIFY_PEER.

If you know anything about the version of the package that causes this problem, please let us know.
(I'm wondering if this behavior has been this way for a long time, or if the behavior has changed with recent packages.)

MaxTownley · 2024-04-12T08:42:16Z

@MaxTownley Thanks for reporting this in detail!

It seems the if statement here should also have an else that will explicitly set the verify_mode to VERIFY_NONE.

In my first impression, this looks reasonable. It seems that the current implementation fails to take into account that the default mode is VERIFY_PEER.

If you know anything about the version of the package that causes this problem, please let us know. (I'm wondering if this behavior has been this way for a long time, or if the behavior has changed with recent packages.)

I think it has been this way for a very long time. I am just using the latest package (fluentd 1.16.5) but if we take a look a git blame it been this way for 6 years (d20d686)

You wouldn't notice if you just don't present a certificate at all as the verify cert code is just not called and it doesn't set VERIFY_FAIL_IF_NO_PEER_CERT by default. So it only tried to verify the client cert if one is provided but some clients might connect to many syslog servers some requiring a cert and some not. I believe the correct behavior here should be to ignore to ignore the client certificate if client_cert_auth is false.

MaxTownley · 2024-04-12T09:10:35Z

There seems to be a more troubling problem here to in further tests I have conducted. It seems the original intention of using open ssl DEFAULT_PARAMS is to stop using weak ciphers. It would appear if you are using a newer version of openssl the insecure false is actually using weaker ciphers than the ones openssl sets by default.

I am using OpenSSL 1.1.1k FIPS 25 Mar 2021 with insecure false set and scanning the port used for the TLS syslog source these are the cipher available for use

 Supported Server Cipher(s):
Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve 25519 DHE 253
Accepted  TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve 25519 DHE 253
Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve 25519 DHE 253
Accepted  TLSv1.3  128 bits  TLS_AES_128_CCM_SHA256        Curve 25519 DHE 253
Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-GCM-SHA384     DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-CHACHA20-POLY1305     DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM            DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-ARIA256-GCM-SHA384      Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-ARIA256-GCM-SHA384    DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-GCM-SHA256     DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM            DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-ARIA128-GCM-SHA256      Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-ARIA128-GCM-SHA256    DHE 2048 bits
Accepted  TLSv1.2  64 bits   DHE-RSA-AES256-CCM8           DHE 2048 bits
Accepted  TLSv1.2  64 bits   DHE-RSA-AES128-CCM8           DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA384       Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA256         DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CAMELLIA256-SHA384  Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-CAMELLIA256-SHA256    DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA256         DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-CAMELLIA128-SHA256  Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-CAMELLIA128-SHA256    DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA            DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-CAMELLIA256-SHA       DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA            DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-CAMELLIA128-SHA       DHE 2048 bits
Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384
Accepted  TLSv1.2  256 bits  AES256-CCM
Accepted  TLSv1.2  256 bits  ARIA256-GCM-SHA384
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256
Accepted  TLSv1.2  128 bits  AES128-CCM
Accepted  TLSv1.2  128 bits  ARIA128-GCM-SHA256
Accepted  TLSv1.2  64 bits   AES256-CCM8
Accepted  TLSv1.2  64 bits   AES128-CCM8
Accepted  TLSv1.2  256 bits  AES256-SHA256
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA256
Accepted  TLSv1.2  128 bits  AES128-SHA256
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA
Accepted  TLSv1.2  256 bits  CAMELLIA256-SHA
Accepted  TLSv1.2  128 bits  AES128-SHA
Accepted  TLSv1.2  128 bits  CAMELLIA128-SHA
Accepted  TLSv1.2  112 bits  TLS_RSA_WITH_3DES_EDE_CBC_SHA
Accepted  TLSv1.2  112 bits  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
Accepted  TLSv1.2  ?? bits    TLS_RSA_WITH_SEED_CBC_SHA
Accepted  TLSv1.2  ?? bits    TLS_DHE_RSA_WITH_SEED_CBC_SHA
Accepted  TLSv1.2  112 bits  TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SH

With insecure true I see the following list

  Supported Server Cipher(s):
Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve 25519 DHE 253
Accepted  TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve 25519 DHE 253
Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve 25519 DHE 253
Accepted  TLSv1.3  128 bits  TLS_AES_128_CCM_SHA256        Curve 25519 DHE 253
Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-GCM-SHA384     DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-CHACHA20-POLY1305     DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-CCM            DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-GCM-SHA256     DHE 2048 bits
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-CCM            DHE 2048 bits
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA256         DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA256       Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA256         DHE 2048 bits
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  DHE-RSA-AES256-SHA            DHE 2048 bits
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  DHE-RSA-AES128-SHA            DHE 2048 bits
Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384
Accepted  TLSv1.2  256 bits  AES256-CCM
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256
Accepted  TLSv1.2  128 bits  AES128-CCM
Accepted  TLSv1.2  256 bits  AES256-SHA256
Accepted  TLSv1.2  128 bits  AES128-SHA256
Accepted  TLSv1.2  256 bits  AES256-SHA
Accepted  TLSv1.2  128 bits  AES128-SHA

The list with insecure true is actually a list of stronger cipher suites.

I do have the min_version set to tls 1.2 and the max set to tls 1.3

But I guess this is a separate issue to one I have opened this for.......

daipom · 2024-08-19T01:38:05Z

Sorry for my late response.
We will check and fix this.

daipom · 2024-08-19T01:40:57Z

https://github.com/fluent/fluentd/blob/284bf4064a9831f5262330a94e3505e09ef6a068/lib/fluent/plugin_helper/cert_option.rb#L32C9-L32C23

This code prevents from selecting insecure TLS version.
However, looks like it makes verify_mode VERIFY_PEER unintendedly.

ashie · 2024-08-19T09:02:02Z

if conf.client_cert_auth
    ctx.verify_mode = OpenSSL::SSL::VERIFY_PEER | OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
else
    ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
end

It looks good, we should just apply this.

If you know anything about the version of the package that causes this problem, please let us know.
(I'm wondering if this behavior has been this way for a long time, or if the behavior has changed with recent packages.)

Why we overlooked it long time is that it's only affected only when clients send its certificate.
(In most cases, client certification isn't used.)
When a client doesn't send it, just ignored because VERIFY_FAIL_IF_NO_PEER_CERT isn't set by default:

https://github.com/ruby/openssl/blob/13b03ba4ed429cc8d46f5643cf5f9dd40275b4b2/lib/openssl/ssl.rb#L24-L33

      DEFAULT_PARAMS = { # :nodoc:
        :verify_mode => OpenSSL::SSL::VERIFY_PEER,
        :verify_hostname => true,
        :options => -> {
          opts = OpenSSL::SSL::OP_ALL
          opts &= ~OpenSSL::SSL::OP_DONT_INSERT_EMPTY_FRAGMENTS
          opts |= OpenSSL::SSL::OP_NO_COMPRESSION
          opts
        }.call
      }

In this case, verification won't performed.

https://docs.openssl.org/3.3/man3/SSL_CTX_set_verify/#notes

SSL_VERIFY_PEER

Server mode: the server sends a client certificate request to the client.
The certificate returned (if any) is checked.

...

SSL_VERIFY_FAIL_IF_NO_PEER_CERT

Server mode: if the client did not return a certificate, the TLS/SSL handshake is immediately terminated
with a "handshake failure" alert. This flag must be used together with SSL_VERIFY_PEER.

if any is the what we should focus on.

MaxTownley added the waiting-for-triage label Apr 11, 2024

daipom added bug Something isn't working and removed waiting-for-triage labels Apr 12, 2024

kenhys added the work-in-progress label Aug 19, 2024

ashie added this to the v1.18.0 milestone Sep 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. #4462

Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. #4462

MaxTownley commented Apr 11, 2024 •

edited by daipom

Loading

MaxTownley commented Apr 11, 2024

daipom commented Apr 12, 2024

MaxTownley commented Apr 12, 2024

MaxTownley commented Apr 12, 2024 •

edited

Loading

daipom commented Aug 19, 2024

daipom commented Aug 19, 2024

ashie commented Aug 19, 2024 •

edited

Loading

Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. #4462

Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. #4462

Comments

MaxTownley commented Apr 11, 2024 • edited by daipom Loading

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

MaxTownley commented Apr 11, 2024

daipom commented Apr 12, 2024

MaxTownley commented Apr 12, 2024

MaxTownley commented Apr 12, 2024 • edited Loading

daipom commented Aug 19, 2024

daipom commented Aug 19, 2024

ashie commented Aug 19, 2024 • edited Loading

MaxTownley commented Apr 11, 2024 •

edited by daipom

Loading

MaxTownley commented Apr 12, 2024 •

edited

Loading

ashie commented Aug 19, 2024 •

edited

Loading