How to send knative logs via TCP?

[pfilaretov42]

Hi, I'm looking for a way to send logs produced by knative pods to OpenSearch.

We have a standard way of doing this for our applications running in k8s:

app --[sends logs via TCP Socket Appender to]--> logging-agent --[sends to]--> Kafka --> validator/enricher --> OpenSearch

So, ideally I want to use the same mechanism for knative, with no new components installed in the k8s cluster (fluentbit, sidecars, etc).

The app logging configuration uses logback and looks like this (logging to logging-agent-svc-main:5170 using LogstashTcpSocketAppender):

<configuration>
  <appender name="TCP" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>logging-agent-svc-main:5170</destination>
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
      <providers>
        <pattern>
          <pattern>
            {
            "appName": "my-app",
            "threadName": "%thread",
            "loggerName": "%logger",
            "text": "%message",
            "level": "%level",
            "...": "..."
            }
          </pattern>
        </pattern>
        <!-- ... -->
      </providers>
    </encoder>
  </appender>
  <appender name="ASYNC_TCP" class="ch.qos.logback.classic.AsyncAppender">
    <neverBlock>true</neverBlock>
    <appender-ref ref="TCP"/>
  </appender>
  <root level="INFO">
    <appender-ref ref="ASYNC_TCP"/>
  </root>
</configuration>

And now I want the same thing for knative pods, but I cannot figure out how to configure it. I read the docs on logging and tried to update config-logging ConfigMap, i.e. adding data.zap-logger-config field:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-logging
#...
data:
  zap-logger-config: |
    {
      "level": "info",
      "development": false,
      "outputPaths": ["stdout"],
      "errorOutputPaths": ["stderr"],
      "encoding": "json",
      "encoderConfig": {
        ...
      }
    }

I looked at the zap documentation and tried to set outputPaths with one of the following:

• "outputPaths": ["logging-agent-svc-main:5170"]
• "outputPaths": ["http://logging-agent-svc-main:5170"]

However, it does not work and I can see the warning message in a knative pod (e.g., controller) during the start:

{
  "severity": "WARNING",
  "timestamp": "2025-01-31T11:53:41.106347634Z",
  "logger": "fallback-logger",
  "caller": "logging/config.go:73",
  "message": "Failed to parse logging config - using default zap production config",
  "commit": "6a27004"
}

Could you suggest the way this can be done, please?

[dprotaso]

Parsing happens here
https://github.com/knative/pkg/blob/5be486852d2713f8204aad8707e80da0004b6a5e/logging/config.go#L125

I'd honestly use fluentbit or something - I doubt anyone's done this before and I don't think we'd support this if you have problems. eg. what happens to latency in the activator when logging over the URL is slow.

[pfilaretov42]

Thanks for the reply, I'll look into fluentbit setup since looks like I have no other choice.

In regards to the following:

eg. what happens to latency in the activator when logging over the URL is slow.

Is there something like async TCP appender for zap? (something similar to the config I mentioned for logback)

[skonto]

If you dive deeper (I added some logging here):

diff --git a/vendor/go.uber.org/zap/writer.go b/vendor/go.uber.org/zap/writer.go
index 06768c679..695cdeba1 100644
--- a/vendor/go.uber.org/zap/writer.go
+++ b/vendor/go.uber.org/zap/writer.go
@@ -70,6 +70,7 @@ func open(paths []string) ([]zapcore.WriteSyncer, func(), error) {
        for _, path := range paths {
                sink, err := _sinkRegistry.newSink(path)
                if err != nil {
+                       fmt.Printf("DEBUG: %v\n", err)
                        openErr = multierr.Append(openErr, fmt.Errorf("open sink %q: %w", path, err))
                        continue
                }

the error is this:
a) "outputPaths": ["logging-agent-svc-main:5170"] causes

no sink found for scheme "logging-agent-svc-main"

b) "outputPaths": ["http://logging-agent-svc-main:5170"]

no sink found for scheme "http"

You need a custom async sink it seems as in here which you will have to register e.g. RegisterHTTPSink("http", "http://logging-agent-svc-main:5170") before you parse the configuration. However, as @dprotaso mentioned probably it is easier to handle this via parsing logs with some tooling (especially if it is already there for other purposes).

[dprotaso]

Going to close this out as there isn't anything actionable here