ECS Fargate parse JSON stdout logs with firelens fluentbit compatibility

[tomiszili]

Describe the question/issue

How can i configure ECS Fargate Firelens logging with the fluentbit-init image to send JSON formatted logs to splunk and observe as well? The container logs in json format, but when i tried to use the docker PARSER and a FILTER to the log key it sends nothing to splunk nor observe.

Configuration

[PARSER]
    Name         docker
    Format       json
    # Command      |  Decoder | Field | Optional Action
    # =============|==================|=================
    Decode_Field_As   escaped_utf8    log    do_next
    Decode_Field_As   json       log

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser docker
    Reserve_Data True
    Preserve_Key True

[OUTPUT]
    name         http
    match        *
    host         <masked>collect.observeinc.com
    port         443
    tls          on

    uri          /v1/http/fluentbit

    format       json
    header       Authorization     Bearer <masked>
    header       X-Observe-Decoder fluent
    header       Content-Type application/json
    compress     gzip

[OUTPUT]
    Name        splunk
    Match       *
    Host        <masked>.splunkcloud.com
    Port        443
    TLS         On
    TLS.Verify  Off
    Splunk_Token <masked>
    Splunk_Send_Raw Off

    event_index nonprod
    event_source /app/solaris

Task definition

{
  "executionRoleArn": "arn:aws:iam::...task_execution",
  "containerDefinitions": [
    {
      "essential": true,
      "cpu": 0,
      "volumesFrom": [],
      "readonlyRootFilesystem": false,
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:init-latest",
      "name": "log_router",
      "firelensConfiguration": {
        "type": "fluentbit"
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/solaris/log_router",
          "awslogs-region": "us-west-2",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "firelens-init"
        }
      },
      "environment": [
        {
          "name": "aws_fluent_bit_init_s3_1",
          "value": "arn:aws:s3:::somebucket/fluentbit-splunk-observe.conf"
        }
      ],
      "memoryReservation": 50
    },
    {
      "entryPoint": [
        "/entrypoint.sh"
      ],
      "environment": [],
      "secrets": [],
      "essential": true,
      "cpu": 0,
      "mountPoints": [
        {
          "sourceVolume": "app_tmp_volume",
          "containerPath": "/tmp",
          "readOnly": false
        },
        {
          "sourceVolume": "managed-agents",
          "containerPath": "/managed-agents",
          "readOnly": false
        },
        {
          "sourceVolume": "var-lib-amazon-ssm",
          "containerPath": "/var/lib/amazon/ssm",
          "readOnly": false
        },
        {
          "sourceVolume": "var-log-amazon-ssm",
          "containerPath": "/var/log/amazon/ssm",
          "readOnly": false
        }
      ],
      "volumesFrom": [],
      "readonlyRootFilesystem": true,
      "image": "<image>",
      "logConfiguration": {"logDriver":"awsfirelens","options":{}},
      "name": "pro",
      "portMappings": [
        {
          "containerPort": 8443,
          "hostPort": 8443,
          "protocol": "tcp"
        }
      ],
      "ulimits": [
        {
          "name": "nofile",
          "softLimit": 65535,
          "hardLimit": 65535
        }
      ]
    }
  ]
  ,
  "placementConstraints": [],
  "memory": "6144",
  "taskRoleArn": "arn:aws:iam::...task_role",
  "family": "pro_family",
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "2048",
  "volumes": [{"name":"app_tmp_volume"},{"name":"managed-agents"},{"name":"var-lib-amazon-ssm"},{"name":"var-log-amazon-ssm"}]
}

Fluent Bit Log Output

Fluent Bit Version Info

1.9.10

Cluster Details

Fargate
Sidecar deployment of fluentbit init image.

Application Details

The application logs in JSON format to the STDOUT inside the container. The task definition restricts the root filesystem to readonly for the app container.

Steps to reproduce issue

Related Issues