Docker logging to fluentd failed.

[kindule]

Describe the bug

Docker fails to output logs via Fluentd. The application, created using docker-compose, is configured to route logs through Fluentd to a specific directory and archive them daily. However, after running for a few days, the issue of Fluentd symbolic link files not existing arises.

1. The Fluentd configuration file is as follows

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter *>
  @type record_transformer
  <record>
    message ${record["log"]}
  </record>
</filter>

<match *>
  @type file
  path /opt/hawkeye/logs/${tag}.%Y%m%d
  symlink_path /opt/hawkeye/logs/${tag}.log
  compress gzip
  append true
  <format>
    @type single_value
  </format>
  <buffer tag,time>
    timekey_use_utc false
    timekey_zone "Asia/Shanghai"
  </buffer>

2. The docker version and parts of the docker-compose.yaml configuration file are as follows:

Server:
 Containers: 20
  Running: 18
  Paused: 0
  Stopped: 2
 Images: 150
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

docker-compose.yaml

4. after running for a few days, the issue of Fluentd symbolic link files not existing arises.
   

To Reproduce

Use the specified Docker version along with the Fluentd configuration file and a similar docker-compose configuration to start the Docker application. Then, let it run for several days.

Expected behavior

The expected result is for Fluentd to correctly create log files daily and accurately generate the corresponding symbolic link files.

Your Environment

- Fluentd version:

fluentd 1.16.2 

• Fluentd docker images:

fluent/fluentd:v1.16-1

• Operating system:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel version:

3.10.0-1127.el7.x86_64

### Your Configuration

```apache
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter *>
  @type record_transformer
  <record>
    message ${record["log"]}
  </record>
</filter>

<match *>
  @type file
  path /opt/hawkeye/logs/${tag}.%Y%m%d
  symlink_path /opt/hawkeye/logs/${tag}.log
  compress gzip
  append true
  <format>
    @type single_value
  </format>
  <buffer tag,time>
    timekey_use_utc false
    timekey_zone "Asia/Shanghai"
  </buffer>
</match>

### Your Error Log

```shell
No error log found. 

hawkeye-fluentd-1  | 2023-12-01 09:13:57 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
hawkeye-fluentd-1  | 2023-12-01 09:13:57 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
hawkeye-fluentd-1  | 2023-12-01 09:13:57 +0000 [info]: gem 'fluentd' version '1.16.2'
hawkeye-fluentd-1  | 2023-12-01 09:13:57 +0000 [info]: using configuration file: <ROOT>
hawkeye-fluentd-1  |   <source>
hawkeye-fluentd-1  |     @type forward
hawkeye-fluentd-1  |     port 24224
hawkeye-fluentd-1  |     bind "0.0.0.0"
hawkeye-fluentd-1  |   </source>
hawkeye-fluentd-1  |   <filter *>
hawkeye-fluentd-1  |     @type record_transformer
hawkeye-fluentd-1  |     <record>
hawkeye-fluentd-1  |       message ${record["log"]}
hawkeye-fluentd-1  |     </record>
hawkeye-fluentd-1  |   </filter>
hawkeye-fluentd-1  |   <match *>
hawkeye-fluentd-1  |     @type file
hawkeye-fluentd-1  |     path "/opt/hawkeye/logs/${tag}.%Y%m%d"
hawkeye-fluentd-1  |     symlink_path "/opt/hawkeye/logs/${tag}.log"
hawkeye-fluentd-1  |     compress gzip
hawkeye-fluentd-1  |     append true
hawkeye-fluentd-1  |     <format>
hawkeye-fluentd-1  |       @type "single_value"
hawkeye-fluentd-1  |     </format>
hawkeye-fluentd-1  |     <buffer tag,time>
hawkeye-fluentd-1  |       timekey_use_utc false
hawkeye-fluentd-1  |       timekey_zone "Asia/Shanghai"
hawkeye-fluentd-1  |       path "/opt/hawkeye/logs/${tag}.%Y%m%d"
hawkeye-fluentd-1  |     </buffer>
hawkeye-fluentd-1  |   </match>
hawkeye-fluentd-1  | </ROOT>
hawkeye-fluentd-1  | 2023-12-01 09:13:57 +0000 [info]: starting fluentd-1.16.2 pid=7 ruby="3.1.4"
hawkeye-fluentd-1  | 2023-12-01 09:13:57 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "--config", "/fluentd/etc/fluent.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: adding filter pattern="*" type="record_transformer"
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: adding match pattern="*" type="file"
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: adding source type="forward"
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
hawkeye-fluentd-1  | 2023-12-01 09:13:59 +0000 [info]: #0 fluentd worker is now running worker=0

### Additional context

_No response_