Skip to content

File sink, text codec only appends empty lines to the destination file. #24371

@jkimalane

Description

@jkimalane

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

According to the documentation, the file sink with the text codec (and the same happens with raw_message) should store the content of the .message field as-is into the destination file. This does not work: only lines containing an LF character are appended. No error messages or warnings are written to vector.log.

When switching the codec to csv with only one field defined (using the encoding.csv.fields option), the contents of the message field are correctly written to the destination file. This is a bit of a workaround, but the standard text codec should work as well.

Configuration

# Configuration that works
sinks:
  snk_file_online_text:
    type: file
    path: "/data/ONLINE/text/{{ .metamap.log_path }}/{{ .metamap.log_file }}.log"
    compression: none
    idle_timeout_secs: 40
    timezone: local
    inputs:
    - etl_build_text_message
    buffer:
      type: disk
      max_size: 314572800
      when_full: block
    encoding:
      codec: csv
      csv:
        fields:
        - message

# Configuration that does not work
sinks:
  snk_file_online_text:
    type: file
    path: "/data/ONLINE/text/{{ .metamap.log_path }}/{{ .metamap.log_file }}.log"
    compression: none
    idle_timeout_secs: 40
    timezone: local
    inputs:
    - etl_build_text_message
    buffer:
      type: disk
      max_size: 314572800
      when_full: block
    encoding:
      codec: text

# VRL etl_build_text_message

transforms:
  etl_build_text_message:
    type: remap
    inputs:
    - etl_parse_kafka_json_payload
    file: "/etc/vector/vrl.d/build_text_message.vrl"

#
# Build text message (tab separated)
#

host = to_string!(."fromhost-dns")
msg  = to_string!(.message)
ts   = to_string!(.timereceived)

new_text_message = "{{ts }}\t{{ host }}\t{{ msg }}"

.message = new_text_message

Version

vector 0.51.1 (x86_64-unknown-linux-gnu 44c8f1c 2025-11-13 15:16:05.303418529)

Debug Output


Example Data

{
  "timereported": "2025-12-12T01:00:33.101775+02:00",
  "timereceived": "2025-12-12T01:00:33.102150+02:00",
  "inputname": "syslog_ietf_tcp",
  "uuid": "02719b95-b753-43f2-88cc-a5139939d889",
  "fromhost-dns": "host.local.net",
  "fromhost-ip": "192.168.1.1",
  "hostname": "host.local.net",
  "pri": "133",
  "procid": "-",
  "msgid": "-",
  "severity": "notice",
  "facility": "local0",
  "tag": "infra.pihole.dns-01.dnsmasq",
  "app-name": "infra.pihole.dns-01.dnsmasq",
  "message": "Dec 12 01:00:33 dnsmasq[1175]: UDP 456659 reply configuration.ls.v.aaplimg.com is 17.253.73.205",
  "synthetic-tag": "-",
  "synthetic-log": "dns-01.dnsmasq"
}

Additional Context

No response

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugA code related bug.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions