-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Open
Labels
type: bugA code related bug.A code related bug.
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
According to the documentation, the file sink with the text codec (and the same happens with raw_message) should store the content of the .message field as-is into the destination file. This does not work: only lines containing an LF character are appended. No error messages or warnings are written to vector.log.
When switching the codec to csv with only one field defined (using the encoding.csv.fields option), the contents of the message field are correctly written to the destination file. This is a bit of a workaround, but the standard text codec should work as well.
Configuration
# Configuration that works
sinks:
snk_file_online_text:
type: file
path: "/data/ONLINE/text/{{ .metamap.log_path }}/{{ .metamap.log_file }}.log"
compression: none
idle_timeout_secs: 40
timezone: local
inputs:
- etl_build_text_message
buffer:
type: disk
max_size: 314572800
when_full: block
encoding:
codec: csv
csv:
fields:
- message
# Configuration that does not work
sinks:
snk_file_online_text:
type: file
path: "/data/ONLINE/text/{{ .metamap.log_path }}/{{ .metamap.log_file }}.log"
compression: none
idle_timeout_secs: 40
timezone: local
inputs:
- etl_build_text_message
buffer:
type: disk
max_size: 314572800
when_full: block
encoding:
codec: text
# VRL etl_build_text_message
transforms:
etl_build_text_message:
type: remap
inputs:
- etl_parse_kafka_json_payload
file: "/etc/vector/vrl.d/build_text_message.vrl"
#
# Build text message (tab separated)
#
host = to_string!(."fromhost-dns")
msg = to_string!(.message)
ts = to_string!(.timereceived)
new_text_message = "{{ts }}\t{{ host }}\t{{ msg }}"
.message = new_text_message
Version
vector 0.51.1 (x86_64-unknown-linux-gnu 44c8f1c 2025-11-13 15:16:05.303418529)
Debug Output
Example Data
{
"timereported": "2025-12-12T01:00:33.101775+02:00",
"timereceived": "2025-12-12T01:00:33.102150+02:00",
"inputname": "syslog_ietf_tcp",
"uuid": "02719b95-b753-43f2-88cc-a5139939d889",
"fromhost-dns": "host.local.net",
"fromhost-ip": "192.168.1.1",
"hostname": "host.local.net",
"pri": "133",
"procid": "-",
"msgid": "-",
"severity": "notice",
"facility": "local0",
"tag": "infra.pihole.dns-01.dnsmasq",
"app-name": "infra.pihole.dns-01.dnsmasq",
"message": "Dec 12 01:00:33 dnsmasq[1175]: UDP 456659 reply configuration.ls.v.aaplimg.com is 17.253.73.205",
"synthetic-tag": "-",
"synthetic-log": "dns-01.dnsmasq"
}
Additional Context
No response
References
No response
Metadata
Metadata
Assignees
Labels
type: bugA code related bug.A code related bug.