Logs Truncated and Subsequent Entries Appended in Log Files

[kvssg123]

Description

We are encountering an issue where some log entries within our log files are being truncated. These truncated logs are directly followed by the subsequent log entry, leading to a single, combined log line. This situation also raises concerns about the potential for data loss, as log entries may be missing between those that have been merged. We are also using Log Redactor for masking logs.

Configuration

JDK Version: 21
OS: Linux

Rolling File Appender configuration:

<RollingFile name="File" fileName="${logFile}.log" filePattern="${logbackupdirectory}/${podName}_${sys:port}-%i.log.gz"> <PatternLayout> <Pattern>%d{DATE}{GMT+5:30} %5p %t %X{Id} %c{1}:%L - %m%n %xEx</Pattern> </PatternLayout> <Policies> <SizeBasedTriggeringPolicy size="${sys:LOGSIZE:-50} MB" /> </Policies> <DefaultRolloverStrategy max="${sys:MAXLOGS:-300}" /> </RollingFile>

Logs

Expected Behaviour :

2025-01-03 12:34:56 INFO [Module] Operation completed successfully
2025-01-03 12:34:57 ERROR [Module] An error occurred

Actual behaviour :

2025-01-03 12:34:56 INFO [Module] Operation completed successfu2025-01-03 12:34:57 ERROR [Module] An error occurred

[vy]

1. Which Log4j version?
2. If you're not using the most recent version (i.e., 2.24.3), do you still observe the problem when you upgrade to 2.24.3?
3. Do you still observe the problem when you remove the Log Redactor?

[kvssg123]

Thank you for your response. We are currently using Log4j version 2.17.2. The issue occurs exclusively in our production clusters running on AWS EKS. We attempted to reproduce the problem in a controlled environment by generating dummy logs in a multi-threaded setup using the same Log4j2 configuration and version but were unable to replicate the behavior.
Since this issue manifests only in production, it has been challenging to investigate without log redaction enabled. Additionally, these logs are exported to our ELK stack for further analysis. Is there an alternative way to debug this issue without directly experimenting in the production environment?

[ppkarwasz]

@kvssg123,

Is there an alternative way to debug this issue without directly experimenting in the production environment?

Did you exactly replicate your production environment, including the Java system properties and JRE? I would double check if the garbage-free properties are the same in the two environments.

Can you add %t to your pattern layout to find out if the merged messages are from the same or different threads? I am a little bit surprised you don't use JSON Template Layout with an ECS template: it is the ideal choice for EKS.