v0.29.1

[0.29.1] - 2022-04-15

Fixed

• Complete removal of preserve_to setting and ensure parse_from field is always retained. (PR468)

v0.29.0

[0.29.0] - 2022-04-11

Upgrading to v0.29.0

Several changes have been made that affect configuration for the filelog, syslog, tcplog, udplog, and journald receivers. The following steps should be taken to migrate configuration of these receivers:

Update all usages of field syntax

Field syntax no longer requires the $ character. Instead each field must begin with body, attributes, or resource.

Deprecated Example	Updated Equivalent
$$attributes["log.file.name"]	attributes["log.file.name"]
$$resource["host.name"]	resource["host.name"]
$$body.foo	body.foo
$$.foo	body.foo
foo	body.foo

Tip for updating sub-parsers

To update the parse_from field in a "sub-parser", such as timestamp or severity, consider where the value would reside if the sub-parser was excluded.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Time=(?P<time>\d{4}-\d{2}-\d{2})...' parse_to: body # default timestamp: parse_from: time ...	operators: - type: regex_parser regex: '^Time=(?P<time>\d{4}-\d{2}-\d{2})...' parse_to: attributes # default timestamp: parse_from: attributes.time ...

attributes and resource now support arbitrary structure

Field syntax can now refer to nested fields in both attributes and resource. Nested fields were previously only supported in body. For example, attributes.foo.bar refers to the value "hello" in the following entry:

{
  "attributes": {
    "foo": {
      "bar": "hello"
    }
  }
}

Update references to parsed values

The default value of parse_to has been changed from body to attributes. As a general rule of thumb, any parsers that previously specified the parse_to field are unaffected. If the parse_to field was not previously specified, then subsequent operators may need to be updated.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body # parse_to: body (old default) - type: move from: foo to: attributes.bar ...	operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body # parse_to: attributes (new default) - type: move from: attributes.foo to: attributes.bar ...

Remove or replace usages of preserve_to

Parsers no longer support a preserve_to setting. Instead, the parse_from field is preserved by default. To preserve and move the field, use the move operator after the parse operator. To remove the field after parsing, use the remove operator.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body preserve_to: attributes.backup ...	operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body - type: move from: body to: attributes.backup ...
operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body # implicitly deleted ...	operators: - type: regex_parser regex: '^Foo=(?P<foo>\s.*)...' parse_from: body - type: remove field: body ...

Replace usages of restructure operator

The restructure operator has been removed. Use add, copy, flatten, move, remove, and retain operators instead.

Deprecated Example	Updated Equivalent
operators: - type: restructure ops: - add: field: set_me value: foo - add: field: overwrite_me value: bar - move: from: details.env to: env - remove: field: delete_me	operators: - type: add field: body.set_me value: foo - type: add field: body.overwrite_me value: bar - type: move from: body.details.env to: body.env - type: remove field: body.delete_me

Replace usages of metadata operator

The metadata operator has been removed. Use add, copy, or move operators instead.

Deprecated Example	Updated Equivalent
operators: - type: metadata attributes: environment: production file: 'EXPR( $$body.file )' resource: cluster: blue	operators: - type: add field: attributes.environment value: production - type: copy from: body.file to: attributes.file - type: add field: resource.cluster value: blue - type: move from: body.foo to: attributes.bar

Update filelog attribute references

The filelog receiver has adopted slightly different attribute names in order to match newly established semantic conventions. Configurations that previously refered to the file.* attributes should be updated.

Deprecated Attribute	Updated Equivalent
file.name	log.file.name
file.path	log.file.path
file.name.resolved	log.file.name_resolved
file.path.resolved	log.file.path_resolved

Note to Vendors: A log record's Timestamp field may now be nil

A recent change to the Log Data Model has redefined the usage of the Timestamp field. Correspondingly, this field is no longer initialized by default. All Log Exporters should be evaluated to ensure this change is handled accordingly.

Log exporters can use the following logic to mimic the previous functionality (psuedocode):

timestamp := log.ObservedTimestamp
if log.Timestamp != nil {
  timestamp = log.Timestamp
}

---

Breaking Changes

• The default value of parse_to field in all parsers has been changed to attributes. (PR463)
• Parsers that contain a parse_to setting will no longer delete the parse_from field. (PR464)
• The preserve_to setting has been removed from parsers. (PR464)

Added

• key_value_parser (PR459)
• severity parsign can now use preset: otel to recognize both numeric and text representations of OpenTelemetry's log data model. (PR460)
• regex_parser can now cache parsing parsing results using the cache setting. This can dramatically increase performance in cases where the same string is parsed repeatedly. (PR440)

Fixed

• Issue where scope name parser would fail to initialize. (PR465)

v0.28.0

Upgrading to version 0.28.0

Several changes have been made that affect configuration for the filelog, syslog, tcplog, udplog, and journald receivers.

Update all usages of field syntax

Field syntax no longer requires the $ character. Instead each field must begin with body, attributes, or resource.

Deprecated Example	Updated Equivalent
$attributes["log.file.name"]	attributes["log.file.name"]
$resource["host.name"]	resource["host.name"]
$body.foo	body.foo
$.foo	body.foo
foo	body.foo

Tip for updating sub-parsers

To update the parse_from field in a "sub-parser", such as timestamp or severity, consider where the value would reside if the sub-parser was excluded.

Deprecated Example	Updated Equivalent
operators: - type: regex_parser regex: '^Time=(?P<time>\d{4}-\d{2}-\d{2})...' parse_to: body # default timestamp: parse_from: time ...	operators: - type: regex_parser regex: '^Time=(?P<time>\d{4}-\d{2}-\d{2})...' parse_to: body # default timestamp: parse_from: body.time ...

Replace usages of restructure operator

The restructure operator has been removed. Use add, copy, flatten, move, remove, and retain operators instead.

Deprecated Example	Updated Equivalent
operators: - type: restructure ops: - add: field: set_me value: foo - add: field: overwrite_me value: bar - move: from: details.env to: env - remove: field: delete_me	operators: - type: add field: body.set_me value: foo - type: add field: body.overwrite_me value: bar - type: move from: body.details.env to: body.env - type: remove field: body.delete_me

Replace usages of metadata operator

The metadata operator has been removed. Use add, copy, or move operators instead.

Deprecated Example	Updated Equivalent
operators: - type: metadata attributes: environment: production file: 'EXPR( $body.file )' resource: cluster: blue	operators: - type: add field: attributes.environment value: production - type: copy from: body.file to: attributes.file - type: add field: resource.cluster value: blue - type: move from: body.foo to: attributes.bar

Update filelog attribute references

The filelog receiver has adopted slightly different attribute names in order to match newly established semantic conventions. Configurations that previously refered to the file.* attributes should be updated.

Deprecated Attribute	Updated Equivalent
file.name	log.file.name
file.path	log.file.path
file.name.resolved	log.file.name_resolved
file.path.resolved	log.file.path_resolved

Note to Vendors: A log record's Timestamp field may now be nil

A recent change to the Log Data Model has redefined the usage of the Timestamp field. Correspondingly, this field is no longer initialized by default. All Log Exporters should be evaluated to ensure this change is handled accordingly.

Log exporters can use the following logic to mimic the previous funcationality (psuedocode):

timestamp := log.ObservedTimestamp
if log.Timestamp != nil {
  timestamp = log.Timestamp
}

---

[0.28.0] - 2022-03-28

Breaking Changes

• $ has been removed from field syntax. (PR364)
  • Use body instead of $body.
    • e.g. body.foo.
  • Use attributes instead of $attributes.
    • e.g. attributes.["log.file.name"]
  • Use resource instead of $resource.
    • e.g. resource.["host.name"]
  • There is no longer a default top-level field.
    • i.e. foo is no longer equivalent to $body.foo. (It is invalid.)
  • A top-level field MUST be specified at the beginning of each field.
    • e.g. body.foo, attributes.foo, or resource.foo.
• entry.Entry.Timestamp field is no longer required and is not initialized by default. (PR370)
  • The value can be set using the timestamp block on any parser, or the using the standalone time_parser operator.
• Removed metadata operator. (PR429)
  • Use add, copy, or move operators instead.
• Removed restructure operator. (PR371)
  • Use add, copy, flatten, move, remove, and retain operators instead.
• Changed the names of attributes added by file_input operator to match new semantic conventions. (PR372)
• Switch to original go-syslog library, restoring strict enforcement of SD-NAME length. (PR439)

v0.27.2

Fixed

• Revert version update on syslog-go, which introduced incompatibility with 386 architecture. (PR438)

v0.27.1

Fixed

• Issue where pipelines could fail to build when running on Go 1.18. (PR347)

v0.27.0

Added

• csv_parser can now handle fields containing line breaks. (PR425)

Fixed

• Issue where recombine operator would combine entire file in certain specific circumstances. (PR416)

v0.26.0

Fixed

• Time parsing will now correctly parse timestamps from 1970. (PR417)
• Issue where file_input operator could duplicate a small number of log entries. (PR413)

Changed

• entry.Attributes data type from map[string]string to map[string]interface{}. (PR401)
• entry.Resource data type from map[string]string to map[string]interface{}. (PR411)

Removed

• write_to configuration setting from all input operators. Use move operator instead. (PR412)

v0.25.0

This release contains a few minor updates as well as a major cleanup of the codebase. See the Reduce Complexity milestone for full details on the cleanup.

Added

• source_identifier setting to recombine operator, to ensure partial entries are joined to others from the same file, or other source. (PR341)
• max_sources setting to recombine operator, which limits the number of unique sources that may accumulate partial entries. (PR341)

Fixed

• On Windows, file_input will immediately close files after reading. (PR366)

Changed

• When file_input cannot open a file, it will print a debug level log instead of an error level log. (PR357)

v0.24.0

Added

• force_flush_period setting to recombine operator, to prevent recombine taking to long to process (PR325)
• lazy_quotes setting to csv parser operator. When enabled will preserve internal quotes in a csv field (PR324)
• header_attribute setting to csv parser operator. When set will dynamically parse the csv headers from the specified attribute on a log entry. (PR335)

Changed

• Updated CSV Parser to use idiomatic Go errors (PR323)

v0.23.0

Added

• combine_with setting to recombine operator, to allow for joining on custom delimiter (PR315)

Fixed

• Issue where force_flush_period could cause line splitting to be skipped (PR303)
• Issue where tcp_input and udp_input could panic when stopping (PR273)
• Syslog severity mapping is now aligned with log specification (PR300)

Changed

• Improve error message when timezone database is not found (PR289)