filter_parser: Hash_key for parsed result #9274

Athishpranav2003 · 2024-08-24T04:28:32Z

Add Hash key for parsed value in Filter parser (similar to fluentd)

Addresses #8893
Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature
filter_parser: Add new param to doc fluent-bit-docs#1444

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 · 2024-08-24T06:12:42Z

Configuration used for testing

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    
[INPUT]
    Name stdin
    Tag  myapp.*

[FILTER]
    Name parser
    Match myapp.*
    Parser inner_json_parser
    Key_name MESSAGE
    Reserve_Data On
    Hash_Value_Field On

[OUTPUT]
    Name        stdout
    Match       *
    Format      json_lines

[PARSER]
    Name        inner_json_parser
    Format      regex
    Regex       ^(?<STRING1>[^ ]+) (?<STRING2>.+)$

The manual testing output

root@127f08db3cdb:/workspaces/fluent-bit/build/bin# echo -n "{\"MESSAGE\": \"some value\"}" | ./fluent-bit -c config.conf -q
[2024/08/24 06:10:35] [ info] Configuration:
[2024/08/24 06:10:35] [ info]  flush time     | 1.000000 seconds
[2024/08/24 06:10:35] [ info]  grace          | 5 seconds
[2024/08/24 06:10:35] [ info]  daemon         | 0
[2024/08/24 06:10:35] [ info] ___________
[2024/08/24 06:10:35] [ info]  inputs:
[2024/08/24 06:10:35] [ info]      stdin
[2024/08/24 06:10:35] [ info] ___________
[2024/08/24 06:10:35] [ info]  filters:
[2024/08/24 06:10:35] [ info]      parser.0
[2024/08/24 06:10:35] [ info] ___________
[2024/08/24 06:10:35] [ info]  outputs:
[2024/08/24 06:10:35] [ info]      stdout.0
[2024/08/24 06:10:35] [ info] ___________
[2024/08/24 06:10:35] [ info]  collectors:
[2024/08/24 06:10:35] [ info] [fluent bit] version=3.1.7, commit=, pid=40121
[2024/08/24 06:10:35] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/08/24 06:10:35] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/24 06:10:35] [ info] [cmetrics] version=0.9.4
[2024/08/24 06:10:35] [ info] [ctraces ] version=0.5.5
[2024/08/24 06:10:35] [ info] [input:stdin:stdin.0] initializing
[2024/08/24 06:10:35] [ info] [input:stdin:stdin.0] storage_strategy='memory' (memory only)
[2024/08/24 06:10:35] [debug] [stdin:stdin.0] created event channels: read=21 write=22
[2024/08/24 06:10:35] [debug] [input:stdin:stdin.0] buf_size=16000
[2024/08/24 06:10:35] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/08/24 06:10:35] [ info] [sp] stream processor started
[2024/08/24 06:10:35] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/24 06:10:35] [ warn] [input:stdin:stdin.0] end of file (stdin closed by remote end)
[2024/08/24 06:10:35] [debug] [task] created task=0x7f86b00237d0 id=0 OK
{"date":1724479835.275198,"STRING1":"some","STRING2":"value","parsed":{"STRING1":"some","STRING2":"value"}}
[2024/08/24 06:10:35] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/08/24 06:10:35] [ warn] [engine] service will shutdown in max 5 seconds
[2024/08/24 06:10:35] [debug] [out flush] cb_destroy coro_id=0
[2024/08/24 06:10:35] [debug] [task] destroy task=0x7f86b00237d0 (task_id=0)
[2024/08/24 06:10:35] [ info] [engine] service has stopped (0 pending tasks)
[2024/08/24 06:10:35] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/08/24 06:10:35] [ info] [output:stdout:stdout.0] thread worker #0 stopped

Athishpranav2003 · 2024-08-26T09:26:58Z

Valgrind output

root@3aeb5205afb0:/workspaces/fluent-bit/build/bin# valgrind ./fluent-bit -c config.conf 
==3576== Memcheck, a memory error detector
==3576== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==3576== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==3576== Command: ./fluent-bit -c config.conf
==3576== 
Fluent Bit v3.1.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __  
|  ___| |                | |   | ___ (_) |         |____ |/  | 
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| | 
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | | 
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/08/26 09:22:44] [ info] Configuration:
[2024/08/26 09:22:44] [ info]  flush time     | 1.000000 seconds
[2024/08/26 09:22:44] [ info]  grace          | 5 seconds
[2024/08/26 09:22:44] [ info]  daemon         | 0
[2024/08/26 09:22:44] [ info] ___________
[2024/08/26 09:22:44] [ info]  inputs:
[2024/08/26 09:22:44] [ info]      stdin
[2024/08/26 09:22:44] [ info] ___________
[2024/08/26 09:22:44] [ info]  filters:
[2024/08/26 09:22:44] [ info]      parser.0
[2024/08/26 09:22:44] [ info] ___________
[2024/08/26 09:22:44] [ info]  outputs:
[2024/08/26 09:22:44] [ info]      stdout.0
[2024/08/26 09:22:44] [ info] ___________
[2024/08/26 09:22:44] [ info]  collectors:
[2024/08/26 09:22:44] [ info] [fluent bit] version=3.1.7, commit=, pid=3576
[2024/08/26 09:22:44] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/08/26 09:22:44] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/08/26 09:22:44] [ info] [cmetrics] version=0.9.4
[2024/08/26 09:22:44] [ info] [ctraces ] version=0.5.5
[2024/08/26 09:22:44] [ info] [input:stdin:stdin.0] initializing
[2024/08/26 09:22:44] [ info] [input:stdin:stdin.0] storage_strategy='memory' (memory only)
[2024/08/26 09:22:44] [ info] [output:stdout:stdout.0] worker #0 started
[2024/08/26 09:22:44] [debug] [stdin:stdin.0] created event channels: read=21 write=22
[2024/08/26 09:22:44] [debug] [input:stdin:stdin.0] buf_size=16000
[2024/08/26 09:22:44] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/08/26 09:22:44] [ info] [sp] stream processor started
s
[2024/08/26 09:23:07] [debug] [input:stdin:stdin.0] invalid JSON message, skipping
{"MESSAGE":"some"}
[2024/08/26 09:23:59] [debug] [task] created task=0x577e5f0 id=0 OK
[2024/08/26 09:23:59] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
{"date":1724664238.915407,"MESSAGE":"some"}
[2024/08/26 09:23:59] [debug] [task] destroy task=0x577e5f0 (task_id=0)
[2024/08/26 09:23:59] [debug] [out flush] cb_destroy coro_id=0
{"MESSAGE":"some value"}
{"date":1724664286.568816,"STRING1":"some","STRING2":"value","parsed":{"STRING1":"some","STRING2":"value"}}
[2024/08/26 09:24:47] [debug] [task] created task=0x7188dd0 id=0 OK
[2024/08/26 09:24:47] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/08/26 09:24:47] [debug] [out flush] cb_destroy coro_id=1
[2024/08/26 09:24:47] [debug] [task] destroy task=0x7188dd0 (task_id=0)
^C[2024/08/26 09:24:54] [engine] caught signal (SIGINT)
[2024/08/26 09:24:54] [ warn] [engine] service will shutdown in max 5 seconds
[2024/08/26 09:24:55] [ info] [engine] service has stopped (0 pending tasks)
[2024/08/26 09:24:55] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/08/26 09:24:55] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==3576== 
==3576== HEAP SUMMARY:
==3576==     in use at exit: 8,279 bytes in 3 blocks
==3576==   total heap usage: 2,511 allocs, 2,508 frees, 5,707,028 bytes allocated
==3576== 
==3576== LEAK SUMMARY:
==3576==    definitely lost: 79 bytes in 2 blocks
==3576==    indirectly lost: 8,200 bytes in 1 blocks
==3576==      possibly lost: 0 bytes in 0 blocks
==3576==    still reachable: 0 bytes in 0 blocks
==3576==         suppressed: 0 bytes in 0 blocks
==3576== Rerun with --leak-check=full to see details of leaked memory
==3576== 
==3576== For lists of detected and suppressed errors, rerun with: -s
==3576== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Athishpranav2003 · 2024-08-26T09:27:54Z

@edsiper, @leonardo-albertovich and others.
I have added all necessary logs and content. Could take a look at the PR?

Athishpranav2003 · 2024-08-26T11:31:01Z

I will check the failing UTs

Athishpranav2003 · 2024-08-26T13:19:23Z

@cosmo0920 corrected it, i missed 2 mem leaks

Athishpranav2003 · 2024-08-27T07:57:07Z

@cosmo0920 could you please approve the workflow?

Athishpranav2003 · 2024-09-02T09:08:52Z

@cosmo0920 addressed your comments

Athishpranav2003 · 2024-09-03T08:46:51Z

@cosmo0920 i guess the tests the are passing. I have addressed your comments as well. Anything else left?

Athishpranav2003 · 2024-09-11T12:54:00Z

hey @cosmo0920 any further updates?

plugins/filter_parser/filter_parser.c

Athishpranav2003 · 2024-09-12T05:43:53Z

@cosmo0920 addressed the comments

plugins/filter_parser/filter_parser.c

src/flb_msgpack_append_message.c

tests/runtime/filter_parser.c

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 · 2024-09-12T08:35:22Z

@cosmo0920 addressed the comments

Add arg for hash-value-field

c021e35

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners August 24, 2024 04:28

Athishpranav2003 marked this pull request as draft August 24, 2024 04:28

github-actions bot added the docs-required label Aug 24, 2024

Athishpranav2003 force-pushed the filter-parser-hash-key branch from 13f7b10 to 57e6b0d Compare August 24, 2024 04:56

Athishpranav2003 mentioned this pull request Aug 24, 2024

filter_parser: Add new param to doc fluent/fluent-bit-docs#1444

Open

Athishpranav2003 changed the title ~~Filter parser hash key~~ filter_parser: Hash_key for parsed result Aug 24, 2024

Athishpranav2003 marked this pull request as ready for review August 24, 2024 05:16

Athishpranav2003 temporarily deployed to pr August 26, 2024 10:12 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr August 26, 2024 10:32 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr August 26, 2024 10:33 — with GitHub Actions Inactive

Athishpranav2003 force-pushed the filter-parser-hash-key branch from 57e6b0d to 7fdf7fe Compare August 26, 2024 13:18

Athishpranav2003 temporarily deployed to pr August 27, 2024 14:49 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr August 27, 2024 15:09 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr August 27, 2024 15:10 — with GitHub Actions Inactive

Athishpranav2003 requested a review from cosmo0920 August 30, 2024 16:54

Athishpranav2003 temporarily deployed to pr September 3, 2024 05:54 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr September 3, 2024 06:19 — with GitHub Actions Inactive

cosmo0920 reviewed Sep 12, 2024

View reviewed changes

plugins/filter_parser/filter_parser.c Outdated Show resolved Hide resolved

Athishpranav2003 force-pushed the filter-parser-hash-key branch from 9ecd6cf to 3756d96 Compare September 12, 2024 05:41

Athishpranav2003 requested a review from cosmo0920 September 12, 2024 05:43

cosmo0920 requested changes Sep 12, 2024

View reviewed changes

plugins/filter_parser/filter_parser.c Outdated Show resolved Hide resolved

src/flb_msgpack_append_message.c Outdated Show resolved Hide resolved

tests/runtime/filter_parser.c Show resolved Hide resolved

Athishpranav2003 added 5 commits September 12, 2024 13:49

Code refactoring

a30a976

Signed-off-by: Athish Pranav D <[email protected]>

Add UTs

6114210

Signed-off-by: Athish Pranav D <[email protected]>

Made the param configurable

96b4589

Signed-off-by: Athish Pranav D <[email protected]>

Addressed comments

0e98334

Signed-off-by: Athish Pranav D <[email protected]>

Add seperate err msg for Unpack issue

6b35eb9

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 force-pushed the filter-parser-hash-key branch from 3756d96 to 6b35eb9 Compare September 12, 2024 08:33

cosmo0920 approved these changes Sep 12, 2024

View reviewed changes

cosmo0920 added this to the Fluent Bit v3.2.0 milestone Sep 12, 2024

Athishpranav2003 temporarily deployed to pr September 13, 2024 02:21 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr September 13, 2024 02:42 — with GitHub Actions Inactive

Athishpranav2003 temporarily deployed to pr September 13, 2024 02:43 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter_parser: Hash_key for parsed result #9274

filter_parser: Hash_key for parsed result #9274

Athishpranav2003 commented Aug 24, 2024 •

edited

Loading

Athishpranav2003 commented Aug 24, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 27, 2024

Athishpranav2003 commented Sep 2, 2024

Athishpranav2003 commented Sep 3, 2024

Athishpranav2003 commented Sep 11, 2024

Athishpranav2003 commented Sep 12, 2024

Athishpranav2003 commented Sep 12, 2024

filter_parser: Hash_key for parsed result #9274

Are you sure you want to change the base?

filter_parser: Hash_key for parsed result #9274

Conversation

Athishpranav2003 commented Aug 24, 2024 • edited Loading

Add Hash key for parsed value in Filter parser (similar to fluentd)

Athishpranav2003 commented Aug 24, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 26, 2024

Athishpranav2003 commented Aug 27, 2024

Athishpranav2003 commented Sep 2, 2024

Athishpranav2003 commented Sep 3, 2024

Athishpranav2003 commented Sep 11, 2024

Athishpranav2003 commented Sep 12, 2024

Athishpranav2003 commented Sep 12, 2024

Athishpranav2003 commented Aug 24, 2024 •

edited

Loading