out_doris: add new doris out plugin #9514

joker-star-l · 2024-10-23T10:22:39Z

Add new doris out plugin.

#9501

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

[INPUT]
    Name cpu
    Interval_Sec 3

[OUTPUT]
    name doris
    match *
    host localhost
    port 8030
    user admin
    password admin
    database d_fb
    table t_fb
    time_key date
    header columns date, cpu_p, timestamp=from_unixtime(date), log=cast(cpu_p as string)

Debug log output from testing the change

Fluent Bit v3.2.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/10/23 18:16:58] [ info] Configuration:
[2024/10/23 18:16:58] [ info]  flush time     | 1.000000 seconds
[2024/10/23 18:16:58] [ info]  grace          | 5 seconds
[2024/10/23 18:16:58] [ info]  daemon         | 0
[2024/10/23 18:16:58] [ info] ___________
[2024/10/23 18:16:58] [ info]  inputs:
[2024/10/23 18:16:58] [ info]      cpu
[2024/10/23 18:16:58] [ info] ___________
[2024/10/23 18:16:58] [ info]  filters:
[2024/10/23 18:16:58] [ info] ___________
[2024/10/23 18:16:58] [ info]  outputs:
[2024/10/23 18:16:58] [ info]      doris.0
[2024/10/23 18:16:58] [ info] ___________
[2024/10/23 18:16:58] [ info]  collectors:
[2024/10/23 18:16:58] [ info] [fluent bit] version=3.2.0, commit=1828228f55, pid=47353
[2024/10/23 18:16:58] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/10/23 18:16:58] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/10/23 18:16:58] [ info] [cmetrics] version=0.9.8
[2024/10/23 18:16:58] [ info] [ctraces ] version=0.5.7
[2024/10/23 18:16:58] [ info] [input:cpu:cpu.0] initializing
[2024/10/23 18:16:58] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2024/10/23 18:16:58] [debug] [cpu:cpu.0] created event channels: read=29 write=30
[2024/10/23 18:16:58] [debug] [doris:doris.0] created event channels: read=31 write=32
[2024/10/23 18:16:58] [ info] [output:doris:doris.0] worker #0 started
[2024/10/23 18:16:58] [ info] [output:doris:doris.0] worker #1 started
[2024/10/23 18:16:58] [ info] [sp] stream processor started
[2024/10/23 18:17:01] [debug] [task] created task=0x7f003c026da0 id=0 OK
[2024/10/23 18:17:01] [debug] [output:doris:doris.0] task_id=0 assigned to thread #0
[2024/10/23 18:17:01] [debug] [output:doris:doris.0] http body: [{"date":1729678620,"cpu_p":0.5925925925925926,"user_p":0.3333333333333333,"system_p":0.2592592592592593,"cpu0.p_cpu":1.0,"cpu0.p_user":0.0,"cpu0.p_system":1.0,"cpu1.p_cpu":0.3333333333333333,"cpu1.p_user":0.3333333333333333,"cpu1.p_system":0.0,"cpu2.p_cpu":1.0,"cpu2.p_user":0.6666666666666666,"cpu2.p_system":0.3333333333333333,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0,"cpu4.p_cpu":1.0,"cpu4.p_user":0.6666666666666666,"cpu4.p_system":0.3333333333333333,"cpu5.p_cpu":0.0,"cpu5.p_user":0.0,"cpu5.p_system":0.0,"cpu6.p_cpu":0.6666666666666666,"cpu6.p_user":0.0,"cpu6.p_system":0.6666666666666666,"cpu7.p_cpu":0.6666666666666666,"cpu7.p_user":0.3333333333333333,"cpu7.p_system":0.3333333333333333,"cpu8.p_cpu":0.6666666666666666,"cpu8.p_user":0.3333333333333333,"cpu8.p_system":0.3333333333333333,"cpu9.p_cpu":0.6666666666666666,"cpu9.p_user":0.6666666666666666,"cpu9.p_system":0.0,"cpu10.p_cpu":1.0,"cpu10.p_user":0.3333333333333333,"cpu10.p_system":0.6666666666666666,"cpu11.p_cpu":0.0,"cpu11.p_user":0.0,"cpu11.p_system":0.0,"cpu12.p_cpu":0.6666666666666666,"cpu12.p_user":0.3333333333333333,"cpu12.p_system":0.3333333333333333,"cpu13.p_cpu":0.0,"cpu13.p_user":0.0,"cpu13.p_system":0.0,"cpu14.p_cpu":0.6666666666666666,"cpu14.p_user":0.6666666666666666,"cpu14.p_system":0.0,"cpu15.p_cpu":0.0,"cpu15.p_user":0.0,"cpu15.p_system":0.0,"cpu16.p_cpu":0.6666666666666666,"cpu16.p_user":0.0,"cpu16.p_system":0.6666666666666666,"cpu17.p_cpu":0.6666666666666666,"cpu17.p_user":0.6666666666666666,"cpu17.p_system":0.0}]
[2024/10/23 18:17:01] [debug] [upstream] KA connection #63 to localhost:8030 is connected
[2024/10/23 18:17:01] [debug] [http_client] not using http_proxy for header
[2024/10/23 18:17:01] [debug] [http_client] server localhost:8030 will close connection #63
[2024/10/23 18:17:01] [debug] [output:doris:doris.0] localhost:8030, HTTP status=307
(null)

[2024/10/23 18:17:01] [debug] [upstream] KA connection #64 to 127.0.0.1:8040 is connected
[2024/10/23 18:17:01] [debug] [http_client] not using http_proxy for header
[2024/10/23 18:17:02] [debug] [output:doris:doris.0] 127.0.0.1:8040, HTTP status=200
{
    "TxnId": 28028,
    "Label": "ce92e8a5-c4d6-430e-8e0d-3298f2f9c738",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1525,
    "LoadTimeMs": 218,
    "BeginTxnTimeMs": 17,
    "StreamLoadPutTimeMs": 127,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 24,
    "CommitAndPublishTimeMs": 49
}


[2024/10/23 18:17:02] [debug] [upstream] KA connection #64 to 127.0.0.1:8040 is now available
[2024/10/23 18:17:02] [debug] [out flush] cb_destroy coro_id=0
[2024/10/23 18:17:02] [debug] [task] destroy task=0x7f003c026da0 (task_id=0)
[2024/10/23 18:17:04] [debug] [task] created task=0x7f003c0670d0 id=0 OK
[2024/10/23 18:17:04] [debug] [output:doris:doris.0] task_id=0 assigned to thread #1
[2024/10/23 18:17:04] [debug] [output:doris:doris.0] http body: [{"date":1729678623,"cpu_p":3.444444444444445,"user_p":3.0,"system_p":0.4444444444444444,"cpu0.p_cpu":8.666666666666666,"cpu0.p_user":8.0,"cpu0.p_system":0.6666666666666666,"cpu1.p_cpu":0.6666666666666666,"cpu1.p_user":0.6666666666666666,"cpu1.p_system":0.0,"cpu2.p_cpu":5.0,"cpu2.p_user":4.333333333333333,"cpu2.p_system":0.6666666666666666,"cpu3.p_cpu":1.333333333333333,"cpu3.p_user":1.333333333333333,"cpu3.p_system":0.0,"cpu4.p_cpu":12.33333333333333,"cpu4.p_user":11.33333333333333,"cpu4.p_system":1.0,"cpu5.p_cpu":0.0,"cpu5.p_user":0.0,"cpu5.p_system":0.0,"cpu6.p_cpu":1.0,"cpu6.p_user":0.3333333333333333,"cpu6.p_system":0.6666666666666666,"cpu7.p_cpu":2.0,"cpu7.p_user":2.0,"cpu7.p_system":0.0,"cpu8.p_cpu":6.666666666666667,"cpu8.p_user":6.0,"cpu8.p_system":0.6666666666666666,"cpu9.p_cpu":5.0,"cpu9.p_user":5.0,"cpu9.p_system":0.0,"cpu10.p_cpu":1.333333333333333,"cpu10.p_user":0.6666666666666666,"cpu10.p_system":0.6666666666666666,"cpu11.p_cpu":0.0,"cpu11.p_user":0.0,"cpu11.p_system":0.0,"cpu12.p_cpu":4.333333333333333,"cpu12.p_user":3.666666666666667,"cpu12.p_system":0.6666666666666666,"cpu13.p_cpu":0.6666666666666666,"cpu13.p_user":0.3333333333333333,"cpu13.p_system":0.3333333333333333,"cpu14.p_cpu":3.666666666666667,"cpu14.p_user":2.333333333333333,"cpu14.p_system":1.333333333333333,"cpu15.p_cpu":0.0,"cpu15.p_user":0.0,"cpu15.p_system":0.0,"cpu16.p_cpu":3.333333333333333,"cpu16.p_user":3.333333333333333,"cpu16.p_system":0.0,"cpu17.p_cpu":6.0,"cpu17.p_user":4.666666666666667,"cpu17.p_system":1.333333333333333}]
[2024/10/23 18:17:04] [debug] [upstream] KA connection #63 to localhost:8030 is connected
[2024/10/23 18:17:04] [debug] [http_client] not using http_proxy for header
[2024/10/23 18:17:04] [debug] [http_client] server localhost:8030 will close connection #63
[2024/10/23 18:17:04] [debug] [output:doris:doris.0] localhost:8030, HTTP status=307
(null)

[2024/10/23 18:17:04] [debug] [upstream] KA connection #64 to 127.0.0.1:8040 is connected
[2024/10/23 18:17:04] [debug] [http_client] not using http_proxy for header
[2024/10/23 18:17:04] [debug] [output:doris:doris.0] 127.0.0.1:8040, HTTP status=200
{
    "TxnId": 28029,
    "Label": "1be0852a-1bbb-4029-a643-11c3cc2b426e",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1537,
    "LoadTimeMs": 55,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 8,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 19,
    "CommitAndPublishTimeMs": 26
}


[2024/10/23 18:17:04] [debug] [upstream] KA connection #64 to 127.0.0.1:8040 is now available
[2024/10/23 18:17:04] [debug] [out flush] cb_destroy coro_id=0
[2024/10/23 18:17:04] [debug] [task] destroy task=0x7f003c0670d0 (task_id=0)
[2024/10/23 18:17:07] [debug] [task] created task=0x7f003c027190 id=0 OK
[2024/10/23 18:17:07] [debug] [output:doris:doris.0] task_id=0 assigned to thread #0
[2024/10/23 18:17:07] [debug] [output:doris:doris.0] http body: [{"date":1729678626,"cpu_p":14.07407407407407,"user_p":12.40740740740741,"system_p":1.666666666666667,"cpu0.p_cpu":13.66666666666667,"cpu0.p_user":10.66666666666667,"cpu0.p_system":3.0,"cpu1.p_cpu":14.0,"cpu1.p_user":12.0,"cpu1.p_system":2.0,"cpu2.p_cpu":15.0,"cpu2.p_user":14.0,"cpu2.p_system":1.0,"cpu3.p_cpu":23.66666666666667,"cpu3.p_user":23.0,"cpu3.p_system":0.6666666666666666,"cpu4.p_cpu":22.0,"cpu4.p_user":19.0,"cpu4.p_system":3.0,"cpu5.p_cpu":5.333333333333333,"cpu5.p_user":4.333333333333333,"cpu5.p_system":1.0,"cpu6.p_cpu":20.33333333333333,"cpu6.p_user":20.0,"cpu6.p_system":0.3333333333333333,"cpu7.p_cpu":8.666666666666666,"cpu7.p_user":7.0,"cpu7.p_system":1.666666666666667,"cpu8.p_cpu":16.0,"cpu8.p_user":12.33333333333333,"cpu8.p_system":3.666666666666667,"cpu9.p_cpu":8.333333333333334,"cpu9.p_user":8.333333333333334,"cpu9.p_system":0.0,"cpu10.p_cpu":8.0,"cpu10.p_user":5.666666666666667,"cpu10.p_system":2.333333333333333,"cpu11.p_cpu":16.66666666666667,"cpu11.p_user":15.0,"cpu11.p_system":1.666666666666667,"cpu12.p_cpu":16.0,"cpu12.p_user":12.33333333333333,"cpu12.p_system":3.666666666666667,"cpu13.p_cpu":17.66666666666667,"cpu13.p_user":16.66666666666667,"cpu13.p_system":1.0,"cpu14.p_cpu":8.333333333333334,"cpu14.p_user":8.0,"cpu14.p_system":0.3333333333333333,"cpu15.p_cpu":8.0,"cpu15.p_user":7.333333333333333,"cpu15.p_system":0.6666666666666666,"cpu16.p_cpu":9.0,"cpu16.p_user":6.666666666666667,"cpu16.p_system":2.333333333333333,"cpu17.p_cpu":22.66666666666667,"cpu17.p_user":21.0,"cpu17.p_system":1.666666666666667}]
[2024/10/23 18:17:07] [debug] [upstream] KA connection #63 to localhost:8030 is connected
[2024/10/23 18:17:07] [debug] [http_client] not using http_proxy for header
[2024/10/23 18:17:07] [debug] [http_client] server localhost:8030 will close connection #63
[2024/10/23 18:17:07] [debug] [output:doris:doris.0] localhost:8030, HTTP status=307
(null)

[2024/10/23 18:17:07] [debug] [upstream] KA connection #64 to 127.0.0.1:8040 is connected
[2024/10/23 18:17:07] [debug] [http_client] not using http_proxy for header
[2024/10/23 18:17:07] [debug] [output:doris:doris.0] 127.0.0.1:8040, HTTP status=200
{
    "TxnId": 28030,
    "Label": "8c20c58f-ffd6-4aa1-8946-58c72a6b761a",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1553,
    "LoadTimeMs": 53,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 23,
    "CommitAndPublishTimeMs": 25
}


[2024/10/23 18:17:07] [debug] [upstream] KA connection #64 to 127.0.0.1:8040 is now available
[2024/10/23 18:17:07] [debug] [out flush] cb_destroy coro_id=1
[2024/10/23 18:17:07] [debug] [task] destroy task=0x7f003c027190 (task_id=0)
^C[2024/10/23 18:17:09] [engine] caught signal (SIGINT)
[2024/10/23 18:17:09] [ warn] [engine] service will shutdown in max 5 seconds
[2024/10/23 18:17:09] [ info] [input] pausing cpu.0
[2024/10/23 18:17:09] [ info] [engine] service has stopped (0 pending tasks)
[2024/10/23 18:17:09] [ info] [input] pausing cpu.0
[2024/10/23 18:17:09] [ info] [output:doris:doris.0] thread worker #0 stopping...
[2024/10/23 18:17:09] [ info] [output:doris:doris.0] thread worker #0 stopped
[2024/10/23 18:17:09] [ info] [output:doris:doris.0] thread worker #1 stopping...
[2024/10/23 18:17:09] [ info] [output:doris:doris.0] thread worker #1 stopped

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent-bit-docs/pull/1483

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

patrick-stephens · 2024-10-28T11:58:31Z

Does this need additional dependencies?
Do we need to update the packaging builds for the releases for each target (specifically older ones like CentOS 7 if new deps are required)?

joker-star-l · 2024-10-30T20:43:34Z

Does this need additional dependencies? Do we need to update the packaging builds for the releases for each target (specifically older ones like CentOS 7 if new deps are required)?

This component has no new additional dependencies.

Signed-off-by: composer <[email protected]>

joker-star-l · 2024-10-31T18:29:51Z

It seems that msvc does not have __sync_fetch_and_add function. <stdatomic.h> is not available until after c11. How can I use a general atomic counting method? @patrick-stephens

patrick-stephens · 2024-11-01T10:04:32Z

It seems that msvc does not have __sync_fetch_and_add function. <stdatomic.h> is not available until after c11. How can I use a general atomic counting method? @patrick-stephens

Yes, you have to support all legacy targets I'm afraid so the various vendored libraries have atomic support in place. I'm not sure if there is a general way in Fluent Bit to do it @cosmo0920 ?

cosmo0920 · 2024-11-01T13:48:08Z

It seems that msvc does not have __sync_fetch_and_add function. <stdatomic.h> is not available until after c11. How can I use a general atomic counting method? @patrick-stephens

Yes, you have to support all legacy targets I'm afraid so the various vendored libraries have atomic support in place. I'm not sure if there is a general way in Fluent Bit to do it @cosmo0920 ?

We need to use InterlockedAdd in Windows for the equivalent operation of __sync_fetch_and_add.

Signed-off-by: composer <[email protected]>

joker-star-l · 2024-11-02T03:11:09Z

It seems that msvc does not have __sync_fetch_and_add function. <stdatomic.h> is not available until after c11. How can I use a general atomic counting method? @patrick-stephens

Yes, you have to support all legacy targets I'm afraid so the various vendored libraries have atomic support in place. I'm not sure if there is a general way in Fluent Bit to do it @cosmo0920 ?

We need to use InterlockedAdd in Windows for the equivalent operation of __sync_fetch_and_add.

Thank you both!

plugins/out_doris/doris.c

Signed-off-by: composer <[email protected]>

joker-star-l · 2024-11-12T13:37:37Z

Could someone help me run the ci?

patrick-stephens · 2024-11-12T14:13:29Z

@joker-star-l I've triggered it, it is limited to keep costs down and for security best practice

joker-star-l requested review from fujimotos, niedbalski, patrick-stephens, celalettin1286, edsiper, leonardo-albertovich and koleini as code owners October 23, 2024 10:22

github-actions bot added the docs-required label Oct 23, 2024

joker-star-l force-pushed the dev branch from 1828228 to 492f5fd Compare October 23, 2024 10:24

joker-star-l mentioned this pull request Oct 23, 2024

out_doris: doc for new doris out plugin fluent/fluent-bit-docs#1483

Open

patrick-stephens added the ok-package-test Run PR packaging tests label Oct 28, 2024

joker-star-l temporarily deployed to pr October 28, 2024 11:58 — with GitHub Actions Inactive

joker-star-l temporarily deployed to pr October 28, 2024 12:20 — with GitHub Actions Inactive

joker-star-l added 7 commits October 31, 2024 04:46

feat: add doris out plugin

d47f9ce

Signed-off-by: composer <[email protected]>

feat: add unit test

b81a93d

Signed-off-by: composer <[email protected]>

feat: support http redirect

e504e53

Signed-off-by: composer <[email protected]>

feat: add doris build config

daa4540

Signed-off-by: composer <[email protected]>

feat: add more doris config

e24c442

Signed-off-by: composer <[email protected]>

add label config

7b44ce6

Signed-off-by: composer <[email protected]>

fix: unit test

e8a2a60

Signed-off-by: composer <[email protected]>

joker-star-l force-pushed the dev branch from ba91eb3 to e8a2a60 Compare October 30, 2024 20:46

joker-star-l added 3 commits October 31, 2024 05:10

fix: mem leak

c02e1c7

Signed-off-by: composer <[email protected]>

fix: typo

14f2526

Signed-off-by: composer <[email protected]>

fix: only calculate speed when interval > 0

affcdc5

Signed-off-by: composer <[email protected]>

joker-star-l had a problem deploying to pr October 31, 2024 10:44 — with GitHub Actions Failure

fix: windows atomic

0dbeebf

Signed-off-by: composer <[email protected]>

cosmo0920 reviewed Nov 5, 2024

View reviewed changes

plugins/out_doris/doris.c Outdated Show resolved Hide resolved

joker-star-l temporarily deployed to pr November 5, 2024 10:14 — with GitHub Actions Inactive

joker-star-l temporarily deployed to pr November 5, 2024 10:36 — with GitHub Actions Inactive

refactor: atomic add

0bee5f4

Signed-off-by: composer <[email protected]>

joker-star-l temporarily deployed to pr November 12, 2024 14:13 — with GitHub Actions Inactive

joker-star-l temporarily deployed to pr November 12, 2024 14:38 — with GitHub Actions Inactive

joker-star-l temporarily deployed to pr November 12, 2024 14:39 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

out_doris: add new doris out plugin #9514

out_doris: add new doris out plugin #9514

joker-star-l commented Oct 23, 2024 •

edited

Loading

patrick-stephens commented Oct 28, 2024

joker-star-l commented Oct 30, 2024

joker-star-l commented Oct 31, 2024 •

edited

Loading

patrick-stephens commented Nov 1, 2024

cosmo0920 commented Nov 1, 2024

joker-star-l commented Nov 2, 2024

joker-star-l commented Nov 12, 2024

patrick-stephens commented Nov 12, 2024

out_doris: add new doris out plugin #9514

Are you sure you want to change the base?

out_doris: add new doris out plugin #9514

Conversation

joker-star-l commented Oct 23, 2024 • edited Loading

#9501

patrick-stephens commented Oct 28, 2024

joker-star-l commented Oct 30, 2024

joker-star-l commented Oct 31, 2024 • edited Loading

patrick-stephens commented Nov 1, 2024

cosmo0920 commented Nov 1, 2024

joker-star-l commented Nov 2, 2024

joker-star-l commented Nov 12, 2024

patrick-stephens commented Nov 12, 2024

joker-star-l commented Oct 23, 2024 •

edited

Loading

joker-star-l commented Oct 31, 2024 •

edited

Loading