add support for org level APM_TRACING configs #9360

shatzi · 2025-08-13T18:50:26Z

What Does This Do

This improve APM_TRACING config handling by removing the assumption we only get one config at a time. In order to handle multiple configs, we merge them based on priority, when more specific configs get precedence over more generic one (org level configs)

Motivation

This come to enable Exception Replay, Code Origin and Distributed Debugging slow rollout.

Additional Notes

Please see RFC for merging APM_TRACING

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: DEBUG-4326

github-actions · 2025-08-13T18:50:40Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

datadog-datadog-prod-us1 · 2025-08-13T19:09:15Z

🎯 Code Coverage
• Patch Coverage: 98.65%
• Total Coverage: 59.71% (+2.08%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a93db23 | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-08-13T19:25:23Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	shatzi/DEBUG-4326/apm-tracing-override-merging
git_commit_date	1756901714	1756909355
git_commit_sha	`030f9bb`	`a93db23`
release_version	1.53.0-SNAPSHOT~030f9bbb36	1.53.0-SNAPSHOT~a93db23764

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1756911148	1756911148
ci_job_id	1111263806	1111263806
ci_pipeline_id	75424544	75424544
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-sle0lp4z 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-sle0lp4z 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 10 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1051335
Total [baseline] (10.716 s) : 0, 10715783
Agent [candidate] (1.056 s) : 0, 1055912
Total [candidate] (10.916 s) : 0, 10915927
section appsec
Agent [baseline] (1.225 s) : 0, 1225497
Total [baseline] (10.829 s) : 0, 10829431
Agent [candidate] (1.227 s) : 0, 1226764
Total [candidate] (10.834 s) : 0, 10834328
section iast
Agent [baseline] (1.182 s) : 0, 1182123
Total [baseline] (11.014 s) : 0, 11014006
Agent [candidate] (1.183 s) : 0, 1183026
Total [candidate] (10.993 s) : 0, 10993273
section profiling
Agent [baseline] (1.208 s) : 0, 1208348
Total [baseline] (10.99 s) : 0, 10990469
Agent [candidate] (1.2 s) : 0, 1199986
Total [candidate] (10.916 s) : 0, 10915503

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	appsec	1.225 s	174.162 ms (16.6%)
Agent	iast	1.182 s	130.788 ms (12.4%)
Agent	profiling	1.208 s	157.013 ms (14.9%)
Total	tracing	10.716 s	-
Total	appsec	10.829 s	113.648 ms (1.1%)
Total	iast	11.014 s	298.223 ms (2.8%)
Total	profiling	10.99 s	274.686 ms (2.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.227 s	170.852 ms (16.2%)
Agent	iast	1.183 s	127.114 ms (12.0%)
Agent	profiling	1.2 s	144.074 ms (13.6%)
Total	tracing	10.916 s	-
Total	appsec	10.834 s	-81.599 ms (-0.7%)
Total	iast	10.993 s	77.346 ms (0.7%)
Total	profiling	10.916 s	-424.362 µs (-0.0%)

gantt
    title petclinic - break down per module: candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.473 ms) : 0, 1473
crashtracking [candidate] (1.471 ms) : 0, 1471
BytebuddyAgent [baseline] (734.316 ms) : 0, 734316
BytebuddyAgent [candidate] (737.784 ms) : 0, 737784
GlobalTracer [baseline] (243.048 ms) : 0, 243048
GlobalTracer [candidate] (245.968 ms) : 0, 245968
AppSec [baseline] (30.103 ms) : 0, 30103
AppSec [candidate] (30.446 ms) : 0, 30446
Debugger [baseline] (6.119 ms) : 0, 6119
Debugger [candidate] (6.117 ms) : 0, 6117
Remote Config [baseline] (671.881 µs) : 0, 672
Remote Config [candidate] (685.343 µs) : 0, 685
Telemetry [baseline] (14.451 ms) : 0, 14451
Telemetry [candidate] (12.33 ms) : 0, 12330
section appsec
crashtracking [baseline] (1.454 ms) : 0, 1454
crashtracking [candidate] (1.456 ms) : 0, 1456
BytebuddyAgent [baseline] (756.783 ms) : 0, 756783
BytebuddyAgent [candidate] (757.206 ms) : 0, 757206
GlobalTracer [baseline] (235.913 ms) : 0, 235913
GlobalTracer [candidate] (237.073 ms) : 0, 237073
IAST [baseline] (23.511 ms) : 0, 23511
IAST [candidate] (23.524 ms) : 0, 23524
AppSec [baseline] (170.236 ms) : 0, 170236
AppSec [candidate] (170.662 ms) : 0, 170662
Debugger [baseline] (5.787 ms) : 0, 5787
Debugger [candidate] (5.835 ms) : 0, 5835
Remote Config [baseline] (637.02 µs) : 0, 637
Remote Config [candidate] (637.535 µs) : 0, 638
Telemetry [baseline] (10.03 ms) : 0, 10030
Telemetry [candidate] (9.24 ms) : 0, 9240
section iast
crashtracking [baseline] (1.45 ms) : 0, 1450
crashtracking [candidate] (1.446 ms) : 0, 1446
BytebuddyAgent [baseline] (852.846 ms) : 0, 852846
BytebuddyAgent [candidate] (852.596 ms) : 0, 852596
GlobalTracer [baseline] (233.402 ms) : 0, 233402
GlobalTracer [candidate] (235.209 ms) : 0, 235209
IAST [baseline] (29.025 ms) : 0, 29025
IAST [candidate] (31.291 ms) : 0, 31291
AppSec [baseline] (27.686 ms) : 0, 27686
AppSec [candidate] (26.766 ms) : 0, 26766
Debugger [baseline] (7.617 ms) : 0, 7617
Debugger [candidate] (5.704 ms) : 0, 5704
Remote Config [baseline] (612.254 µs) : 0, 612
Remote Config [candidate] (601.11 µs) : 0, 601
Telemetry [baseline] (8.384 ms) : 0, 8384
Telemetry [candidate] (8.265 ms) : 0, 8265
section profiling
crashtracking [baseline] (1.457 ms) : 0, 1457
crashtracking [candidate] (1.427 ms) : 0, 1427
BytebuddyAgent [baseline] (770.194 ms) : 0, 770194
BytebuddyAgent [candidate] (763.137 ms) : 0, 763137
GlobalTracer [baseline] (224.141 ms) : 0, 224141
GlobalTracer [candidate] (223.841 ms) : 0, 223841
AppSec [baseline] (30.437 ms) : 0, 30437
AppSec [candidate] (29.91 ms) : 0, 29910
Debugger [baseline] (6.309 ms) : 0, 6309
Debugger [candidate] (6.305 ms) : 0, 6305
Remote Config [baseline] (709.788 µs) : 0, 710
Remote Config [candidate] (706.689 µs) : 0, 707
Telemetry [baseline] (16.595 ms) : 0, 16595
Telemetry [candidate] (16.353 ms) : 0, 16353
ProfilingAgent [baseline] (107.547 ms) : 0, 107547
ProfilingAgent [candidate] (107.839 ms) : 0, 107839
Profiling [baseline] (108.19 ms) : 0, 108190
Profiling [candidate] (108.511 ms) : 0, 108511

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1049535
Total [baseline] (8.662 s) : 0, 8661606
Agent [candidate] (1.049 s) : 0, 1049471
Total [candidate] (8.64 s) : 0, 8639946
section iast
Agent [baseline] (1.196 s) : 0, 1196209
Total [baseline] (9.413 s) : 0, 9413319
Agent [candidate] (1.181 s) : 0, 1180662
Total [candidate] (9.471 s) : 0, 9470840

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.05 s	-
Agent	iast	1.196 s	146.674 ms (14.0%)
Total	tracing	8.662 s	-
Total	iast	9.413 s	751.714 ms (8.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	iast	1.181 s	131.191 ms (12.5%)
Total	tracing	8.64 s	-
Total	iast	9.471 s	830.895 ms (9.6%)

gantt
    title insecure-bank - break down per module: candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (734.06 ms) : 0, 734060
BytebuddyAgent [candidate] (733.509 ms) : 0, 733509
GlobalTracer [baseline] (243.043 ms) : 0, 243043
GlobalTracer [candidate] (244.347 ms) : 0, 244347
AppSec [baseline] (30.279 ms) : 0, 30279
AppSec [candidate] (30.137 ms) : 0, 30137
Debugger [baseline] (6.079 ms) : 0, 6079
Debugger [candidate] (6.123 ms) : 0, 6123
Remote Config [baseline] (668.044 µs) : 0, 668
Remote Config [candidate] (675.739 µs) : 0, 676
Telemetry [baseline] (12.308 ms) : 0, 12308
Telemetry [candidate] (12.233 ms) : 0, 12233
section iast
crashtracking [baseline] (1.491 ms) : 0, 1491
crashtracking [candidate] (1.454 ms) : 0, 1454
BytebuddyAgent [baseline] (865.212 ms) : 0, 865212
BytebuddyAgent [candidate] (851.244 ms) : 0, 851244
GlobalTracer [baseline] (234.196 ms) : 0, 234196
GlobalTracer [candidate] (234.451 ms) : 0, 234451
IAST [baseline] (28.058 ms) : 0, 28058
IAST [candidate] (31.847 ms) : 0, 31847
AppSec [baseline] (29.11 ms) : 0, 29110
AppSec [candidate] (25.979 ms) : 0, 25979
Debugger [baseline] (6.701 ms) : 0, 6701
Debugger [candidate] (5.782 ms) : 0, 5782
Remote Config [baseline] (627.294 µs) : 0, 627
Remote Config [candidate] (601.799 µs) : 0, 602
Telemetry [baseline] (9.509 ms) : 0, 9509
Telemetry [candidate] (8.362 ms) : 0, 8362

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	shatzi/DEBUG-4326/apm-tracing-override-merging
git_commit_date	1756901714	1756909355
git_commit_sha	`030f9bb`	`a93db23`
release_version	1.53.0-SNAPSHOT~030f9bbb36	1.53.0-SNAPSHOT~a93db23764

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1756910934	1756910934
ci_job_id	1111263807	1111263807
ci_pipeline_id	75424544	75424544
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-j0e6ri76 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-j0e6ri76 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:tracing:high_load	worse [+342.709µs; +584.198µs] or [+4.618%; +7.872%]	unstable [-110.617op/s; +38.430op/s] or [-17.735%; +6.161%]	7.884ms	587.625op/s	7.421ms	623.719op/s
scenario:load:insecure-bank:profiling:high_load	worse [+572.173µs; +871.003µs] or [+6.899%; +10.503%]	unstable [-113.919op/s; +25.982op/s] or [-20.381%; +4.648%]	9.015ms	514.969op/s	8.293ms	558.938op/s
scenario:load:insecure-bank:iast_FULL:high_load	better [-880.438µs; -298.027µs] or [-6.316%; -2.138%]	unstable [-22.904op/s; +52.717op/s] or [-6.855%; +15.778%]	13.351ms	349.031op/s	13.940ms	334.125op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.265 ms) : 4212, 4317
.   : milestone, 4265,
iast (9.469 ms) : 9306, 9631
.   : milestone, 9469,
iast_FULL (13.94 ms) : 13664, 14217
.   : milestone, 13940,
iast_GLOBAL (10.211 ms) : 10027, 10395
.   : milestone, 10211,
profiling (8.293 ms) : 8155, 8431
.   : milestone, 8293,
tracing (7.421 ms) : 7311, 7531
.   : milestone, 7421,
section candidate
no_agent (4.397 ms) : 4347, 4447
.   : milestone, 4397,
iast (9.316 ms) : 9160, 9472
.   : milestone, 9316,
iast_FULL (13.351 ms) : 13086, 13616
.   : milestone, 13351,
iast_GLOBAL (9.967 ms) : 9783, 10151
.   : milestone, 9967,
profiling (9.015 ms) : 8875, 9155
.   : milestone, 9015,
tracing (7.884 ms) : 7770, 7999
.   : milestone, 7884,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.265 ms [4.212 ms, 4.317 ms]	-
iast	9.469 ms [9.306 ms, 9.631 ms]	5.204 ms (122.0%)
iast_FULL	13.94 ms [13.664 ms, 14.217 ms]	9.676 ms (226.9%)
iast_GLOBAL	10.211 ms [10.027 ms, 10.395 ms]	5.946 ms (139.4%)
profiling	8.293 ms [8.155 ms, 8.431 ms]	4.028 ms (94.5%)
tracing	7.421 ms [7.311 ms, 7.531 ms]	3.156 ms (74.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.397 ms [4.347 ms, 4.447 ms]	-
iast	9.316 ms [9.16 ms, 9.472 ms]	4.919 ms (111.9%)
iast_FULL	13.351 ms [13.086 ms, 13.616 ms]	8.954 ms (203.6%)
iast_GLOBAL	9.967 ms [9.783 ms, 10.151 ms]	5.569 ms (126.7%)
profiling	9.015 ms [8.875 ms, 9.155 ms]	4.617 ms (105.0%)
tracing	7.884 ms [7.77 ms, 7.999 ms]	3.487 ms (79.3%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36
    dateFormat X
    axisFormat %s
section baseline
no_agent (37.723 ms) : 37417, 38029
.   : milestone, 37723,
appsec (47.625 ms) : 47197, 48052
.   : milestone, 47625,
code_origins (45.419 ms) : 45043, 45796
.   : milestone, 45419,
iast (45.288 ms) : 44893, 45682
.   : milestone, 45288,
profiling (47.574 ms) : 47076, 48073
.   : milestone, 47574,
tracing (44.91 ms) : 44540, 45281
.   : milestone, 44910,
section candidate
no_agent (36.982 ms) : 36689, 37275
.   : milestone, 36982,
appsec (47.914 ms) : 47480, 48347
.   : milestone, 47914,
code_origins (45.628 ms) : 45245, 46011
.   : milestone, 45628,
iast (44.518 ms) : 44121, 44916
.   : milestone, 44518,
profiling (47.699 ms) : 47258, 48140
.   : milestone, 47699,
tracing (45.018 ms) : 44621, 45415
.   : milestone, 45018,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.723 ms [37.417 ms, 38.029 ms]	-
appsec	47.625 ms [47.197 ms, 48.052 ms]	9.902 ms (26.2%)
code_origins	45.419 ms [45.043 ms, 45.796 ms]	7.696 ms (20.4%)
iast	45.288 ms [44.893 ms, 45.682 ms]	7.564 ms (20.1%)
profiling	47.574 ms [47.076 ms, 48.073 ms]	9.851 ms (26.1%)
tracing	44.91 ms [44.54 ms, 45.281 ms]	7.187 ms (19.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.982 ms [36.689 ms, 37.275 ms]	-
appsec	47.914 ms [47.48 ms, 48.347 ms]	10.932 ms (29.6%)
code_origins	45.628 ms [45.245 ms, 46.011 ms]	8.646 ms (23.4%)
iast	44.518 ms [44.121 ms, 44.916 ms]	7.536 ms (20.4%)
profiling	47.699 ms [47.258 ms, 48.14 ms]	10.717 ms (29.0%)
tracing	45.018 ms [44.621 ms, 45.415 ms]	8.036 ms (21.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	shatzi/DEBUG-4326/apm-tracing-override-merging
git_commit_date	1756901714	1756909355
git_commit_sha	`030f9bb`	`a93db23`
release_version	1.53.0-SNAPSHOT~030f9bbb36	1.53.0-SNAPSHOT~a93db23764

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1756911345	1756911345
ci_job_id	1111263808	1111263808
ci_pipeline_id	75424544	75424544
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-f4fei027 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-f4fei027 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.477 ms) : 1465, 1488
.   : milestone, 1477,
appsec (2.482 ms) : 2428, 2535
.   : milestone, 2482,
iast (2.207 ms) : 2144, 2270
.   : milestone, 2207,
iast_GLOBAL (2.259 ms) : 2195, 2322
.   : milestone, 2259,
profiling (2.051 ms) : 2001, 2102
.   : milestone, 2051,
tracing (2.034 ms) : 1984, 2083
.   : milestone, 2034,
section candidate
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (3.609 ms) : 3396, 3822
.   : milestone, 3609,
iast (2.211 ms) : 2148, 2274
.   : milestone, 2211,
iast_GLOBAL (2.247 ms) : 2184, 2311
.   : milestone, 2247,
profiling (2.065 ms) : 2013, 2117
.   : milestone, 2065,
tracing (2.03 ms) : 1980, 2079
.   : milestone, 2030,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.465 ms, 1.488 ms]	-
appsec	2.482 ms [2.428 ms, 2.535 ms]	1.005 ms (68.0%)
iast	2.207 ms [2.144 ms, 2.27 ms]	730.169 µs (49.4%)
iast_GLOBAL	2.259 ms [2.195 ms, 2.322 ms]	781.626 µs (52.9%)
profiling	2.051 ms [2.001 ms, 2.102 ms]	574.317 µs (38.9%)
tracing	2.034 ms [1.984 ms, 2.083 ms]	556.754 µs (37.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.479 ms [1.468 ms, 1.491 ms]	-
appsec	3.609 ms [3.396 ms, 3.822 ms]	2.129 ms (143.9%)
iast	2.211 ms [2.148 ms, 2.274 ms]	731.763 µs (49.5%)
iast_GLOBAL	2.247 ms [2.184 ms, 2.311 ms]	767.822 µs (51.9%)
profiling	2.065 ms [2.013 ms, 2.117 ms]	585.363 µs (39.6%)
tracing	2.03 ms [1.98 ms, 2.079 ms]	550.135 µs (37.2%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~a93db23764, baseline=1.53.0-SNAPSHOT~030f9bbb36
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.321 s) : 15321000, 15321000
.   : milestone, 15321000,
appsec (14.826 s) : 14826000, 14826000
.   : milestone, 14826000,
iast (18.472 s) : 18472000, 18472000
.   : milestone, 18472000,
iast_GLOBAL (17.783 s) : 17783000, 17783000
.   : milestone, 17783000,
profiling (15.467 s) : 15467000, 15467000
.   : milestone, 15467000,
tracing (14.801 s) : 14801000, 14801000
.   : milestone, 14801000,
section candidate
no_agent (15.217 s) : 15217000, 15217000
.   : milestone, 15217000,
appsec (14.782 s) : 14782000, 14782000
.   : milestone, 14782000,
iast (18.64 s) : 18640000, 18640000
.   : milestone, 18640000,
iast_GLOBAL (18.234 s) : 18234000, 18234000
.   : milestone, 18234000,
profiling (15.402 s) : 15402000, 15402000
.   : milestone, 15402000,
tracing (14.848 s) : 14848000, 14848000
.   : milestone, 14848000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.321 s [15.321 s, 15.321 s]	-
appsec	14.826 s [14.826 s, 14.826 s]	-495.0 ms (-3.2%)
iast	18.472 s [18.472 s, 18.472 s]	3.151 s (20.6%)
iast_GLOBAL	17.783 s [17.783 s, 17.783 s]	2.462 s (16.1%)
profiling	15.467 s [15.467 s, 15.467 s]	146.0 ms (1.0%)
tracing	14.801 s [14.801 s, 14.801 s]	-520.0 ms (-3.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.217 s [15.217 s, 15.217 s]	-
appsec	14.782 s [14.782 s, 14.782 s]	-435.0 ms (-2.9%)
iast	18.64 s [18.64 s, 18.64 s]	3.423 s (22.5%)
iast_GLOBAL	18.234 s [18.234 s, 18.234 s]	3.017 s (19.8%)
profiling	15.402 s [15.402 s, 15.402 s]	185.0 ms (1.2%)
tracing	14.848 s [14.848 s, 14.848 s]	-369.0 ms (-2.4%)

…ging

mhlidd

Overall looks good to me. Just some questions and cleanup nits

dd-trace-core/src/main/java/datadog/trace/core/TracingConfigPoller.java

mhlidd · 2025-08-22T15:24:29Z

dd-trace-core/src/test/groovy/datadog/trace/core/TracingConfigPollerTest.groovy

+    configOverrides.isClusterTarget() == false
+  }
+
+  def "test org level + service level config priority comparison"() {
+    setup:
+    def orgConfig = new TracingConfigPoller.ConfigOverrides()
+    orgConfig.libConfig = new TracingConfigPoller.LibConfig(
+      tracingEnabled: true,


Can we combine these tests into one? You can use a where block to specify preset values like this example, and reuse the shared testing logic

change tests to be more compact

dd-trace-core/src/test/groovy/datadog/trace/core/TracingConfigPollerTest.groovy

… logic This is still in development for Java and Python here: - DataDog/dd-trace-java#9360 - DataDog/dd-trace-py#14364 But we want to have system-tests upfront to ensure it's functioning correctly for Sept releases. Refs: DEBUG-4399

This is still in development for Java and Python here: - DataDog/dd-trace-java#9360 - DataDog/dd-trace-py#14364 But we want to have system-tests upfront to ensure it's functioning correctly for Sept releases. Refs: DEBUG-4399

…ging

mhlidd

LGTM from a Java perspective. Please make sure to have system-tests passing with this change prior to release. :)

…ging

add support for org level APM_TRACING configs

97b6e05

shatzi requested review from a team as code owners August 13, 2025 18:50

shatzi requested review from dougqh and removed request for a team August 13, 2025 18:50

shatzi marked this pull request as draft August 13, 2025 19:54

shatzi and others added 4 commits August 14, 2025 09:12

fix coverage error by adding more tests

69dff6d

fix capability name and index

0bc1ad1

fix codenarc error

117096e

Merge branch 'master' into shatzi/DEBUG-4326/apm-tracing-override-mer…

97447c7

…ging

shatzi marked this pull request as ready for review August 20, 2025 13:01

shatzi added the comp: remote config Configuration at Runtime label Aug 21, 2025

Merge branch 'master' into shatzi/DEBUG-4326/apm-tracing-override-mer…

362d24f

…ging

mhlidd requested changes Aug 22, 2025

View reviewed changes

address PR comments

8a80bee

shatzi requested a review from mhlidd August 25, 2025 02:40

jpbempel changed the title ~~[DEBUG 4326] add support for org level APM_TRACING configs~~ add support for org level APM_TRACING configs Aug 26, 2025

tylfin mentioned this pull request Aug 27, 2025

feat(APM_TRACING): Test new APM_TRACING_MULTICONFIG merging logic DataDog/system-tests#5154

Merged

6 tasks

tylfin mentioned this pull request Aug 27, 2025

feat(er): Ensure APM_MULTICONFIG compat works DataDog/system-tests#5156

Merged

6 tasks

jpbempel added 2 commits August 29, 2025 11:31

Merge branch 'master' into shatzi/DEBUG-4326/apm-tracing-override-mer…

94e6937

…ging

fix spotless

acee689

mhlidd approved these changes Aug 29, 2025

View reviewed changes

shatzi and others added 2 commits September 2, 2025 22:15

add another test

694f353

Merge branch 'master' into shatzi/DEBUG-4326/apm-tracing-override-mer…

a93db23

…ging

jpbempel added the type: enhancement Enhancements and improvements label Sep 3, 2025

jpbempel merged commit 87351d8 into master Sep 4, 2025
503 of 504 checks passed

jpbempel deleted the shatzi/DEBUG-4326/apm-tracing-override-merging branch September 4, 2025 11:50

github-actions bot added this to the 1.54.0 milestone Sep 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add support for org level APM_TRACING configs #9360

add support for org level APM_TRACING configs #9360

Uh oh!

shatzi commented Aug 13, 2025 •

edited by jira bot

Loading

Uh oh!

github-actions bot commented Aug 13, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented Aug 13, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Aug 13, 2025 •

edited

Loading

Uh oh!

mhlidd left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhlidd Aug 22, 2025

Uh oh!

shatzi Aug 25, 2025

Uh oh!

Uh oh!

mhlidd left a comment

Uh oh!

Uh oh!

Uh oh!

add support for org level APM_TRACING configs #9360

add support for org level APM_TRACING configs #9360

Uh oh!

Conversation

shatzi commented Aug 13, 2025 • edited by jira bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

github-actions bot commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 bot commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

mhlidd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhlidd Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

shatzi Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mhlidd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

shatzi commented Aug 13, 2025 •

edited by jira bot

Loading

github-actions bot commented Aug 13, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Aug 13, 2025 •

edited

Loading

pr-commenter bot commented Aug 13, 2025 •

edited

Loading