[envoy / haproxy] Create new components manifests

[e-n-0]

Motivation

• Envoy and HAProxy are two proxies supported by AAP and have to be rightfully tested under the same test suite as other libraries (even if the underlying implementation for these proxies are done with the go tracer).
• Proxies supports a different set of feature as of the go tracer and have their own versions that are not completely on par with the go tracer: so they need to have their own manifest file to keep track of the tests supported.

Changes

This PR made changes:

• Re-introduce different scenarios for each components (Envoy and HAProxy) with ENVOY_DEFAULT and ENVOY_APPSEC_BLOCKING (and similarly with haproxy)
• As they are 2 specific components with specific release process that supports tests on different versions, they now have their own unique manifest
• Scenario groups named go_proxies and go_proxies_blocking have been introduced to link the tests for the both proxies (so envoy and haproxy are running the same set of tests).

Note: This could be confusing but, envoy and haproxy, are the two tested components and come with their own weblogs. Currently they only have one weblog per proxy, but this will be increased in the future.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

manifests/envoy.yml                                                     @DataDog/system-tests-core
manifests/haproxy.yml                                                   @DataDog/system-tests-core
.cursor/rules/repository-structure.mdc                                  @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
run.sh                                                                  @DataDog/system-tests-core
tests/appsec/test_alpha.py                                              @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_blocking_addresses.py                                 @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_ip_blocking_full_denylist.py                          @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_reports.py                                            @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_request_blocking.py                                   @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_traces.py                                             @DataDog/asm-libraries @DataDog/system-tests-core
tests/appsec/test_versions.py                                           @DataDog/asm-libraries @DataDog/system-tests-core
tests/external_processing/test_apm.py                                   @DataDog/asm-libraries @DataDog/system-tests-core
tests/test_config_consistency.py                                        @DataDog/system-tests-core
tests/test_scrubbing.py                                                 @DataDog/system-tests-core
tests/test_semantic_conventions.py                                      @DataDog/system-tests-core
tests/test_standard_tags.py                                             @DataDog/system-tests-core
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-core
tests/test_the_test/test_group_rules.py                                 @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/go_proxies.py                                 @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/manifest/_internal/parser.py                                      @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core
utils/scripts/compute-workflow-parameters.py                            @DataDog/system-tests-core
utils/scripts/compute_libraries_and_scenarios.py                        @DataDog/system-tests-core
utils/scripts/get-image-list.py                                         @DataDog/system-tests-core
utils/scripts/libraries_and_scenarios_rules.yml                         @DataDog/system-tests-core
utils/scripts/load-binary.sh                                            @DataDog/system-tests-core
docs/scenarios/go_proxies_envoy_haproxy.md                              @DataDog/system-tests-core
utils/build/docker/envoy/envoy.yaml                                     @DataDog/system-tests-core

[datadog-datadog-prod-us1]

⚠️ Tests

✨ Fix all issues with Cursor

⚠️ Warnings

❄️ 16 New flaky tests detected

tests.appsec.api_security.test_endpoint_discovery.Test_Endpoint_Discovery.test_single_is_first[uwsgi-poc] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Expected one is_first=true payload, found 2
assert 2 == 1

self = <tests.appsec.api_security.test_endpoint_discovery.Test_Endpoint_Discovery object at 0x7f29d85abd40>

    def test_single_is_first(self):
        """Verify that the is_first flag appears exactly once in telemetry.
    
        Mandatory for the feature.
        """
...

tests.appsec.test_alpha.Test_Basic.test_headers[haproxy-spoa] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_alpha.Test_Basic object at 0x7fb25eb6d970>

    def test_headers(self):
        """Via server.request.headers.no_cookies"""
        # Note: we do not check the returned key_path nor rule_id for the alpha version
        address = "server.request.headers.no_cookies"
        pattern = "../"
>       interfaces.library.assert_waf_attack(self.r_headers_1, pattern=pattern, address=address)
...

tests.appsec.test_alpha.Test_Basic.test_no_cookies[haproxy-spoa] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_alpha.Test_Basic object at 0x7fb25eb6c440>

    def test_no_cookies(self):
        """Address server.request.headers.no_cookies should not include cookies."""
        # Relying on rule crs-930-110, test the following LFI attack is caught
        # on server.request.headers.no_cookies and then retry it with the cookies
        # to validate that cookies are properly excluded from server.request.headers.no_cookies.
        address = "server.request.headers.no_cookies"
...

View all

ℹ️ Info

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8855d57 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

[cbeauchesne]

We're getting somewhere :)

The CI prod on happroxy (and also probably the other) is not using the last release, but a dev one. We can go though this together on monday.