build: add scripts for privileged docker test execution (envoyproxy#5268)

We want to run some integration tests which require various privileges. I have added some scripts which wrap a bazel test target in another script which sets up such a remote environment. The general approach is similar to bazel-test-gdb, except no intermediate script is generated.

The script works fairly well. See the documentation portion of the PR for some examples. There are a few gotchas with it, however, that we should probably resolve over time:

It won't work with tests that depend on anything outside the test itself -- e.g. input files.
If compiling locally, there may be some challenges invoking other binaries in the target container, since we end up overwriting some runtime libraries that the container's gdb may need.
There is no strategy just yet to run a privileged test in gdb... I suspect that sudo bazel-test-gdb will have to be sufficient.
Example: tools/bazel-test-docker.sh //test/integration:integration_test --jobs=4 -c dbg

Risk Level: Low. This is not used by anything, and it will only ever be used explicitly, so it should be tested substantially then.

Testing: Tested locally with various integration tests, as well as in circle with my integration test which requires privileges: https://circleci.com/gh/klarose/envoy-circle/11. This commit was based off my hacked up fork of envoy, which only ran the one test: https://github.com/klarose/envoy-circle

Docs Changes: I added a description of the script to bazel/README.md.

Release Notes: Only affects tooling, so I don't think one is required.

Fixes Issue envoyproxy#5246

Signed-off-by: Kyle Larose <[email protected]>

Author: klarose

bazel/README.md

@@ -200,6 +200,50 @@ tools/bazel-test-gdb //test/common/http:async_client_impl_test -c dbg
 Without the `-c dbg` Bazel option at the end of the command line the test
 binaries will not include debugging symbols and GDB will not be very useful.
 
+# Running Bazel tests requiring privileges
+
+Some tests may require privileges (e.g. CAP_NET_ADMIN) in order to execute. One option is to run
+them with elevated privileges, e.g. `sudo test`. However, that may not always be possible,
+particularly if the test needs to run in a CI pipeline. `tools/bazel-test-docker.sh` may be used in
+such situations to run the tests in a privileged docker container.
+
+The script works by wrapping the test execution in the current repository's circle ci build
+container, then executing it either locally or on a remote docker container. In both cases, the
+container runs with the `--privileged` flag, allowing it to execute operations which would otherwise
+be restricted.
+
+The command line format is:
+`tools/bazel-test-docker.sh <bazel-test-target> [optional-flags-to-bazel]`
+
+The script uses two optional environment variables to control its behaviour:
+
+* `RUN_REMOTE=<yes|no>`: chooses whether to run on a remote docker server.
+* `LOCAL_MOUNT=<yes|no>`: copy/mount local libraries onto the docker container.
+
+Use `RUN_REMOTE=yes` when you don't want to run against your local docker instance. Note that you
+will need to override a few environment variables to set up the remote docker. The list of variables
+can be found in the [Documentation](https://docs.docker.com/engine/reference/commandline/cli/).
+
+Use `LOCAL_MOUNT=yes` when you are not building with the envoy build container. This will ensure
+that the libraries against which the tests dynmically link will be available and of the correct
+version.
+
+## Examples
+
+Running the http integration test in a privileged container:
+
+```bash
+tools/bazel-test-docker.sh  //test/integration:integration_test --jobs=4 -c dbg
+```
+
+Running the http integration test compiled locally against a privileged remote container:
+
+```bash
+setup_remote_docker_variables
+RUN_REMOTE=yes MOUNT_LOCAL=yes tools/bazel-test-docker.sh  //test/integration:integration_test \
+  --jobs=4 -c dbg
+```
+
 # Additional Envoy build and test options
 
 In general, there are 3 [compilation

tools/bazel-test-docker.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# Run a single Bazel test target under a privileged docker. Usage:
+#
+# tools/bazel-test-docker //test/foo:bar --some_other --bazel_args
+# By default, this will run in a local docker container, mounting the local shared library paths
+# into the counter. To run remotely, use RUN_REMOTE=yes. If the test was compiled with a different
+# toolchain than the envoy-build container, passing in LOCAL_MOUNT=yes will force it to copy the
+# local libraries into the container.
+
+if [[ -z "$1" ]]; then
+  echo "First argument to $0 must be a [@repo]//test/foo:bar label identifying a set of test to run"
+  echo "\"$1\" does not match this pattern"
+  exit 1
+fi
+
+SCRIPT_DIR="$(realpath "$(dirname "$0")")"
+[[ -z "${BAZEL}" ]] && BAZEL=bazel
+[[ -z "${DOCKER}" ]] && DOCKER=docker
+
+if [[ -z "${RUN_REMOTE}" ]]; then
+  LOCAL_MOUNT="${LOCAL_MOUNT:-yes}"
+  RUN_REMOTE=no
+else
+  LOCAL_MOUNT="${LOCAL_MOUNT:-no}"
+  RUN_REMOTE=yes
+fi
+
+# Pass through the docker environment
+DOCKER_ENV=$(mktemp -t docker_env.XXXXXX)
+function cleanup() {
+  rm -f "${DOCKER_ENV}"
+}
+
+trap cleanup EXIT
+cat > "${DOCKER_ENV}" <<EOF
+  #!/bin/bash
+  export DOCKER_CERT_PATH="${DOCKER_CERT_PATH}"
+  export DOCKER_HOST="${DOCKER_HOST}"
+  export DOCKER_MACHINE_NAME="${DOCKER_MACHINE_NAME}"
+  export DOCKER_TLS_VERIFY="${DOCKER_TLS_VERIFY}"
+  export NO_PROXY="${NO_PROXY}"
+EOF
+
+. ./ci/envoy_build_sha.sh
+IMAGE=envoyproxy/envoy-build:${ENVOY_BUILD_SHA}
+
+# Note docker_wrapper.sh is tightly coupled to the order of arguments here due to where the test
+# name is passed in.
+"${BAZEL}" test "$@" --strategy=TestRunner=standalone --cache_test_results=no \
+  --test_output=summary --run_under="${SCRIPT_DIR}/docker_wrapper.sh ${IMAGE} ${RUN_REMOTE} \
+   ${LOCAL_MOUNT} ${DOCKER_ENV}"

tools/docker_wrapper.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+#
+# Wraps a test invocation in docker.
+
+set -e
+IMAGE=$1
+RUN_REMOTE=$2
+LOCAL_MOUNT=$3
+DOCKER_ENV=$4
+TEST_PATH=$(realpath "$5")
+shift 5
+
+if [ "${RUN_REMOTE}" == "yes" ]; then
+  echo "Using docker environment from ${DOCKER_ENV}:"
+  cat "${DOCKER_ENV}"
+fi
+. "${DOCKER_ENV}"
+
+CONTAINER_NAME="envoy-test-runner"
+ENVFILE=$(mktemp -t "bazel-test-env.XXXXXX")
+function cleanup() {
+  rm -f "${ENVFILE}"
+  if [ "${RUN_REMOTE}" == "yes" ]; then
+    docker rm -f "${CONTAINER_NAME}"  || true # We don't really care if it fails.
+  fi
+}
+
+trap cleanup EXIT
+
+cat > ${ENVFILE} <<EOF
+TEST_WORKSPACE=/tmp/workspace
+TEST_SRCDIR=/tmp/src
+ENVOY_IP_TEST_VERSIONS=v4only
+EOF
+
+CMDLINE="set -a && . /env && env && /test $@"
+LIB_PATHS="/lib/x86_64-linux-gnu/ /usr/lib/x86_64-linux-gnu/ /lib64/"
+
+
+if [ "${RUN_REMOTE}" != "yes" ]; then
+  # We're running locally. If told to, mount the library directories locally.
+  LIB_MOUNTS=""
+  if [ "${LOCAL_MOUNT}" == "yes" ]
+  then
+    for path in $LIB_PATHS; do
+      LIB_MOUNTS="${LIB_MOUNTS} -v ${path}:${path}:ro"
+      done
+  fi
+
+  # Note: we don't quote LIB_MOUNTS on purpose; we want it to expand.
+  docker run --rm --privileged  -v "${TEST_PATH}:/test" ${LIB_MOUNTS} -i -v "${ENVFILE}:/env" \
+    "${IMAGE}" bash -c "${CMDLINE}"
+else
+  # In this case, we need to create the container, then make new layers on top of it, since we
+  # can't mount everything into it.
+  docker create -t --privileged  --name "${CONTAINER_NAME}" "${IMAGE}" \
+    bash -c "${CMDLINE}"
+  docker cp "$TEST_PATH" "${CONTAINER_NAME}:/test"
+  docker cp "$ENVFILE" "${CONTAINER_NAME}:/env"
+
+  # If some local libraries are necessary, copy them over.
+  if [ "${LOCAL_MOUNT}" == "yes" ]; then
+    for path in ${LIB_PATHS}; do
+      # $path. gives us a path ending it /. This means that we will copy the contents into the
+      # destination directory, not overwrite the entire directory.
+      docker cp -L "${path}." "${CONTAINER_NAME}:${path}"
+    done
+  fi
+
+  docker start -a "${CONTAINER_NAME}"
+fi