tools: check_format_test to run under CI under 'format' (envoyproxy#3516)

check_format_test is (I think) hard to run under bazel because it must run under the docker image to get the right clang, but bazel is hermetic and I don't know what it would take to call docker from bazel. But it can easily run under CI. In fact it's fast enough to just run it in front of check_format.py itself. Having it in CI and available to run manually via tools/check_format_test.sh should be sufficient to keep check_format.py sane, once more tests are added.

Risk Level: Low
Testing: test/check_format_test.sh --loglevel={0,1,2} and observing CI
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Joshua Marantz <[email protected]>

Author: jmarantz

ci/do_ci.sh

@@ -180,8 +180,10 @@ elif [[ "$1" == "fix_format" ]]; then
   ./tools/check_format.py fix
   exit 0
 elif [[ "$1" == "check_format" ]]; then
-  echo "check_format..."
   cd "${ENVOY_SRCDIR}"
+  echo "check_format_test..."
+  ./tools/check_format_test_helper.py --log=WARN
+  echo "check_format..."
   ./tools/check_format.py check
   exit 0
 elif [[ "$1" == "docs" ]]; then

tools/check_format.py

@@ -327,3 +327,5 @@ def checkFormatVisitor(arg, dir_name, names):
       print "ERROR: %s" % e
     print "ERROR: check format failed. run 'tools/check_format.py fix'"
     sys.exit(1)
+  if operation_type == "check":
+    print "PASS"

tools/check_format_test.sh

@@ -4,4 +4,4 @@ tools=$(dirname $(realpath $0))
 root=$(realpath $tools/..)
 ci=$root/ci
 cd $root
-exec ./ci/run_envoy_docker.sh ./tools/check_format_test_run_under_docker.py
+exec ./ci/run_envoy_docker.sh ./tools/check_format_test_helper.py "$@"

…ls/check_format_test_run_under_docker.py tools/check_format_test_helper.pytools/check_format_test_run_under_docker.py renamed to tools/check_format_test_helper.py tools/check_format_test_run_under_docker.py renamed to tools/check_format_test_helper.py

@@ -1,11 +1,14 @@
 #!/usr/bin/env python
 
-# Tests check_format.py. This must be run under docker via
-# check_format_test.py, or you are liable to get the wrong clang, and
-# all kinds of bad results.
+# Tests check_format.py. This must be run in a context where the clang
+# version and settings are compatible with the one in the Envoy
+# docker.  Normally this is run via check_format_test.sh, which
+# executes it in under docker.
 
+import argparse
 import os
 import shutil
+import logging
 import subprocess
 
 def getenvFallback(envvar, fallback):
@@ -33,7 +36,7 @@ def runCommand(command):
     status = e.returncode
     for line in e.output.splitlines():
       stdout.append(line)
-  print("%s" % command)
+  logging.info("%s" % command)
   return status, stdout
 
 # Runs the 'check_format' operation, on the specified file, printing
@@ -79,19 +82,19 @@ def fixFileExpectingNoChange(file):
     return 1
   return 0
 
-def emitStdout(stdout):
+def emitStdoutAsError(stdout):
   for line in stdout:
-    print("    %s" % line)
+    logging.error("    %s" % line)
 
 def expectError(status, stdout, expected_substring):
   if status == 0:
-    print("Expected failure, but succeeded")
+    logging.error("Expected failure, but succeeded")
     return 1
   for line in stdout:
     if expected_substring in line:
       return 0
-  print("Could not find '%s' in:\n" % expected_substring)
-  emitStdout(stdout)
+  logging.error("Could not find '%s' in:\n" % expected_substring)
+  emitStdoutAsError(stdout)
   return 1
 
 def fixFileExpectingFailure(filename, expected_substring):
@@ -105,11 +108,15 @@ def checkFileExpectingError(filename, expected_substring):
 def checkFileExpectingOK(filename):
   command, status, stdout = runCheckFormat("check", getInputFile(filename))
   if status != 0:
-    print("status=%d, output:\n" % status)
-    emitStdout(stdout)
+    logging.error("status=%d, output:\n" % status)
+    emitStdoutAsError(stdout)
   return 0
 
 if __name__ == "__main__":
+  parser = argparse.ArgumentParser(description='tester for check_foramt.py.')
+  parser.add_argument('--log', choices=['INFO', 'WARN', 'ERROR'], default='INFO')
+  args = parser.parse_args()
+  logging.basicConfig(format='%(message)s', level=args.log)
   errors = 0
 
   # Now create a temp directory to copy the input files, so we can fix them
@@ -128,7 +135,6 @@ def checkFileExpectingOK(filename):
   errors += checkFileExpectingOK("ok_file.cc")
 
   if errors != 0:
-    print("%d FAILURES" % errors)
+    logging.error("%d FAILURES" % errors)
     exit(1)
-  print("PASS")
-  exit(0)
+  logging.warn("PASS")