Merge #2872: [Depends] Bump Rust toolchain to 1.69.0

1bc92be Depends: Bump rust toolchain to v1.69.0 (Fuzzbawls)
5708543 Scripts: Bump minimum supported glibc to v2.27 (Fuzzbawls)
f2cfadb GA: Run security-check on depends based builds (Fuzzbawls)
0a27b16 GA: Run symbol-check on depends based builds (Fuzzbawls)
02b7cb9 scripts: add PE dylib checking to symbol-check.py (fanquake)
ee3bdc6 depends: build gmp with PIC on all platforms (Fuzzbawls)
85d1632 scripts: add MACHO dylib checking to symbol-check.py (fanquake)
c2facf0 scripts: fix check-symbols & check-security argument passing (fanquake)
3af2b2f scripts: add MACHO NOUNDEFS check to security-check.py (fanquake)
83f86c3 scripts: add MACHO PIE check to security-check.py (fanquake)

Pull request description:

  Alternative to #2871.

  This bumps our Rust Toolchain to v1.69.0, but comes at the cost of ALSO bumping our minimum supported linux glibc to 2.27, which officially drops support for Ubuntu 16.04, Debian 8 (Jessie), Debian 9 (Stretch), and CentOS 7 for binary releases.

  Ubuntu 16.04 and CentOS 7 technically don't reach EOL (End Of Life) until 2024, but both are largely unsupported in linux communities already.

  The major benefit to this outside of being able to use a more recent Rust toolchain version, is that it allows us to use the official Rust distribution toolchains and no longer maintain our own modified versions, a significant time saver if this ability is carried forward in the future.

ACKs for top commit:
  Duddino:
    tACK 1bc92be, works fine with #2870
  panleone:
    tACK 1bc92be

Tree-SHA512: b59cf43045c189d6d68ca9770ef17f59baa4d5d4c854a9fb94afa8ac59a2fe576216926416689e6ad39161fccea51791552558da95d22850ba8b9af4c7abe015

.github/workflows/build-and-test.yml

@@ -422,6 +422,7 @@ jobs:
  apt_get: python3 nsis g++-mingw-w64-x86-64 wine-binfmt wine64
  unit_tests: false
  functional_tests: false
+ symbol_check: true
  goal: deploy
  BITCOIN_CONFIG: "--with-gui=auto --enable-reduce-exports --disable-online-rust"
 
@@ -456,6 +457,7 @@ jobs:
  XCODE_BUILD_ID: 11C505
  unit_tests: false
  functional_tests: false
+ symbol_check: true
  goal: deploy
  BITCOIN_CONFIG: "--enable-gui --enable-reduce-exports --enable-werror --disable-online-rust"
 
@@ -573,6 +575,16 @@ jobs:
  echo ::endgroup::
  fi
 
+ echo ::group::Security-Check
+ make -j2 -C src check-security
+ echo ::endgroup::
+
+ if [ "${{matrix.config.symbol_check }}" = "true" ]; then
+ echo ::group::Symbol-Check
+ make -j2 -C src check-symbols
+ echo ::endgroup::
+ fi
+
  if [ "${{ matrix.config.unit_tests }}" = "true" ]; then
  echo ::group::Unit-Tests
  if [[ ${{ matrix.config.os }} = ubuntu* ]]; then

configure.ac

@@ -523,6 +523,7 @@ case $host in
  AC_CHECK_LIB([shlwapi], [main],, AC_MSG_ERROR(libshlwapi missing))
  AC_CHECK_LIB([iphlpapi], [main],, AC_MSG_ERROR(libiphlpapi missing))
  AC_CHECK_LIB([crypt32], [main],, AC_MSG_ERROR(libcrypt32 missing))
+ AC_CHECK_LIB([bcrypt], [main],, AC_MSG_ERROR(libbcrypt missing))
 
  # -static is interpreted by libtool, where it has a different meaning.
  # In libtool-speak, it's -all-static.

contrib/devtools/README.md

@@ -158,22 +158,26 @@ repository (requires pngcrush).
 security-check.py and test-security-check.py
 ============================================
 
-Perform basic ELF security checks on a series of executables.
+Perform basic security checks on a series of executables.
 
 symbol-check.py
 ===============
 
-A script to check that the (Linux) executables produced by gitian only contain
-allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
-still compatible with the minimum supported Linux distribution versions.
+A script to check that the executables produced by gitian only contain
+certain symbols and are only linked against allowed libraries.
+
+For Linux this means checking for allowed gcc, glibc and libstdc++ version symbols.
+This makes sure they are still compatible with the minimum supported distribution versions.
+
+For macOS and Windows we check that the executables are only linked against libraries we allow.
 
 Example usage after a gitian build:
 
  find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
 
-If only supported symbols are used the return value will be 0 and the output will be empty.
+If no errors occur the return value will be 0 and the output will be empty.
 
-If there are 'unsupported' symbols, the return value will be 1 a list like this will be printed:
+If there are any errors the return value will be 1 and output like this will be printed:
 
  .../64/test_pivx: symbol memcpy from unsupported version GLIBC_2.14
  .../64/test_pivx: symbol __fdelt_chk from unsupported version GLIBC_2.15

contrib/devtools/security-check.py

@@ -3,17 +3,18 @@
 # Distributed under the MIT software license, see the accompanying
 # file COPYING or http://www.opensource.org/licenses/mit-license.php.
 '''
-Perform basic ELF security checks on a series of executables.
+Perform basic security checks on a series of executables.
 Exit status will be 0 if successful, and the program will be silent.
 Otherwise the exit status will be 1 and it will log which executables failed which checks.
-Needs `readelf` (for ELF) and `objdump` (for PE).
+Needs `readelf` (for ELF), `objdump` (for PE) and `otool` (for MACHO).
 '''
 import subprocess
 import sys
 import os
 
 READELF_CMD = os.getenv('READELF', '/usr/bin/readelf')
 OBJDUMP_CMD = os.getenv('OBJDUMP', '/usr/bin/objdump')
+OTOOL_CMD = os.getenv('OTOOL', '/usr/bin/otool')
 NONFATAL = {} # checks which are non-fatal for now but only generate a warning
 
 def check_ELF_PIE(executable):
@@ -162,6 +163,40 @@ def check_PE_NX(executable):
  (arch,bits) = get_PE_dll_characteristics(executable)
  return (bits & IMAGE_DLL_CHARACTERISTICS_NX_COMPAT) == IMAGE_DLL_CHARACTERISTICS_NX_COMPAT
 
+def get_MACHO_executable_flags(executable):
+ p = subprocess.Popen([OTOOL_CMD, '-vh', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
+ (stdout, stderr) = p.communicate()
+ if p.returncode:
+ raise IOError('Error opening file')
+
+ flags = []
+ for line in stdout.splitlines():
+ tokens = line.split()
+ # filter first two header lines
+ if 'magic' in tokens or 'Mach' in tokens:
+ continue
+ # filter ncmds and sizeofcmds values
+ flags += [t for t in tokens if not t.isdigit()]
+ return flags
+
+def check_MACHO_PIE(executable) -> bool:
+ '''
+ Check for position independent executable (PIE), allowing for address space randomization.
+ '''
+ flags = get_MACHO_executable_flags(executable)
+ if 'PIE' in flags:
+ return True
+ return False
+
+def check_MACHO_NOUNDEFS(executable) -> bool:
+ '''
+ Check for no undefined references.
+ '''
+ flags = get_MACHO_executable_flags(executable)
+ if 'NOUNDEFS' in flags:
+ return True
+ return False
+
 CHECKS = {
 'ELF': [
  ('PIE', check_ELF_PIE),
@@ -173,6 +208,10 @@ def check_PE_NX(executable):
  ('DYNAMIC_BASE', check_PE_DYNAMIC_BASE),
  ('HIGH_ENTROPY_VA', check_PE_HIGH_ENTROPY_VA),
  ('NX', check_PE_NX)
+],
+'MACHO': [
+ ('PIE', check_MACHO_PIE),
+ ('NOUNDEFS', check_MACHO_NOUNDEFS),
 ]
 }
 
@@ -183,6 +222,8 @@ def identify_executable(executable):
  return 'PE'
  elif magic.startswith(b'\x7fELF'):
  return 'ELF'
+ elif magic.startswith(b'\xcf\xfa'):
+ return 'MACHO'
  return None
 
 if __name__ == '__main__':