Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Align how http proxies are handled when fetching gpg keys for download verification #1125

Closed
wants to merge 2 commits into from
Closed

Align how http proxies are handled when fetching gpg keys for download verification #1125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5a4bb3e
Align how http proxies are handled when fetching gpg keys for downloa…
rbambrough-intel Sep 19, 2024
e784034
Remove curl_args from get_gpg_key_servers
rbambrough-intel Sep 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions src/git-lfs/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ GIT_LFS_ARCHIVE_GPG_KEY_URI="https://packagecloud.io/github/git-lfs/gpgkey"
GIT_LFS_ARCHIVE_ARCHITECTURES="amd64 arm64"
GIT_LFS_ARCHIVE_VERSION_CODENAMES="stretch buster bullseye bionic focal jammy"
GIT_LFS_CHECKSUM_GPG_KEYS="0x88ace9b29196305ba9947552f1ba225c0223b187 0x86cd3297749375bcf8206715f54fe648088335a9 0xaa3b3450295830d2de6db90caba67be5a5795889"
KEYSERVER_PROXY="${HTTP_PROXY:-""}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rbambrough-intel Thank you so much for taking the time to contribute this PR.

Instead of these changes, how about we making similar changes #1119 ?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the link is not pointing to the right PR and I don't understand your statement as it seems to be the common way to define a default variable value in this project (e.g.:

features/src/ruby/install.sh

Line 18 in 0be7941

ADDITIONAL_VERSIONS="${ADDITIONALVERSIONS:-""}"
,

features/src/java/install.sh

Line 29 in 302feca

ADDITIONAL_VERSIONS="${ADDITIONALVERSIONS:-""}"
,

features/src/powershell/install.sh

Line 16 in 302feca

POWERSHELL_MODULES="${MODULES:-""}"
). However I think this following will be enough:

Suggested change
KEYSERVER_PROXY="${HTTP_PROXY:-""}"
KEYSERVER_PROXY="${HTTP_PROXY}"

But maybe I missed something.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, I pasted the incorrect link.

I was recommending #1124 which tests the servers before using them to download the keys.

KEYSERVER_PROXY="${HTTP_PROXY:-""}"

This simply assigns the KEYSERVER_PROXY to "" if HTTP_PROXY is not defined.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jarsop If we are fine with removing the httpProxy setting from some of the features then it can be simplified into:

KEYSERVER_PROXY="$HTTP_PROXY"

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@samruddhikhandale It does do curl checking of the key servers, just not the same as the one you linked (map instead of multiple function calls). I wasn't sure touching the function one so I adjusted all the others that looked closee


set -e

Expand Down Expand Up @@ -69,16 +70,11 @@ get_gpg_key_servers() {
["hkp://keyserver.pgp.com"]="http://keyserver.pgp.com:11371"
)

local curl_args=""
local keyserver_reachable=false # Flag to indicate if any keyserver is reachable

if [ ! -z "${KEYSERVER_PROXY}" ]; then
curl_args="--proxy ${KEYSERVER_PROXY}"
fi

for keyserver in "${!keyservers_curl_map[@]}"; do
local keyserver_curl_url="${keyservers_curl_map[${keyserver}]}"
if curl -s ${curl_args} --max-time 5 ${keyserver_curl_url} > /dev/null; then
if curl -s --max-time 5 ${keyserver_curl_url} > /dev/null; then
echo "keyserver ${keyserver}"
keyserver_reachable=true
else
Expand All @@ -95,6 +91,16 @@ get_gpg_key_servers() {
# Import the specified key in a variable name passed in as
receive_gpg_keys() {
local keys=${!1}
local keyring_args=""
local gpg_cmd="gpg"

if [ ! -z "$2" ]; then
mkdir -p "$(dirname \"$2\")"
keyring_args="--no-default-keyring --keyring $2"
fi
if [ ! -z "${KEYSERVER_PROXY}" ]; then
keyring_args="${keyring_args} --keyserver-options http-proxy=${KEYSERVER_PROXY}"
fi

# Install curl
if ! type curl > /dev/null 2>&1; then
Expand All @@ -110,16 +116,15 @@ receive_gpg_keys() {
local retry_count=0
local gpg_ok="false"
set +e
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ];
do
echo "(*) Downloading GPG key..."
( echo "${keys}" | xargs -n 1 gpg --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ]; do
echo "(*) Downloading GPG key..."
(echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
set -e
if [ "${gpg_ok}" = "false" ]; then
echo "(!) Failed to get gpg key."
Expand Down
32 changes: 16 additions & 16 deletions src/git/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ GIT_VERSION=${VERSION} # 'system' checks the base image first, else installs 'la
USE_PPA_IF_AVAILABLE=${PPA}

GIT_CORE_PPA_ARCHIVE_GPG_KEY=E1DD270288B4E6030699E45FA1715D88E1DF1F24
KEYSERVER_PROXY="${HTTP_PROXY:-""}"

if [ "$(id -u)" -ne 0 ]; then
echo -e 'Script must be run as root. Use sudo, su, or add "USER root" to your Dockerfile before running this script.'
Expand Down Expand Up @@ -74,17 +75,12 @@ clean_up

# Get the list of GPG key servers that are reachable
get_gpg_key_servers() {
local curl_args=""
local keyserver_reachable=false # Flag to indicate if any keyserver is reachable

if [ ! -z "${KEYSERVER_PROXY}" ]; then
curl_args="--proxy ${KEYSERVER_PROXY}"
fi

test_keyserver() {
local keyserver="$1"
local keyserver_curl_url="$2"
if curl -s ${curl_args} --max-time 5 "${keyserver_curl_url}" > /dev/null; then
if curl -s --max-time 5 "${keyserver_curl_url}" > /dev/null; then
echo "keyserver ${keyserver}"
keyserver_reachable=true
else
Expand Down Expand Up @@ -112,10 +108,15 @@ get_gpg_key_servers() {
receive_gpg_keys() {
local keys=${!1}
local keyring_args=""
local gpg_cmd="gpg"

if [ ! -z "$2" ]; then
mkdir -p "$(dirname \"$2\")"
keyring_args="--no-default-keyring --keyring $2"
fi
if [ ! -z "${KEYSERVER_PROXY}" ]; then
keyring_args="${keyring_args} --keyserver-options http-proxy=${KEYSERVER_PROXY}"
fi

# Install curl
if ! type curl > /dev/null 2>&1; then
Expand All @@ -131,16 +132,15 @@ receive_gpg_keys() {
local retry_count=0
local gpg_ok="false"
set +e
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ];
do
echo "(*) Downloading GPG key..."
( echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ]; do
echo "(*) Downloading GPG key..."
(echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
set -e
if [ "${gpg_ok}" = "false" ]; then
echo "(!) Failed to get gpg key."
Expand Down
33 changes: 17 additions & 16 deletions src/github-cli/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ CLI_VERSION=${VERSION:-"latest"}
INSTALL_DIRECTLY_FROM_GITHUB_RELEASE=${INSTALLDIRECTLYFROMGITHUBRELEASE:-"true"}

GITHUB_CLI_ARCHIVE_GPG_KEY=23F3D4EA75716059
KEYSERVER_PROXY="${HTTP_PROXY:-""}"

set -e

Expand All @@ -31,16 +32,11 @@ get_gpg_key_servers() {
["hkp://keyserver.pgp.com"]="http://keyserver.pgp.com:11371"
)

local curl_args=""
local keyserver_reachable=false # Flag to indicate if any keyserver is reachable

if [ ! -z "${KEYSERVER_PROXY}" ]; then
curl_args="--proxy ${KEYSERVER_PROXY}"
fi

for keyserver in "${!keyservers_curl_map[@]}"; do
local keyserver_curl_url="${keyservers_curl_map[${keyserver}]}"
if curl -s ${curl_args} --max-time 5 ${keyserver_curl_url} > /dev/null; then
if curl -s --max-time 5 ${keyserver_curl_url} > /dev/null; then
echo "keyserver ${keyserver}"
keyserver_reachable=true
else
Expand All @@ -58,9 +54,15 @@ get_gpg_key_servers() {
receive_gpg_keys() {
local keys=${!1}
local keyring_args=""
local gpg_cmd="gpg"

if [ ! -z "$2" ]; then
mkdir -p "$(dirname \"$2\")"
keyring_args="--no-default-keyring --keyring $2"
fi
if [ ! -z "${KEYSERVER_PROXY}" ]; then
keyring_args="${keyring_args} --keyserver-options http-proxy=${KEYSERVER_PROXY}"
fi

# Install curl
if ! type curl > /dev/null 2>&1; then
Expand All @@ -76,16 +78,15 @@ receive_gpg_keys() {
local retry_count=0
local gpg_ok="false"
set +e
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ];
do
echo "(*) Downloading GPG key..."
( echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ]; do
echo "(*) Downloading GPG key..."
(echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
set -e
if [ "${gpg_ok}" = "false" ]; then
echo "(!) Failed to get gpg key."
Expand Down
8 changes: 2 additions & 6 deletions src/kubectl-helm-minikube/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ MINIKUBE_SHA256="${MINIKUBE_SHA256:-"automatic"}"
USERNAME="${USERNAME:-"${_REMOTE_USER:-"automatic"}"}"

HELM_GPG_KEYS_URI="https://raw.githubusercontent.com/helm/helm/main/KEYS"
KEYSERVER_PROXY="${HTTP_PROXY:-""}"

if [ "$(id -u)" -ne 0 ]; then
echo -e 'Script must be run as root. Use sudo, su, or add "USER root" to your Dockerfile before running this script.'
Expand Down Expand Up @@ -239,16 +240,11 @@ get_gpg_key_servers() {
["hkp://keyserver.pgp.com"]="http://keyserver.pgp.com:11371"
)

local curl_args=""
local keyserver_reachable=false # Flag to indicate if any keyserver is reachable

if [ ! -z "${KEYSERVER_PROXY}" ]; then
curl_args="--proxy ${KEYSERVER_PROXY}"
fi

for keyserver in "${!keyservers_curl_map[@]}"; do
local keyserver_curl_url="${keyservers_curl_map[${keyserver}]}"
if curl -s ${curl_args} --max-time 5 ${keyserver_curl_url} > /dev/null; then
if curl -s --max-time 5 ${keyserver_curl_url} > /dev/null; then
echo "keyserver ${keyserver}"
keyserver_reachable=true
else
Expand Down
21 changes: 10 additions & 11 deletions src/python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,16 @@ Installs the provided version of Python, as well as PIPX, and other common Pytho

## Options

| Options Id | Description | Type | Default Value |
|-----|-----|-----|-----|
| version | Select a Python version to install. | string | os-provided |
| installTools | Flag indicating whether or not to install the tools specified via the 'toolsToInstall' option. Default is 'true'. | boolean | true |
| toolsToInstall | Comma-separated list of tools to install when 'installTools' is true. Defaults to a set of common Python tools like pylint. | string | flake8,autopep8,black,yapf,mypy,pydocstyle,pycodestyle,bandit,pipenv,virtualenv,pytest,pylint |
| optimize | Optimize Python for performance when compiled (slow) | boolean | false |
| enableShared | Enable building a shared Python library | boolean | false |
| installPath | The path where python will be installed. | string | /usr/local/python |
| installJupyterlab | Install JupyterLab, a web-based interactive development environment for notebooks | boolean | false |
| configureJupyterlabAllowOrigin | Configure JupyterLab to accept HTTP requests from the specified origin | string | - |
| httpProxy | Connect to GPG keyservers using a proxy for fetching source code signatures by configuring this option | string | - |
| Options Id | Description | Type | Default Value |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------- |
| version | Select a Python version to install. | string | os-provided |
| installTools | Flag indicating whether or not to install the tools specified via the 'toolsToInstall' option. Default is 'true'. | boolean | true |
| toolsToInstall | Comma-separated list of tools to install when 'installTools' is true. Defaults to a set of common Python tools like pylint. | string | flake8,autopep8,black,yapf,mypy,pydocstyle,pycodestyle,bandit,pipenv,virtualenv,pytest,pylint |
| optimize | Optimize Python for performance when compiled (slow) | boolean | false |
| enableShared | Enable building a shared Python library | boolean | false |
| installPath | The path where python will be installed. | string | /usr/local/python |
| installJupyterlab | Install JupyterLab, a web-based interactive development environment for notebooks | boolean | false |
| configureJupyterlabAllowOrigin | Configure JupyterLab to accept HTTP requests from the specified origin | string | - |

## Customizations

Expand Down
11 changes: 3 additions & 8 deletions src/python/devcontainer-feature.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,6 @@
"type": "string",
"default": "",
"description": "Configure JupyterLab to accept HTTP requests from the specified origin"
},
"httpProxy": {
"type": "string",
"default": "",
"description": "Connect to GPG keyservers using a proxy for fetching source code signatures by configuring this option"
}
},
"containerEnv": {
Expand All @@ -78,9 +73,9 @@
],
"settings": {
"python.defaultInterpreterPath": "/usr/local/python/current/bin/python",
"[python]": {
"editor.defaultFormatter": "ms-python.autopep8"
}
"[python]": {
"editor.defaultFormatter": "ms-python.autopep8"
}
}
}
},
Expand Down
31 changes: 13 additions & 18 deletions src/python/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,7 @@ ADDITIONAL_VERSIONS="${ADDITIONALVERSIONS:-""}"
IFS="," read -r -a DEFAULT_UTILS <<< "${TOOLSTOINSTALL:-flake8,autopep8,black,yapf,mypy,pydocstyle,pycodestyle,bandit,pipenv,virtualenv,pytest}"

PYTHON_SOURCE_GPG_KEYS="64E628F8D684696D B26995E310250568 2D347EA6AA65421D FB9921286F5E1540 3A5CA953F73C700D 04C367C218ADD4FF 0EDDC5F26A45C816 6AF053F07D9DC8D2 C9BE28DEE6DF025C 126EB563A74B06BF D9866941EA5BBD71 ED9D77D5 A821E680E5FA6305"

KEYSERVER_PROXY="${HTTPPROXY:-"${HTTP_PROXY:-""}"}"
KEYSERVER_PROXY="${HTTP_PROXY:-""}"

set -e

Expand Down Expand Up @@ -142,16 +141,11 @@ get_gpg_key_servers() {
["hkp://keyserver.pgp.com"]="http://keyserver.pgp.com:11371"
)

local curl_args=""
local keyserver_reachable=false # Flag to indicate if any keyserver is reachable

if [ ! -z "${KEYSERVER_PROXY}" ]; then
curl_args="--proxy ${KEYSERVER_PROXY}"
fi

for keyserver in "${!keyservers_curl_map[@]}"; do
local keyserver_curl_url="${keyservers_curl_map[${keyserver}]}"
if curl -s ${curl_args} --max-time 5 ${keyserver_curl_url} > /dev/null; then
if curl -s --max-time 5 ${keyserver_curl_url} > /dev/null; then
echo "keyserver ${keyserver}"
keyserver_reachable=true
else
Expand All @@ -170,6 +164,7 @@ receive_gpg_keys() {
local keys=${!1}
local keyring_args=""
local gpg_cmd="gpg"

if [ ! -z "$2" ]; then
mkdir -p "$(dirname \"$2\")"
keyring_args="--no-default-keyring --keyring $2"
Expand All @@ -192,22 +187,22 @@ receive_gpg_keys() {
local retry_count=0
local gpg_ok="false"
set +e
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ];
do
echo "(*) Downloading GPG key..."
( echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
until [ "${gpg_ok}" = "true" ] || [ "${retry_count}" -eq "5" ]; do
echo "(*) Downloading GPG key..."
(echo "${keys}" | xargs -n 1 gpg -q ${keyring_args} --recv-keys) 2>&1 && gpg_ok="true"
if [ "${gpg_ok}" != "true" ]; then
echo "(*) Failed getting key, retrying in 10s..."
(( retry_count++ ))
sleep 10s
fi
done
set -e
if [ "${gpg_ok}" = "false" ]; then
echo "(!) Failed to get gpg key."
exit 1
fi
}

# RHEL7/CentOS7 has an older gpg that does not have dirmngr
# Iterate through keyservers until we have all the keys downloaded
receive_gpg_keys_centos7() {
Expand Down
Loading