Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add parallelism to major chunk of hooks. Check Parallelism section in README #620

Merged
merged 55 commits into from
Feb 17, 2024
Merged
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
6f44b5d
Init paralelizm
MaxymVlasov Feb 8, 2024
4011cfc
Remove not more needed.
MaxymVlasov Feb 8, 2024
007b752
[WIP] Add debug logs for testing
MaxymVlasov Feb 8, 2024
819d913
If plugin_cache_dir specified - lock it to avoid bug
MaxymVlasov Feb 8, 2024
30b3069
fixup flock
MaxymVlasov Feb 8, 2024
b6017f1
Test flock on directory rather than stdin
MaxymVlasov Feb 8, 2024
357b72e
Revert "Test flock on directory rather than stdin"
MaxymVlasov Feb 8, 2024
2b04472
Try --no-fork option and locking plugin_cache_dir, which more logical
MaxymVlasov Feb 9, 2024
92cc899
Try without --no-fork
MaxymVlasov Feb 9, 2024
2c52da1
Add fallback for macOS&Co w/o flock and hook-config=--parallelism_limit
MaxymVlasov Feb 9, 2024
d1a3a70
TEST that tha non-flock works
MaxymVlasov Feb 9, 2024
fd8becf
Add lockdir cleanup after unexcpected end
MaxymVlasov Feb 9, 2024
e430641
return flock, testing parallelism
MaxymVlasov Feb 9, 2024
467959c
dbg
MaxymVlasov Feb 9, 2024
91ffb99
Fix issue when all hook-config ignored except first
MaxymVlasov Feb 9, 2024
6051543
Cleanup debug log
MaxymVlasov Feb 9, 2024
f8d26f8
Add docs and rename CPU_NUM to CPU
MaxymVlasov Feb 9, 2024
8edae71
docs
MaxymVlasov Feb 9, 2024
6475c4b
Add flock to bug report templates
MaxymVlasov Feb 9, 2024
3d60692
Apply suggestions from code review
MaxymVlasov Feb 9, 2024
4f7d28c
Apply suggestions from code review
MaxymVlasov Feb 9, 2024
7932ee1
Merge branch 'master' into feat/parallelizm
MaxymVlasov Feb 9, 2024
a5d0980
Apply review suggestions
MaxymVlasov Feb 9, 2024
7c07f9e
Update hooks/_common.sh - config fix
MaxymVlasov Feb 9, 2024
778350a
Apply suggestions from code review
MaxymVlasov Feb 10, 2024
f034438
Apply review suggestions
MaxymVlasov Feb 10, 2024
114a653
Set to rmdir for safety reasons
MaxymVlasov Feb 10, 2024
9cc5e0e
Apply suggestions from code review
MaxymVlasov Feb 12, 2024
a1cac64
DRY tf-init command
MaxymVlasov Feb 12, 2024
4f76448
Revert "DRY tf-init command"
MaxymVlasov Feb 12, 2024
e0dda37
Apply suggestions from code review
MaxymVlasov Feb 12, 2024
66a8f38
Apply suggestions from code review
MaxymVlasov Feb 12, 2024
2cf3ee7
fixup
MaxymVlasov Feb 12, 2024
d9ec149
Add self-healing mechanizm to lock implementation
MaxymVlasov Feb 12, 2024
f640b7c
Apply suggestions from code review
MaxymVlasov Feb 12, 2024
48d1521
Revie suggestions
MaxymVlasov Feb 12, 2024
7f755b2
Remove duplicated tool names
MaxymVlasov Feb 12, 2024
c9f71b8
Cleanup
MaxymVlasov Feb 12, 2024
fe57983
Apply suggestions from code review
MaxymVlasov Feb 12, 2024
cd2fddc
Fix stat and add comments as requested
MaxymVlasov Feb 12, 2024
92ee184
Suppres errors during serach for CPU cores
MaxymVlasov Feb 12, 2024
35c060d
Fix OSX stat
MaxymVlasov Feb 12, 2024
c17035b
Locks not working, try to live without them
MaxymVlasov Feb 13, 2024
a427a7c
Update test results to new implementation
MaxymVlasov Feb 13, 2024
ac4d89f
Apply suggestions from code review
MaxymVlasov Feb 13, 2024
7022a29
Apply suggestions from code review
MaxymVlasov Feb 13, 2024
598d16b
f
MaxymVlasov Feb 13, 2024
ec22d70
feat: Investigate and fix issue with wrong CPU count for containers (…
MaxymVlasov Feb 15, 2024
01d3236
Fix issue with "no logs shown" when something goes wrong (#624)
MaxymVlasov Feb 15, 2024
2bf56bc
Count --parallelism-ci-cpu-cores only in edge-cases
MaxymVlasov Feb 15, 2024
60a106d
Update README.md
MaxymVlasov Feb 15, 2024
89ccdcc
Update hooks/_common.sh
MaxymVlasov Feb 15, 2024
4cf480f
Add more warnings and deal with possible bugs
MaxymVlasov Feb 16, 2024
60bfeea
Improve comment
MaxymVlasov Feb 16, 2024
d84ce5d
Merge branch 'master' into feat/parallelizm
antonbabenko Feb 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Cleanup debug log
MaxymVlasov committed Feb 9, 2024
commit 605154339ee552bc73b8471e1e4ea911004d72eb
24 changes: 4 additions & 20 deletions hooks/_common.sh
Original file line number Diff line number Diff line change
@@ -240,8 +240,6 @@ function common::per_dir_hook {
key=$(echo "${config[0]}" | sed 's/^[[:space:]]*//')
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
value=${config[1]}

echo "DBG configs: '$key' '$value'"

case $key in
--delegate-chdir)
# this flag will skip pushing and popping directories
@@ -264,20 +262,12 @@ function common::per_dir_hook {
parallelism_disabled=true
fi

echo "DBG HOOK_CONFIG: ${HOOK_CONFIG[*]}"
echo "DBG parallelism_limit: $parallelism_limit"
echo "DBG parallelism_disabled: $parallelism_disabled"
exit 1

# preserve errexit status
shopt -qo errexit

local final_exit_code=0
local pids=()

echo "$(date "+%s %N") DBG parallelism_limit $parallelism_limit"
echo "$(date "+%s %N") DBG TF_PLUGIN_CACHE_DIR: $TF_PLUGIN_CACHE_DIR"

mapfile -t dir_paths_unique < <(echo "${dir_paths[*]}" | tr ' ' '\n' | sort -u)
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
local length=${#dir_paths_unique[@]}
local last_index=$((${#dir_paths_unique[@]} - 1))
@@ -294,13 +284,11 @@ function common::per_dir_hook {
pid=$!
pids+=("$pid")

echo "$(date "+%s %N") DBG $dir_path $pid: right after send to background"
if [ $parallelism_disabled ] ||
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
[ "$i" != 0 ] && [ $((i % parallelism_limit)) == 0 ] || # don't stop on first iteration when parallelism_limit>1
[ "$i" == $last_index ]; then

for pid in "${pids[@]}"; do
echo "$(date "+%s %N") DBG $pid: wait result"
# Get the exit code from the background process
local exit_code=0
wait "$pid" || exit_code=$?
@@ -380,21 +368,18 @@ function common::terraform_init {
fi

if [ ! -d .terraform/modules ] || [ ! -d .terraform/providers ]; then

echo "$(date "+%s %N") DBG $dir_path: 1. before tf init"

# Plugin cache dir can't be write concurrently or read during writing
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
# https://github.com/hashicorp/terraform/issues/31964
if [ -z "$TF_PLUGIN_CACHE_DIR" ] || $parallelism_disabled; then
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
echo "$(date "+%s %N") DBG $dir_path: 2. No lock"
init_output=$(terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
exit_code=$?
else

if command -v flock &> /dev/null; then
echo "$(date "+%s %N") DBG $dir_path: 2. Cache-dir lock"

init_output=$(flock --exclusive "$TF_PLUGIN_CACHE_DIR" terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1)
init_output=$(
flock --exclusive "$TF_PLUGIN_CACHE_DIR" \
terraform init -backend=false "${TF_INIT_ARGS[@]}" 2>&1
)
exit_code=$?
# Fall back "simple-lock" mechanizm if `flock` is not available
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
else
@@ -413,7 +398,6 @@ function common::terraform_init {
common::colorify "green" "Or disable parallelism by setting '--hook-config=--parallelism_limit=1'"
fi
fi
echo "$(date "+%s %N") DBG $dir_path: 3. after tf init"

if [ $exit_code -ne 0 ]; then
common::colorify "red" "'terraform init' failed, '$command_name' skipped: $dir_path"