Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unified CI for C++ and Go #680

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

unified CI for C++ and Go #680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1306a84
added cpp-golang test for CI
mickeyasa Dec 8, 2024
620fcf6
fixed test name
mickeyasa Dec 8, 2024
94e4bb9
skipped check for changed files (wip)
mickeyasa Dec 8, 2024
78d8ab7
checking field
aviadingo Dec 8, 2024
481805e
field test
aviadingo Dec 8, 2024
fdf1389
final cpp-golang
aviadingo Dec 8, 2024
f40e1a8
disabled cpp and golang standalone tests
aviadingo Dec 8, 2024
babe767
bug fix
aviadingo Dec 8, 2024
8e15d34
add c++ examples to the unified test of c++/golang
aviadingo Dec 8, 2024
ec7717a
workflow bug fix
aviadingo Dec 8, 2024
b8fdf11
Compiling only non-main branches of cuda
aviadingo Dec 8, 2024
666399d
bug fix
aviadingo Dec 8, 2024
b8ea011
debug
aviadingo Dec 8, 2024
20906e3
setting export ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib
aviadingo Dec 8, 2024
cbc7234
change parameter for tests
aviadingo Dec 8, 2024
d8d543f
update examples workflow
aviadingo Dec 8, 2024
01cb8ce
bug fix
aviadingo Dec 8, 2024
cca941b
bug fix
aviadingo Dec 8, 2024
dcbeb2a
bug fix
aviadingo Dec 8, 2024
1490ba1
install alternate cuda backend on /tmp
aviadingo Dec 8, 2024
2f6633d
testing CUDA BE from different branch
aviadingo Dec 8, 2024
0d40632
test alternate backend
aviadingo Dec 8, 2024
4faa7e7
another try
aviadingo Dec 8, 2024
5467ee0
fix cuda flag
aviadingo Dec 8, 2024
0d2e643
updated cuda flag
aviadingo Dec 8, 2024
ad3f9df
aadded cuda lag to hash
aviadingo Dec 8, 2024
9ba5067
testing with latest cuda backend
aviadingo Dec 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 .github/workflows/cpp.yml → .github/OLD/cpp.yml
View file
Open in desktop
File renamed without changes.
0 .github/workflows/golang.yml → .github/OLD/golang.yml
View file
Open in desktop
File renamed without changes.
252 changes: 252 additions & 0 deletions .github/workflows/cpp-golang.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
name: C++/CUDA/GoLang & Examples

on:
pull_request:
branches:
- main
push:
branches:
- main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
check-changed-files:
uses: ./.github/workflows/check-changed-files.yml

check-format:
name: Check Code Format
runs-on: ubuntu-22.04
needs: check-changed-files
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Check clang-format
if: needs.check-changed-files.outputs.cpp == 'true'
run: ./scripts/format_all.sh . --check --exclude "build|wrappers"

extract-cuda-backend-branch:
uses: ./.github/workflows/extract-backends.yml
with:
pr-number: ${{ github.event.pull_request.number }}

test-linux-curve:
name: Test curve on Linux
runs-on: [self-hosted, Linux, X64, icicle]
needs: [check-changed-files, check-format, extract-cuda-backend-branch]
strategy:
matrix:
curve:
- name: bn254
build_args: -DG2=ON -DECNTT=ON
- name: bls12_381
build_args: -DG2=ON -DECNTT=ON
- name: bls12_377
build_args: -DG2=ON -DECNTT=ON
- name: bw6_761
build_args: -DG2=ON -DECNTT=ON
- name: grumpkin
build_args:

steps:
- name: Checkout Repo
uses: actions/checkout@v4
- name: Checkout CUDA Backend
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
uses: actions/checkout@v4
with:
repository: ingonyama-zk/icicle-cuda-backend
path: ./icicle/backend/cuda
ssh-key: ${{ secrets.CUDA_PULL_KEY }}
ref: ${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}
- name: Setup go
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
uses: actions/setup-go@v5
with:
go-version: '1.22.0'
- name: Set CUDA backend flag
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
id: cuda-flag
run: |
if [ "${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}" == "main" ]; then
echo "CUDA_FLAG=" >> $GITHUB_OUTPUT
echo "CMAKE_INSTALL_PREFIX=" >> $GITHUB_OUTPUT
echo "ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib" >> $GITHUB_OUTPUT
else
echo "CUDA_FLAG=-DCUDA_BACKEND=local" >> $GITHUB_OUTPUT
echo "CMAKE_INSTALL_PREFIX=-DCMAKE_INSTALL_PREFIX=/tmp/local/" >> $GITHUB_OUTPUT
echo "ICICLE_BACKEND_INSTALL_DIR=/tmp/local/lib" >> $GITHUB_OUTPUT
fi
- name: Build curve
working-directory: ./icicle
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
run: |
mkdir -p build && rm -rf build/*
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON -DCURVE=${{ matrix.curve.name }} ${{ matrix.curve.build_args }} ${{ steps.cuda-flag.outputs.CUDA_FLAG }} ${{ steps.cuda-flag.outputs.CMAKE_INSTALL_PREFIX }} -S . -B build
cmake --build build --target install -j
- name: Run C++ curve Tests
working-directory: ./icicle/build/tests
if: needs.check-changed-files.outputs.cpp == 'true'
run: |
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
ctest

# - name: Run C++ examples
# working-directory: ./examples/c++
# if: needs.check-changed-files.outputs.cpp == 'true' || needs.check-changed-files.outputs.examples == 'true'
# run: |
# CURVE=$(echo ${{ matrix.curve.name }} | sed -e 's/_//g')
# export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
# for dir in $(grep -l -r "\\-DCURVE=$CURVE" . | xargs -L1 dirname | sort -u); do
# if [ -d "$dir" ]; then
# echo "Running command in $dir"
# cd $dir
# ./run.sh -d CUDA
# cd -
# fi
# done

- name: Run Golang curve Tests
working-directory: ./wrappers/golang/curves
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
run: |
CURVE=$(echo ${{ matrix.curve.name }} | sed -e 's/_//g')
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
go test ./$CURVE/tests -count=1 -failfast -p 2 -timeout 60m -v


test-linux-field:
name: Test field on Linux
runs-on: [self-hosted, Linux, X64, icicle]
needs: [check-changed-files, check-format, extract-cuda-backend-branch]
strategy:
matrix:
field:
- name: babybear
build_args: -DEXT_FIELD=ON
- name: stark252
build_args: -DEXT_FIELD=OFF
- name: m31
build_args: -DEXT_FIELD=ON
steps:
- name: Checkout Repo
uses: actions/checkout@v4
- name: Checkout CUDA Backend
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
uses: actions/checkout@v4
with:
repository: ingonyama-zk/icicle-cuda-backend
path: ./icicle/backend/cuda
ssh-key: ${{ secrets.CUDA_PULL_KEY }}
ref: ${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}
- name: Setup go
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
uses: actions/setup-go@v5
with:
go-version: '1.22.0'
- name: Set CUDA backend flag
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
id: cuda-flag
run: |
if [ "${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}" == "main" ]; then
echo "CUDA_FLAG=" >> $GITHUB_OUTPUT
echo "CMAKE_INSTALL_PREFIX=" >> $GITHUB_OUTPUT
echo "ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib" >> $GITHUB_OUTPUT
else
echo "CUDA_FLAG=-DCUDA_BACKEND=local" >> $GITHUB_OUTPUT
echo "CMAKE_INSTALL_PREFIX=-DCMAKE_INSTALL_PREFIX=/tmp/local/" >> $GITHUB_OUTPUT
echo "ICICLE_BACKEND_INSTALL_DIR=/tmp/local/lib" >> $GITHUB_OUTPUT
fi
- name: Build field
working-directory: ./icicle
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
run: |
mkdir -p build && rm -rf build/*
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON -DFIELD=${{ matrix.field.name }} ${{ matrix.field.build_args }} ${{ steps.cuda-flag.outputs.CUDA_FLAG }} ${{ steps.cuda-flag.outputs.CMAKE_INSTALL_PREFIX }} -S . -B build
cmake --build build --target install -j
- name: Run C++ field Tests
working-directory: ./icicle/build/tests
if: needs.check-changed-files.outputs.cpp == 'true'
run: |
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
ctest --output-on-failure

# - name: Run C++ examples
# working-directory: ./examples/c++
# if: needs.check-changed-files.outputs.cpp == 'true' || needs.check-changed-files.outputs.examples == 'true'
# run: |
# FIELD=$(echo ${{ matrix.field.name }} | sed -e 's/_//g')
# export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
# for dir in $(grep -l -r "\\-DFIELD=$FIELD" . | xargs -L1 dirname | sort -u); do
# if [ -d "$dir" ]; then
# echo "Running command in $dir"
# cd $dir
# ./run.sh -d CUDA
# cd -
# fi
# done

- name: Run Golang field Tests (babybear only)
working-directory: ./wrappers/golang/fields
if: (needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true') && matrix.field.name == 'babybear'
# if: matrix.field.name == 'babybear'
run: |
FIELD=$(echo ${{ matrix.field.name }} | sed -e 's/_//g')
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
go test ./$FIELD/tests -count=1 -failfast -p 2 -timeout 60m -v



test-linux-hash:
name: Build and test Go hash on Linux
runs-on: [self-hosted, Linux, X64, icicle]
needs: [check-changed-files, check-format, extract-cuda-backend-branch]
steps:
- name: Checkout Repo
uses: actions/checkout@v4
- name: Checkout CUDA Backend
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
uses: actions/checkout@v4
with:
repository: ingonyama-zk/icicle-cuda-backend
path: ./icicle/backend/cuda
ssh-key: ${{ secrets.CUDA_PULL_KEY }}
ref: ${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}
- name: Setup go
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
uses: actions/setup-go@v5
with:
go-version: '1.22.0'
- name: Set CUDA backend flag
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
id: cuda-flag
run: |
if [ "${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}" == "main" ]; then
echo "CUDA_FLAG=" >> $GITHUB_OUTPUT
echo "CMAKE_INSTALL_PREFIX=" >> $GITHUB_OUTPUT
echo "ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib" >> $GITHUB_OUTPUT
else
echo "CUDA_FLAG=-DCUDA_BACKEND=local" >> $GITHUB_OUTPUT
echo "CMAKE_INSTALL_PREFIX=-DCMAKE_INSTALL_PREFIX=/tmp/local/" >> $GITHUB_OUTPUT
echo "ICICLE_BACKEND_INSTALL_DIR=/tmp/local/lib" >> $GITHUB_OUTPUT
fi
- name: Build
working-directory: ./wrappers/golang
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
# builds the hash and merkle tree lib using a local copy of the CUDA backend
run: |
./build.sh -cuda_backend=local
- name: Test GoLang Hashes
working-directory: ./wrappers/golang/hash
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
run: |
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
go test ./tests -count=1 -failfast -p 2 -timeout 60m -v
- name: Test GoLang Merkle Tree
working-directory: ./wrappers/golang/merkle-tree
if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
run: |
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
go test ./tests -count=1 -failfast -p 2 -timeout 60m -v
12 changes: 11 additions & 1 deletion .github/workflows/examples.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,20 @@ jobs:
path: ./icicle/backend/cuda
ssh-key: ${{ secrets.CUDA_PULL_KEY }}
ref: ${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}
- name: Set CUDA backend flag
id: cuda-flag
run: |
if [ "${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}" == "main" ]; then
echo "ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib" >> $GITHUB_OUTPUT
else
echo "ICICLE_BACKEND_INSTALL_DIR=" >> $GITHUB_OUTPUT
fi
- name: c++ examples
working-directory: ./examples/c++
if: needs.check-changed-files.outputs.cpp == 'true' || needs.check-changed-files.outputs.examples == 'true'
run: |
# loop over all directories in the current directory
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
for dir in $(find . -mindepth 1 -maxdepth 1 -type d); do
if [ -d "$dir" ]; then
echo "Running command in $dir"
Expand All @@ -56,8 +65,9 @@ jobs:
- name: Rust examples
working-directory: ./examples/rust
if: needs.check-changed-files.outputs.rust == 'true' || needs.check-changed-files.outputs.examples == 'true'
run: |
run: |
# loop over all directories in the current directory
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
for dir in $(find . -mindepth 1 -maxdepth 1 -type d); do
if [ -d "$dir" ]; then
echo "Running command in $dir"
Expand Down
Loading