Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI 2.0 - unified CI for C++, Go and RUST - PHASE1 #680

Merged
merged 65 commits into from
Dec 23, 2024
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
1306a84
added cpp-golang test for CI
mickeyasa Dec 8, 2024
620fcf6
fixed test name
mickeyasa Dec 8, 2024
94e4bb9
skipped check for changed files (wip)
mickeyasa Dec 8, 2024
78d8ab7
checking field
aviadingo Dec 8, 2024
481805e
field test
aviadingo Dec 8, 2024
fdf1389
final cpp-golang
aviadingo Dec 8, 2024
f40e1a8
disabled cpp and golang standalone tests
aviadingo Dec 8, 2024
babe767
bug fix
aviadingo Dec 8, 2024
8e15d34
add c++ examples to the unified test of c++/golang
aviadingo Dec 8, 2024
ec7717a
workflow bug fix
aviadingo Dec 8, 2024
b8fdf11
Compiling only non-main branches of cuda
aviadingo Dec 8, 2024
666399d
bug fix
aviadingo Dec 8, 2024
b8ea011
debug
aviadingo Dec 8, 2024
20906e3
setting export ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib
aviadingo Dec 8, 2024
cbc7234
change parameter for tests
aviadingo Dec 8, 2024
d8d543f
update examples workflow
aviadingo Dec 8, 2024
01cb8ce
bug fix
aviadingo Dec 8, 2024
cca941b
bug fix
aviadingo Dec 8, 2024
dcbeb2a
bug fix
aviadingo Dec 8, 2024
1490ba1
install alternate cuda backend on /tmp
aviadingo Dec 8, 2024
2f6633d
testing CUDA BE from different branch
aviadingo Dec 8, 2024
0d40632
test alternate backend
aviadingo Dec 8, 2024
4faa7e7
another try
aviadingo Dec 8, 2024
5467ee0
fix cuda flag
aviadingo Dec 8, 2024
0d2e643
updated cuda flag
aviadingo Dec 8, 2024
ad3f9df
aadded cuda lag to hash
aviadingo Dec 8, 2024
9ba5067
testing with latest cuda backend
aviadingo Dec 8, 2024
50dc6c5
get cuda repo ref
aviadingo Dec 15, 2024
b418ffb
Merge branch 'main' into aviad/CI
aviadingo Dec 15, 2024
316109b
verify main branch
aviadingo Dec 15, 2024
ad9ca09
wip
aviadingo Dec 15, 2024
1fd0614
rm fix
aviadingo Dec 15, 2024
72623f1
fix sha extraction
aviadingo Dec 15, 2024
be1aaca
bug fix
aviadingo Dec 15, 2024
412da03
moved examples to OLD
aviadingo Dec 15, 2024
24b0dcb
golang fix
aviadingo Dec 15, 2024
76d0bda
changed INSTALL_PATH
aviadingo Dec 15, 2024
729ff6f
hash fix
aviadingo Dec 15, 2024
a856cc8
install dir update
aviadingo Dec 15, 2024
4e4ce1a
fix hash
aviadingo Dec 15, 2024
83d2832
fix paths
aviadingo Dec 15, 2024
8bd2e55
fix go dirs
aviadingo Dec 15, 2024
4513004
minor fixes
aviadingo Dec 15, 2024
9b75d0c
alternative backend branch test
aviadingo Dec 15, 2024
3ccd0e4
alternative backend branch test
aviadingo Dec 15, 2024
077b296
main backend branch test
aviadingo Dec 15, 2024
f46b554
main backend branch test
aviadingo Dec 15, 2024
b1962ba
debug commit
aviadingo Dec 15, 2024
b2a5ed7
seperate paths for each curve
aviadingo Dec 16, 2024
ae3d55b
testing rust curve tests
aviadingo Dec 22, 2024
52cbf38
updated rust tests
aviadingo Dec 22, 2024
4320d59
typo fix
aviadingo Dec 22, 2024
3aefa1f
independent build.rs to bw6 (not reliant on bls12-377)
aviadingo Dec 22, 2024
dae9ad1
production ready
aviadingo Dec 23, 2024
86f88c6
Merge remote-tracking branch 'origin/main' into aviad/CI
aviadingo Dec 23, 2024
eaac996
removed commented code
aviadingo Dec 23, 2024
9f17364
remove install dir incase of reinstall to avoid conflicts
aviadingo Dec 23, 2024
5dba50b
added verbosity to ctest
aviadingo Dec 23, 2024
8201c7d
added action to changed-files list
aviadingo Dec 23, 2024
bbe910f
testing alternate backend branch
aviadingo Dec 23, 2024
87372b2
final test - main branch
aviadingo Dec 23, 2024
55198df
explicit arguments for grumpkin
aviadingo Dec 23, 2024
d1ecf6b
fixed grumpkin args
aviadingo Dec 23, 2024
0ef87c5
added timeout to go setup
aviadingo Dec 23, 2024
a49d46a
deleted old files
aviadingo Dec 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
File renamed without changes.
12 changes: 11 additions & 1 deletion .github/workflows/examples.yml → .github/OLD/examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,20 @@ jobs:
path: ./icicle/backend/cuda
ssh-key: ${{ secrets.CUDA_PULL_KEY }}
ref: ${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}
- name: Set CUDA backend flag
id: cuda-flag
run: |
if [ "${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}" == "main" ]; then
echo "ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib" >> $GITHUB_OUTPUT
else
echo "ICICLE_BACKEND_INSTALL_DIR=" >> $GITHUB_OUTPUT
fi
- name: c++ examples
working-directory: ./examples/c++
if: needs.check-changed-files.outputs.cpp == 'true' || needs.check-changed-files.outputs.examples == 'true'
run: |
# loop over all directories in the current directory
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
for dir in $(find . -mindepth 1 -maxdepth 1 -type d); do
if [ -d "$dir" ]; then
echo "Running command in $dir"
Expand All @@ -56,8 +65,9 @@ jobs:
- name: Rust examples
working-directory: ./examples/rust
if: needs.check-changed-files.outputs.rust == 'true' || needs.check-changed-files.outputs.examples == 'true'
run: |
run: |
# loop over all directories in the current directory
export ICICLE_BACKEND_INSTALL_DIR=${{ steps.cuda-flag.outputs.ICICLE_BACKEND_INSTALL_DIR }}
for dir in $(find . -mindepth 1 -maxdepth 1 -type d); do
if [ -d "$dir" ]; then
echo "Running command in $dir"
Expand Down
File renamed without changes.
File renamed without changes.
393 changes: 393 additions & 0 deletions .github/workflows/cpp-golang-rust.yml

Large diffs are not rendered by default.

72 changes: 36 additions & 36 deletions wrappers/rust/icicle-curves/icicle-bls12-377/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,46 +59,46 @@ fn main() {
println!("cargo:rustc-link-lib=icicle_curve_bls12_377");
println!("cargo:rustc-link-lib=icicle_hash");

if cfg!(feature = "bw6-761") {
// Base config
let mut config_bw = Config::new(format!("{}", icicle_src_dir.display()));
config_bw
.define("CURVE", "bw6_761")
.define("FIELD", "bw6_761")
.define("HASH", "OFF")
.define("CMAKE_INSTALL_PREFIX", &icicle_install_dir);
// if cfg!(feature = "bw6-761") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is that commented out?
We still support 761

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

761 got a standalone build.rs and no longer relies on bls12-377. I will remove the comments

// // Base config
// let mut config_bw = Config::new(format!("{}", icicle_src_dir.display()));
// config_bw
// .define("CURVE", "bw6_761")
// .define("FIELD", "bw6_761")
// .define("HASH", "OFF")
// .define("CMAKE_INSTALL_PREFIX", &icicle_install_dir);

// build (or pull and build) cuda backend if feature enabled.
// Note: this requires access to the repo
if cfg!(feature = "cuda_backend") {
config_bw.define("CUDA_BACKEND", "local");
} else if cfg!(feature = "pull_cuda_backend") {
config_bw.define("CUDA_BACKEND", "main");
}
if cfg!(feature = "metal_backend") {
config_bw.define("METAL_BACKEND", "local");
} else if cfg!(feature = "pull_metal_backend") {
config_bw.define("METAL_BACKEND", "main");
}
// // build (or pull and build) cuda backend if feature enabled.
// // Note: this requires access to the repo
// if cfg!(feature = "cuda_backend") {
// config_bw.define("CUDA_BACKEND", "local");
// } else if cfg!(feature = "pull_cuda_backend") {
// config_bw.define("CUDA_BACKEND", "main");
// }
// if cfg!(feature = "metal_backend") {
// config_bw.define("METAL_BACKEND", "local");
// } else if cfg!(feature = "pull_metal_backend") {
// config_bw.define("METAL_BACKEND", "main");
// }

// Optional Features that are default ON (so that default matches any backend)
if cfg!(feature = "no_bw6_g2") {
config_bw.define("G2", "OFF");
}
// if cfg!(feature = "no_ecntt") {
config_bw.define("ECNTT", "OFF");
// }
// // Optional Features that are default ON (so that default matches any backend)
// if cfg!(feature = "no_bw6_g2") {
// config_bw.define("G2", "OFF");
// }
// // if cfg!(feature = "no_ecntt") {
// config_bw.define("ECNTT", "OFF");
// // }

// Build
let _ = config_bw
.build_target("install")
.build();
// // Build
// let _ = config_bw
// .build_target("install")
// .build();

println!("cargo:rustc-link-search={}/lib", icicle_install_dir.display());
println!("cargo:rustc-link-lib=icicle_field_bw6_761");
println!("cargo:rustc-link-lib=icicle_curve_bw6_761");
println!("cargo:rustc-link-lib=icicle_hash");
}
// println!("cargo:rustc-link-search={}/lib", icicle_install_dir.display());
// println!("cargo:rustc-link-lib=icicle_field_bw6_761");
// println!("cargo:rustc-link-lib=icicle_curve_bw6_761");
// println!("cargo:rustc-link-lib=icicle_hash");
// }

println!("cargo:rustc-link-arg=-Wl,-rpath,{}/lib", icicle_install_dir.display()); // Add RPATH linker arguments

Expand Down
90 changes: 79 additions & 11 deletions wrappers/rust/icicle-curves/icicle-bw6-761/build.rs
Original file line number Diff line number Diff line change
@@ -1,23 +1,91 @@
use std::env;
use std::path::PathBuf;
// use std::env;
// use std::path::PathBuf;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove commented code

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


fn main() {
// Retrieve environment variables
let out_dir = env::var("OUT_DIR").expect("OUT_DIR is not set");
// fn main() {
// // Retrieve environment variables
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the rest of the curves/fields?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

761 was the only curve using a build that relies on another curve bls12-377 (probably due to legacy code).

so only 761 needs that update. new build script was tested and is running on the new CI

// let out_dir = env::var("OUT_DIR").expect("OUT_DIR is not set");

// Construct the path to the build directory
let build_dir = PathBuf::from(format!("{}/../../../", &out_dir));
// // Construct the path to the build directory
// let build_dir = PathBuf::from(format!("{}/../../../", &out_dir));

// // Construct the path to the deps directory
// let deps_dir = build_dir.join("deps");

// println!("cargo:rustc-link-arg=-Wl,-rpath,{}/icicle/lib", deps_dir.display()); // Add RPATH linker arguments

// // default backends dir
// if cfg!(feature = "cuda_backend") || cfg!(feature = "pull_cuda_backend") {
// println!(
// "cargo:rustc-env=ICICLE_BACKEND_INSTALL_DIR={}/icicle/lib/backend",
// deps_dir.display()
// );
// }
// }

use cmake::Config;
use std::{env, path::PathBuf};

fn main() {
// Construct the path to the deps directory
let out_dir = env::var("OUT_DIR").expect("OUT_DIR is not set");
let build_dir = PathBuf::from(format!("{}/../../../", &out_dir));
let deps_dir = build_dir.join("deps");

println!("cargo:rustc-link-arg=-Wl,-rpath,{}/icicle/lib", deps_dir.display()); // Add RPATH linker arguments
// Construct the path to icicle source directory
let main_dir = env::current_dir().expect("Failed to get current directory");
let icicle_src_dir = PathBuf::from(format!("{}/../../../../icicle", main_dir.display()));

println!("cargo:rerun-if-env-changed=CXXFLAGS");
println!("cargo:rerun-if-changed={}", icicle_src_dir.display());

// Base config
let mut config = Config::new(format!("{}", icicle_src_dir.display()));
// Check if ICICLE_INSTALL_DIR is defined
let icicle_install_dir = if let Ok(dir) = env::var("ICICLE_INSTALL_DIR") {
PathBuf::from(dir)
} else {
// Define the default install directory to be under the build directory
PathBuf::from(format!("{}/icicle/", deps_dir.display()))
};
config
.define("CURVE", "bw6_761")
.define("HASH", "OFF")
.define("CMAKE_INSTALL_PREFIX", &icicle_install_dir);

// build (or pull and build) cuda backend if feature enabled.
// Note: this requires access to the repo
if cfg!(feature = "cuda_backend") {
config.define("CUDA_BACKEND", "local");
} else if cfg!(feature = "pull_cuda_backend") {
config.define("CUDA_BACKEND", "main");
}
if cfg!(feature = "metal_backend") {
config.define("METAL_BACKEND", "local");
} else if cfg!(feature = "pull_metal_backend") {
config.define("METAL_BACKEND", "main");
}

// Build
let _ = config
.build_target("install")
.build();

println!("cargo:rustc-link-search={}/lib", icicle_install_dir.display());
println!("cargo:rustc-link-lib=icicle_field_bw6_761");
println!("cargo:rustc-link-lib=icicle_curve_bw6_761");
println!("cargo:rustc-link-lib=icicle_hash");
println!("cargo:rustc-link-arg=-Wl,-rpath,{}/lib", icicle_install_dir.display()); // Add RPATH linker arguments

// default backends dir
// default backends dir
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove line

if cfg!(feature = "cuda_backend") || cfg!(feature = "pull_cuda_backend") {
if cfg!(feature = "cuda_backend")
|| cfg!(feature = "pull_cuda_backend")
|| cfg!(feature = "metal_backend")
|| cfg!(feature = "pull_metal_backend")
{
println!(
"cargo:rustc-env=ICICLE_BACKEND_INSTALL_DIR={}/icicle/lib/backend",
deps_dir.display()
"cargo:rustc-env=ICICLE_BACKEND_INSTALL_DIR={}/lib/backend",
icicle_install_dir.display()
);
}
}
Loading