Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Orion PCS Integration #153

Merged
merged 77 commits into from
Jan 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
0fe9dc4
Orion PCS implementation prorotyped
tonyfloatersu Nov 21, 2024
0694446
rust compilation cache again?
tonyfloatersu Nov 22, 2024
6ed985a
orion interface changes, working on aligning with pcs trait
tonyfloatersu Nov 23, 2024
4e81cd9
orion commit from multilinear poly over simd fields
tonyfloatersu Nov 23, 2024
533be26
minor, refactoring orion opening base field, need to work on orion op…
tonyfloatersu Nov 24, 2024
a84e286
prototype simd version of orion opening, need to get some rest and ch…
tonyfloatersu Nov 24, 2024
14f8532
minor, move original orion verify to base field impl
tonyfloatersu Nov 24, 2024
a81d45d
minor, renaming from Circuit*F to Simd*F
tonyfloatersu Nov 24, 2024
4e08b59
minor, prototype for orion verify for simd field
tonyfloatersu Nov 24, 2024
ad2581d
minor, typo
tonyfloatersu Nov 24, 2024
23ce12a
move test to base field test, working on simd test
tonyfloatersu Nov 25, 2024
a065d8b
orion simd commitment consistency test
tonyfloatersu Nov 25, 2024
e664a4b
started working on test orion pcs simd full e2e
tonyfloatersu Nov 25, 2024
955d2b4
e2e test finalized, prototype first round of polishing starts
tonyfloatersu Nov 26, 2024
164ff4c
minor polishing and benchmarks for orion over simd polynomial
tonyfloatersu Nov 26, 2024
ba2ce01
minor, remove old cloned iterator
tonyfloatersu Nov 26, 2024
7838aad
standard pcs trait implementation for both flavors of orion impls
tonyfloatersu Nov 26, 2024
a885ed5
orion simd pcs aligning with PCSForExpanderGKR, restrict constraint t…
tonyfloatersu Nov 27, 2024
0f42507
simplified type constraints for orion, no need for SimdEvalF
tonyfloatersu Nov 27, 2024
441482b
minor, fixing up a unattended impl for standard pcs for simd orion
tonyfloatersu Nov 27, 2024
73e8baa
minor, one liner change
tonyfloatersu Nov 28, 2024
a74cf76
working on mpi versioned commit/open/verify algorithm
tonyfloatersu Nov 28, 2024
3d9006e
change base pcs impl testing to test_pcs from common
tonyfloatersu Nov 28, 2024
f67463d
tame the CI
tonyfloatersu Nov 28, 2024
0e85b7a
testing harness minor changes
tonyfloatersu Nov 29, 2024
1673d3b
minor, renamings and using params in standard pcs trait
tonyfloatersu Nov 29, 2024
e000787
pcs trait impl file too large, separate into 2 files
tonyfloatersu Nov 29, 2024
1fd5655
working on mpi version of open/verify, based on assumption that only …
tonyfloatersu Nov 29, 2024
2c771f5
prototype in orion simd verify aggregated version
tonyfloatersu Nov 30, 2024
0112511
separate aggregated verify for orion simd field into a standalone file
tonyfloatersu Nov 30, 2024
41fb654
minor, refactoring the prototype as a personal review
tonyfloatersu Nov 30, 2024
d436436
aggregate then verify test complete, tweaking mpi environment
tonyfloatersu Dec 1, 2024
1cc2348
minor, refactoring by abstracting out aggregation method
tonyfloatersu Dec 1, 2024
1ec5b76
tame CI clippy
tonyfloatersu Dec 1, 2024
27858e3
at least the mpi exec e2e runs on my machine for now... wait wtf mpi?
tonyfloatersu Dec 3, 2024
edfb4e5
experimental move, should discuss with folks about if it is a good move
tonyfloatersu Dec 4, 2024
9ab727a
experimental move, mpi opening each party precalculate weighted version
tonyfloatersu Dec 4, 2024
213ca4a
Revert "experimental move, mpi opening each party precalculate weight…
tonyfloatersu Dec 5, 2024
f1e298c
use common toolset in pcs testing
tonyfloatersu Dec 5, 2024
df91b0a
pcs init for testing, take rng from the context
tonyfloatersu Dec 5, 2024
1bd79e4
minor - refactoring in favor of poly_commit lib export
tonyfloatersu Dec 5, 2024
446b838
minor - reformatting for code structure
tonyfloatersu Dec 5, 2024
ca88004
minor, circle back to the improved query complexity
tonyfloatersu Dec 5, 2024
fd27a00
Merge branch 'dev' into feature/orion
tonyfloatersu Dec 17, 2024
7d6436b
minor - use multilinear extension trait for orion base field impl
tonyfloatersu Dec 17, 2024
a609b6c
allow for parsing gf2 orion gkr config
tonyfloatersu Dec 18, 2024
28dc09b
use fixed rng seeds to generate deterministic pcs crs in mpi cases
tonyfloatersu Dec 18, 2024
8bcd415
adapting to gkr codebase - setup with number of variables in each pol…
tonyfloatersu Dec 19, 2024
db77168
integrated orion gf2 config into gkr correctness test - single proces…
tonyfloatersu Dec 19, 2024
527b21f
Merge branch 'dev' into feature/orion
tonyfloatersu Dec 21, 2024
403f23e
align fiat shamir transcript among all mpi parties
tonyfloatersu Dec 22, 2024
894cf89
integrated into the benchmarking
tonyfloatersu Dec 22, 2024
ff730e1
inline things from multilinear extension
tonyfloatersu Dec 22, 2024
3e65000
Merge branch 'dev' into feature/orion
tonyfloatersu Dec 23, 2024
50064b8
Merge branch 'dev' into feature/orion
tonyfloatersu Dec 30, 2024
1edbea0
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 2, 2025
04d7f34
base field orion pcs support for M31 prime
tonyfloatersu Jan 10, 2025
4fce220
update on verifier side - sync up the randomness on verifier side
tonyfloatersu Jan 10, 2025
fcce135
orion m31 base pcs commit and open benchmarking
tonyfloatersu Jan 10, 2025
cea7ebd
M31 simd commit testing
tonyfloatersu Jan 10, 2025
0274b94
minor - updated orion implementation with m31 support prototype
tonyfloatersu Jan 12, 2025
b1cfa70
minor - each sumcheck IOP prove/verify sync up randomness beforehand
tonyfloatersu Jan 12, 2025
075abe8
benchmarking m31 case
tonyfloatersu Jan 12, 2025
0ba1d8b
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 13, 2025
7fe83ca
push to 2^32 GF2 benchmark size
tonyfloatersu Jan 13, 2025
16a67f7
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 14, 2025
0c35935
Add some notes to 'transcript_verifier_sync' and make clippy happy
zhiyong1997 Jan 17, 2025
511d8fb
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 17, 2025
fdc0f01
trying to isolate potential problems
tonyfloatersu Jan 17, 2025
22abffa
smaller test size may work?
tonyfloatersu Jan 17, 2025
09c6054
still trying smaller test size...
tonyfloatersu Jan 17, 2025
a2f62ae
hamming weight naming suggestion
tonyfloatersu Jan 20, 2025
ebdb24f
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 20, 2025
75530c9
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 20, 2025
81fae91
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 20, 2025
4f05d70
Merge branch 'dev' into feature/orion
tonyfloatersu Jan 21, 2025
25d4461
Minor: Orion related additional test (#180)
tonyfloatersu Jan 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
matrix:
include:
- os: macos-latest
- os: ubuntu-latest
# - os: ubuntu-latest

steps:
- name: Checkout code
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions config/config_macros/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ quote = "1.0" # For generating code
proc-macro2 = "1.0" # For working with tokens

[dev-dependencies]
gf2 = { path = "../../arith/gf2" }
mersenne31 = { path = "../../arith/mersenne31/" }

[lib]
Expand Down
16 changes: 13 additions & 3 deletions config/config_macros/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ fn parse_fiat_shamir_hash_type(
}

fn parse_polynomial_commitment_type(
field_type: &str,
field_config: &str,
transcript_type: &str,
polynomial_commitment_type: ExprPath,
Expand All @@ -99,11 +100,19 @@ fn parse_polynomial_commitment_type(
.expect("Empty path for polynomial commitment type");

let pcs_type_str = binding.ident.to_string();
match pcs_type_str.as_str() {
"Raw" => (
match (pcs_type_str.as_str(), field_type) {
("Raw", _) => (
"Raw".to_owned(),
format!("RawExpanderGKR::<{field_config}, {transcript_type}>").to_owned(),
),
("Orion", "GF2") => (
"Orion".to_owned(),
format!("OrionPCSForGKR::<{field_config}, GF2x128, {transcript_type}>").to_owned(),
),
("Orion", "M31") => (
"Orion".to_owned(),
format!("OrionPCSForGKR::<{field_config}, M31x16, {transcript_type}>").to_owned(),
),
_ => panic!("Unknown polynomial commitment type in config macro expansion"),
}
}
Expand Down Expand Up @@ -134,7 +143,8 @@ fn declare_gkr_config_impl(input: proc_macro::TokenStream) -> proc_macro::TokenS
let (fiat_shamir_hash_type, transcript_type) =
parse_fiat_shamir_hash_type(&field_type, &field_config, fiat_shamir_hash_type_expr);
let (polynomial_commitment_enum, polynomial_commitment_type) = parse_polynomial_commitment_type(
field_config.as_str(),
&field_type,
&field_config,
&transcript_type,
polynomial_commitment_type,
);
Expand Down
20 changes: 18 additions & 2 deletions config/config_macros/tests/macro_expansion.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@ use gkr_field_config::FieldType;
use config::GKRConfig;
use config_macros::declare_gkr_config;
use field_hashers::{MiMC5FiatShamirHasher, PoseidonFiatShamirHasher};
use gf2::GF2x128;
use gkr_field_config::{BN254Config, GF2ExtConfig, GKRFieldConfig, M31ExtConfig};
use mersenne31::M31x16;
use poly_commit::raw::RawExpanderGKR;
use poly_commit::{OrionPCSForGKR, RawExpanderGKR};
use transcript::{BytesHashTranscript, FieldHashTranscript, Keccak256hasher, SHA256hasher};

fn print_type_name<Cfg: GKRConfig>() {
Expand All @@ -26,11 +27,17 @@ fn main() {
PolynomialCommitmentType::Raw
);
declare_gkr_config!(
M31PoseidonConfig,
M31PoseidonRawConfig,
FieldType::M31,
FiatShamirHashType::Poseidon,
PolynomialCommitmentType::Raw
);
declare_gkr_config!(
M31PoseidonOrionConfig,
FieldType::M31,
FiatShamirHashType::Poseidon,
PolynomialCommitmentType::Orion
);
declare_gkr_config!(
BN254MIMCConfig,
FieldType::BN254,
Expand All @@ -43,8 +50,17 @@ fn main() {
FiatShamirHashType::Keccak256,
PolynomialCommitmentType::Raw
);
declare_gkr_config!(
GF2Keccak256OrionConfig,
FieldType::GF2,
FiatShamirHashType::Keccak256,
PolynomialCommitmentType::Orion
);

print_type_name::<M31Sha256Config>();
print_type_name::<M31PoseidonRawConfig>();
print_type_name::<M31PoseidonOrionConfig>();
print_type_name::<BN254MIMCConfig>();
print_type_name::<GF2Keccak256Config>();
print_type_name::<GF2Keccak256OrionConfig>();
}
8 changes: 4 additions & 4 deletions config/mpi_config/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -134,16 +134,16 @@ impl MPIConfig {

/// Return an u8 vector sharing THE SAME MEMORY SLOT with the input.
#[inline]
unsafe fn vec_to_u8_bytes<F: Field>(vec: &Vec<F>) -> Vec<u8> {
unsafe fn vec_to_u8_bytes<F: Sized>(vec: &Vec<F>) -> Vec<u8> {
Vec::<u8>::from_raw_parts(
vec.as_ptr() as *mut u8,
vec.len() * F::SIZE,
vec.capacity() * F::SIZE,
vec.len() * size_of::<F>(),
vec.capacity() * size_of::<F>(),
)
}

#[allow(clippy::collapsible_else_if)]
pub fn gather_vec<F: Field>(&self, local_vec: &Vec<F>, global_vec: &mut Vec<F>) {
pub fn gather_vec<F: Sized + Clone>(&self, local_vec: &Vec<F>, global_vec: &mut Vec<F>) {
unsafe {
if self.world_size == 1 {
*global_vec = local_vec.clone()
Expand Down
5 changes: 3 additions & 2 deletions gkr/src/executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ use config::{
};
use config_macros::declare_gkr_config;
use field_hashers::{MiMC5FiatShamirHasher, PoseidonFiatShamirHasher};
use gf2::GF2x128;
use gkr_field_config::{BN254Config, GF2ExtConfig, GKRFieldConfig, M31ExtConfig};
use mersenne31::M31x16;
use poly_commit::{expander_pcs_init_testing_only, raw::RawExpanderGKR};
use poly_commit::{expander_pcs_init_testing_only, raw::RawExpanderGKR, OrionPCSForGKR};
use rand::SeedableRng;
use rand_chacha::ChaCha12Rng;
use transcript::{BytesHashTranscript, FieldHashTranscript, SHA256hasher};
Expand Down Expand Up @@ -289,5 +290,5 @@ declare_gkr_config!(
pub GF2ExtConfigSha2,
FieldType::GF2,
FiatShamirHashType::SHA256,
PolynomialCommitmentType::Raw
PolynomialCommitmentType::Orion
);
5 changes: 3 additions & 2 deletions gkr/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@ use circuit::Circuit;
use clap::Parser;
use config::{Config, GKRConfig, GKRScheme};
use config_macros::declare_gkr_config;
use gf2::GF2x128;
use gkr_field_config::{BN254Config, GF2ExtConfig, GKRFieldConfig, M31ExtConfig};
use mpi_config::MPIConfig;

use poly_commit::{expander_pcs_init_testing_only, raw::RawExpanderGKR};
use poly_commit::{expander_pcs_init_testing_only, raw::RawExpanderGKR, OrionPCSForGKR};
use rand::SeedableRng;
use rand_chacha::ChaCha12Rng;
use transcript::{BytesHashTranscript, SHA256hasher};
Expand Down Expand Up @@ -72,7 +73,7 @@ fn main() {
GF2ExtConfigSha2,
FieldType::GF2,
FiatShamirHashType::SHA256,
PolynomialCommitmentType::Raw
PolynomialCommitmentType::Orion
);

match args.field.as_str() {
Expand Down
5 changes: 3 additions & 2 deletions gkr/src/main_mpi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@ use config::{Config, GKRConfig, GKRScheme};
use config_macros::declare_gkr_config;
use mpi_config::MPIConfig;

use gf2::GF2x128;
use gkr_field_config::{BN254Config, GF2ExtConfig, GKRFieldConfig, M31ExtConfig};
use poly_commit::{expander_pcs_init_testing_only, raw::RawExpanderGKR};
use poly_commit::{expander_pcs_init_testing_only, raw::RawExpanderGKR, OrionPCSForGKR};
use rand::SeedableRng;
use rand_chacha::ChaCha12Rng;
use transcript::{BytesHashTranscript, SHA256hasher};
Expand Down Expand Up @@ -63,7 +64,7 @@ fn main() {
GF2ExtConfigSha2,
FieldType::GF2,
FiatShamirHashType::SHA256,
PolynomialCommitmentType::Raw
PolynomialCommitmentType::Orion
);

match args.field.as_str() {
Expand Down
24 changes: 22 additions & 2 deletions gkr/src/tests/gkr_correctness.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ use circuit::Circuit;
use config::{Config, FiatShamirHashType, GKRConfig, GKRScheme, PolynomialCommitmentType};
use config_macros::declare_gkr_config;
use field_hashers::{MiMC5FiatShamirHasher, PoseidonFiatShamirHasher};
use gf2::GF2x128;
use gkr_field_config::{BN254Config, FieldType, GF2ExtConfig, GKRFieldConfig, M31ExtConfig};
use mersenne31::M31x16;
use mpi_config::{root_println, MPIConfig};
use poly_commit::expander_pcs_init_testing_only;
use poly_commit::raw::RawExpanderGKR;
use poly_commit::{expander_pcs_init_testing_only, OrionPCSForGKR, RawExpanderGKR};
use rand::{Rng, SeedableRng};
use rand_chacha::ChaCha12Rng;
use sha2::Digest;
Expand Down Expand Up @@ -67,12 +67,24 @@ fn test_gkr_correctness() {
FiatShamirHashType::MIMC5,
PolynomialCommitmentType::Raw
);
declare_gkr_config!(
C7,
FieldType::GF2,
FiatShamirHashType::Keccak256,
PolynomialCommitmentType::Orion,
);
declare_gkr_config!(
C8,
FieldType::M31,
FiatShamirHashType::Poseidon,
PolynomialCommitmentType::Raw,
);
declare_gkr_config!(
C9,
FieldType::M31,
FiatShamirHashType::Poseidon,
PolynomialCommitmentType::Orion,
);

test_gkr_correctness_helper(
&Config::<C0>::new(GKRScheme::Vanilla, mpi_config.clone()),
Expand Down Expand Up @@ -102,10 +114,18 @@ fn test_gkr_correctness() {
&Config::<C6>::new(GKRScheme::Vanilla, mpi_config.clone()),
Some("../data/gkr_proof.txt"),
);
test_gkr_correctness_helper(
&Config::<C7>::new(GKRScheme::Vanilla, mpi_config.clone()),
None,
);
test_gkr_correctness_helper(
&Config::<C8>::new(GKRScheme::Vanilla, mpi_config.clone()),
None,
);
test_gkr_correctness_helper(
&Config::<C9>::new(GKRScheme::Vanilla, mpi_config.clone()),
None,
);

MPIConfig::finalize();
}
Expand Down
8 changes: 8 additions & 0 deletions poly_commit/src/orion.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ pub use simd_field_impl::{
#[cfg(test)]
mod simd_field_tests;

mod simd_field_agg_impl;

#[cfg(test)]
mod simd_field_agg_tests;

mod pcs_for_expander_gkr;
pub use pcs_for_expander_gkr::OrionPCSForGKR;

mod pcs_trait_impl;
pub use pcs_trait_impl::{OrionBaseFieldPCS, OrionSIMDFieldPCS};

Expand Down
Loading
Loading