Support native stdlib #201

katat · 2024-10-11T09:42:16Z

This PR adds initial native stdlib: comparator and bits, enabling use cases such as bit decomposition and compare values. (these are the two modules copied from the pending PR #197)

Besides, it includes a new download method for stdlib. Currently, instead of treating these stdlib as packages like the user/repo modules, it uses git clone to fetch the branch of a target repo. Then individually processes the .no files as different modules like std/bits and std/comparator.

Alternatively to relying on git command, stdlib could be downloadeded from the latest released version on github.

TODO:

Change the hardcoded repo for downloading stdlib(currently it uses my repo for testing)
Decide if downloading from github release page is preferred

katat · 2024-10-18T07:15:47Z

To avoid making this PR too big to review, there are 2 separate PRs depending on this one for the updates:

There is a bug that prevents the struct fields from being propagated for type checking and monomorphization. Fix: Allow struct fields to propagate constants #204
Add data types for u16/32/64. Add u16/32/64 #206

* fix: make struct field propagate constants

mimoo

so a few things:

we should definitely move to using this repo instead of your own repo ^^
we should also find a way to version the stdlib and allow updates, but maybe let's pun on that for now in the spirit of moving fast (this is related to versioning and release in general)
there's a number of changes that seem unrelated to this PR

src/tests/mod.rs

mimoo · 2024-10-20T00:11:57Z

src/stdlib/native/int.no

+use std::bits;
+
+struct Uint8 {
+    // todo: maybe add a const attribute to Field to forbid reassignment


imo if we do this we should do it the other way around: mark fields that can be mutated without going through the API (through the pub keyword like in rust)

oh yeah, using pub keyword makes more sense, as it is about whether it is accessible publicly.

btw generally in security you want to have a whitelist rather than a blacklist

mimoo · 2024-10-23T16:03:18Z

src/type_checker/mod.rs

+                return;
+            }
+        }
+    }
 }


why do we care if fields of a struct are const? I don't think this is a good idea (for example https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=f5142c25ee550e0a982421e8aa7ef876 won't compile)

src/stdlib/native/int.no

mimoo · 2024-10-23T16:08:57Z

src/stdlib/native/bits.no

+        let bit_num = unsafe nth_bit(value, index);
+
+        // constrain the bit_num to be 0 or 1
+        assert_eq(bit_num * (bit_num - 1), 0);


would be nice to be able to write Bool::new(bit_num) or Bool(bit_num) instead

but not sure how to do that ^^ builtin? a bool.no library? it's hard because the Bool type is a builtin type

src/stdlib/native/bits.no

src/stdlib/mod.rs

mimoo · 2024-10-23T20:01:42Z

src/cli/packages.rs

+pub fn download_stdlib() -> Result<()> {
+    // Hardcoded repository details and target branch
+    let repo_owner = "katat";
+    let repo_name = "noname";


is that a temporary solution :D

one thing we could do is have a github workflow to remove everything but the lib and push that on a stdlib branch or something. This way we can just git clone this repo with only the latest commit on that branch. wdyt?

otherwise it feels a bit wasteful to import all of noname?

ALTHOUGH, it would be cool if, when people click on "go to definition" we could also point to the source code of noname

so it might be an upside to have the whole noname code pulled

Let's create a release branch for now? so we can decide which version of main branch to be merged into the release branch for auto-download

mimoo · 2024-10-23T22:00:15Z

src/stdlib/mod.rs

+        let prefix_stdlib = Path::new(path_prefix);
+        let code = std::fs::read_to_string(prefix_stdlib.join(format!("{lib}.no"))).unwrap();
+        node_id =
+            typecheck_next_file(tast, Some(module), sources, lib.to_string(), code, 0).unwrap();


we should really have a pass that lists the dependencies use, with this approach we're processing a bunch of stdlibs that might not even be used :o

katat · 2024-10-24T03:11:19Z

we should definitely move to using this repo instead of your own repo ^^

How about we create a release branch in this repo for the auto download? This release branch allow us to control the latest version of stdlib.

we should also find a way to version the stdlib and allow updates, but maybe let's pun on that for now in the spirit of moving fast (this is related to versioning and release in general)

Another way to download code other than release branch, is to make releases in github release page and have the code to automatically download from the release page.

Beside, we would need to provide a way for the users to configure their noname packages to decide which versions of stdlib to download.

katat · 2024-10-24T04:54:25Z

there's a number of changes that seem unrelated to this PR

They are mostly about propagating the const value from struct fields to propagate self.bit_len as the constant argument for a generic function call. Maybe I should revert them until we got generic struct.

* add uint16/32/64 * allow [0][0]

katat · 2024-10-24T09:10:02Z

The CI error is due to the outdate stdlib code from my repo.

For the release branch (if we decided to use that branch for downloading stdlib), we would need to:

merge this branch into the main
init the release branch from this branch

Then it would resolve the CI error for the subsequent PRs.

mimoo · 2024-10-24T15:24:39Z

I think it's fine not to use a release branch and use whatever is the latest. Then we can add stdlib to the list of things we need to version/release correctly once we decide to tackle that :o

for the const fields in struct I think it can have its uses, but I'm a bit worried about edge cases (what if we use a const at some point, but then mutate the struct with a non-const, etc.

mimoo · 2024-10-25T18:25:31Z

.github/workflows/snarkjs.sh

@@ -9,6 +9,10 @@ fi
 DIR_PATH=$1
 CURVE=$2

+# Init stdlib in .noname/release/src/stdlib instead of downloading
+echo "Overriding stdlib in .noname/release/src/stdlib..."
+mkdir -p ~/.noname/release/src/stdlib/ && cp -r /app/noname/src/stdlib/* ~/.noname/release/src/stdlib/


I don't think we should do this manually BTW, we should let noname do it

This is because the CI test wants to test against the latest code, including the stdlib. To ensure the latest code, here overrides the default downloaded version with the code from the PR.

* add multiplexer stdlib

* add mimc stdlib

katat added 17 commits October 9, 2024 16:40

fix: avoid re-instantiate functions/methods

fbfa222

fix: monomorphized expr shouldn't override existing nodes

4eaed5b

fmt

10432c3

support unsafe/hint attributes

0221102

fmt

5353d42

fix: always check the resolved qualified name

253f4b5

incorporate native stdlib for testing

64f1758

add int module

be66e55

add comparator stdlib

6f26d5b

native bits stdlib

b283046

move init_stdlib_dep to stdlib

00d230b

add bits stdlib

1d2cba7

remove bits builtin

9b94dfb

download stdlib

64c2352

add comparator methods for uint8

3c0b022

fmt

f54a53c

remove unused import

5c8918e

katat requested review from StefanosChaliasos and gio54321 October 14, 2024 09:52

katat added 3 commits October 22, 2024 19:57

Fix: Allow struct fields to propagate constants (#204)

18825a3

* fix: make struct field propagate constants

add uint16/32/64

ca35175

update tests

55f441a

mimoo approved these changes Oct 23, 2024

View reviewed changes

mimoo reviewed Oct 23, 2024

View reviewed changes

katat added 2 commits October 24, 2024 12:16

fix: remove bit_len from uints

2b47a8d

simplify to_bits

8ef0993

katat mentioned this pull request Oct 24, 2024

Add Bool type in stdlib #213

Open

add const var STDLIB_DIRECTORY

1f8826a

katat added 2 commits October 24, 2024 12:52

move native stdlib to their own folder

8834fff

fmt

a822a1b

Add u16/32/64 (#206)

3f91646

* add uint16/32/64 * allow [0][0]

katat mentioned this pull request Oct 24, 2024

Support hint function declaration to bind with builtin function #199

Merged

remove deadcode

c2c78c9

point to zksecurity repo

328ffbf

katat changed the base branch from feat/hint-builtin-wir to main October 24, 2024 10:47

Merge remote-tracking branch 'origin/main' into feat/init-stdlib

7befb6b

katat added 6 commits October 25, 2024 11:18

remove deadcode

42a520a

doc bits

be8127d

use main branch as the release branch for downloading

665ef2e

clean up

ab75d85

ci: init stdlib from the latest code in pr

34cf5c4

add docs to stdlib

4cfb0c5

mimoo reviewed Oct 25, 2024

View reviewed changes

katat added 2 commits October 28, 2024 14:32

Support multiplexer (#207)

eb1def1

* add multiplexer stdlib

Add MIMC stdlib (#208)

83e3491

* add mimc stdlib

katat merged commit bde1e37 into main Oct 28, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support native stdlib #201

Support native stdlib #201

katat commented Oct 11, 2024 •

edited

Loading

katat commented Oct 18, 2024

mimoo left a comment •

edited

Loading

mimoo Oct 20, 2024

katat Oct 24, 2024

mimoo Oct 25, 2024

mimoo Oct 23, 2024

mimoo Oct 23, 2024

katat Oct 24, 2024

mimoo Oct 23, 2024

mimoo Oct 23, 2024

katat Oct 24, 2024

mimoo Oct 23, 2024

mimoo Oct 23, 2024

katat commented Oct 24, 2024

katat commented Oct 24, 2024 •

edited

Loading

katat commented Oct 24, 2024

mimoo commented Oct 24, 2024 •

edited

Loading

mimoo Oct 25, 2024

katat Oct 28, 2024

Support native stdlib #201

Support native stdlib #201

Conversation

katat commented Oct 11, 2024 • edited Loading

katat commented Oct 18, 2024

mimoo left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

katat commented Oct 24, 2024

katat commented Oct 24, 2024 • edited Loading

katat commented Oct 24, 2024

mimoo commented Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

katat commented Oct 11, 2024 •

edited

Loading

mimoo left a comment •

edited

Loading

katat commented Oct 24, 2024 •

edited

Loading

mimoo commented Oct 24, 2024 •

edited

Loading