Update CI and build process

[ErikKaum]

Resolves: #18 and #21

Tasks for the python package:

• cross compile to native lib to the architectures we want to support
• add tests in CI for all cross compiled architectures (the tests is to import the package and print version)
• add type info for the functions from the rust extension
• update the pypi release CI script accordingly

Tasks for the rust crate:

• CI script for publishing on crates.io
• add documentation & publish to docs.rs, deferred**
• (maybe) add tests to the rust side*

*we probably want to start by duplicating the tests from python to rust --> run them in parallel for a while --> once we're confident enough we can then start deleting some of the duplicate tests.

Supported python versions and architectures in the PR:

• python versions: 3.7 3.8 3.9 3.10 3.11 3.12 (only CPython)
• archs:
  • Linux: x86_64, i686 and aarch64
  • Windows: AMD64 and x86
  • MacOs x86_64 and arm64

notably musllinux is not on the list, maybe we want to support it?
Edit: yes, the alpine linux (docker image) uses musl libc, so it makes sense supporting it.

**Edit: I'll defer any more in depth rust code documentation. The code itself is quite self documenting when published to crates & docs. More detailed stuff is probably better done once we have a more clear understanding what the API will look like in the long run.

Additional question: A lot of the CI workflow are such that they cannot be tested without actually making a release. So it's quite likely that they have some bugs. I'd recommend that we do a pre-release. Something like version 0.1.0-dev0 to make sure everything runs smoothly. The requirement would thus be that we are comfortable in making this work somewhat public. Would that be okay for you?

[ErikKaum]

Btw if everything's cool could you @brandonwillard merge it? I don't seem to have the permissions.

[brandonwillard]

Looks good! I set up the tokens and tried the crates.io publish step locally, so it should work; however, I noticed that we might want to change the crate name to simply outlines-core and we might also need to explicitly list the license type.

[ErikKaum]

Looks good! I set up the tokens and tried the crates.io publish step locally, so it should work; however, I noticed that we might want to change the crate name to simply outlines-core and we might also need to explicitly list the license type.

Thank you 👍
Great! That's a good point, I changed of the package & lib the name in the Cargo.toml file. It still is imported to the python side as outlines_core_rs configuration in setup.py, which is probably good to avoid name collisions. Also added the Apache-2.0 as the license in Cargo.toml

[umut-sahin]

We don't need to do anything for publish to docs.rs.

docs.rs automatically publishes all crates uploaded to crates.io (see https://docs.rs/releases/queue).

[ErikKaum]

Regarding the failing test: this seems to fail on main as well. Not sure if something was updated in the NousResearch/Hermes-2-Pro-Llama-3-8B tokenizer which uncovered something new in our test and therefore broke them. Or if it changed in a way that just breaks the contract.

Regardless, maybe we shouldn't block this PR on that test failing, but rather have a separate issue for it?

[brandonwillard]

Regardless, maybe we shouldn't block this PR on that test failing, but rather have a separate issue for it?

Yeah, it looks like we have some rebasing to do.

[ErikKaum]

Indeed! Thank you for merging 👍