feat: publish workspace crates #4986
Draft
+3,970
−3,301
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This moves our workspace into a `penumbra-sdk` namespace ahead of publishing `v0.82.0-alpha.0` on crates.io. We're using the `penumbra-sdk` prefix for all crate names because it's unique. (The `penumbra`) crate name is already taken on crates.io and belongs to an unrelated project. In addition to a renaming, several other changes are made for crates.io compatibility: * fills out crate metadata fields to meet spec * breaks out app-tests to separate crate
We want all the crates Penumbra protocol workspace to be published to crates.io, so that external developers can depend on them in downstream projects. For now, we'll skip doing so for the binaries, but we can circle back on them. The major obstacle is that the `penumbra-sdk-proof-params` create contains binary keyfiles, which are managed in git-lfs. We cannot upload the raw keyfiles to crates.io, because that'd result in a 100MB crate. Instead, we use a custom script to revert the binary keyfiles to plaintext lfs pointers immediately prior to publishing to crates.io, which stays under the limit, and allows third-party tools to opt into downloading the key material via the `download-proving-keys` feature. The ratelimit of "20s" between crates publishing has been working reliably when publishing alpha versions (i.e. an HTTP 429 is avoided). Publishing all crates takes ~12m. This commit also condenses several "alpha" series releases, culminating in `0.82.0-alpha.15`. Most of these versions have been published to crates.io, to evaluate CI behavior on publishing crates. build: warn-only if download-proving-keys not set The goal is to warn if keys are not set, but still permit the build.
conorsch
force-pushed
the
4978-crate-publishing-take-2
branch
from
ef841dd to
e6f5c30
Compare
conorsch
commented
Jan 15, 2025
| //! | ||
| //! Instead, we'll upload just git raw git-lfs pointer files when publishing to crates.io, | ||
| //! then use the build.rs logic to fetch the assets ahead of compilation. Use the feature | ||
| //! `download-proving-keys` to enable the auto-download behavior. | ||
| use anyhow::Context; |
There was a problem hiding this comment.
The changes to the proof-params build.rs file warrants close scrutiny.
| # https://github.com/git-lfs/git-lfs/issues/951#issuecomment-581477084 | ||
| # This is a destructive action! Which is why we checked for dirty tree above. | ||
| git read-tree HEAD && GIT_LFS_SKIP_SMUDGE=1 git checkout -f HEAD | ||
|
|
There was a problem hiding this comment.
The GIT_LFS_SKIP_SMUDGE=1 approach is slightly cursed: we're altering in-repo assets immediately prior to crate publication. This works™, though, so I'm suggesting it as the right-sized solution to get the workspace published.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes
Workspace refactor, preparing to publish the Penumbra crates. In doing so, renames all crates to have a prefix
penumbra-sdk-, which is unique and controlled by PL. There's also a custom script that handles the publication action, which is non-standard.The major obstacle is that the
penumbra-sdk-proof-paramscrate contains binary keyfiles, which are managed in git-lfs. We cannot upload the raw keyfiles to crates.io, because that'd result in a 100MB crate. Instead, we use a custom script to revert the binary keyfiles to plaintext lfs pointers immediately prior to publishing to crates.io, which stays under the limit, and allows third-party tools to opt into downloading the key material via thedownload-proving-keysfeature.Given that
docs.rsdoesn't permit network access during crate build, we default to warning about the missing keys during build of proof-params, recommending enabling the option. We could go further add fail the build, conditionally warning only in the docsrs build context. I chose to minimize special-casing as much as possible.Issue ticket number and link
Towards #4978.
Testing and review
Checklist before requesting a review
I have added guiding text to explain how a reviewer should test these changes.
If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason: