feat(extensions): install from external registries #2968

smallstepman · 2023-02-13T20:33:05Z

Description

Modifies the extension install command to take a flag --registry that allows the registry to install from to be specified at runtime

dfx extension install foo --registry <other-compatibility.json>

Implements https://dfinity.atlassian.net/browse/SDK-963

How Has This Been Tested?

added e2e test

Checklist:

The title of this PR complies with Conventional Commits.
I have edited the CHANGELOG accordingly.
I have made corresponding changes to the documentation.

…es-to-be-defined-for-dfx-extensions

ericswanson-dfinity · 2023-03-02T17:52:30Z

src/dfx/src/util/terminal.rs

+ .context("Unable to read input")?;
+ let input_string = input_string.trim_end();
+ if input_string != "yes" {
+ bail!("Refusing to install canister without approval");


This message doesn't match the context (installing an extension)

ericswanson-dfinity · 2023-07-20T21:24:35Z

e2e/tests-dfx/extension.bash

+ # strip semver suffix
+ DFX_VERSION=$(dfx --version | sed 's/-.*//' | cut -d ' ' -f 2)
+ # find unoocupied port
+ port=49152


See start_webserver, stop_webserver

Example usage: https://github.com/dfinity/sdk/blob/master/e2e/tests-dfx/deps.bash#L24

ericswanson-dfinity · 2023-07-20T21:29:44Z

CHANGELOG.md

+The registry file is a JSON file that contains a list of extensions. Here is an example:
+
+```json
+{
+ "compatibility": {
+ "0.15.0": {
+ "your_extension": {
+ "versions": ["0.1.0"]
+ }
+ }
+ },
+ "extensions": {
+ "your_extension": {
+ "0.1.0": {
+ "homepage": "https://github.com/your-org/your-repo",
+ "authors": "Your Org",
+ "summary": "Your extension",
+ "categories": ["development"],
+ "keywords": ["cli-helper"],
+ "description": "A longer description.",
+ "subcommands": {
+ "do_something": {
+ "about": "does something",
+ "args": {
+ "the_param": {
+ "about": "some paramater",
+ "long": "the-param"
+ }
+ }
+ }
+ },
+ "binaries": {
+ "unknown-linux-gnu-x86_64": {
+ "url": "https://raw.githubusercontent.com/your-org/your-repo/master/your_extension-v0.1.0-x86_64-unknown-linux-gnu.tar.gz",
+ "sha256": "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890"
+ },
+ "apple-darwin-x86_64": {
+ "url": "https://raw.githubusercontent.com/your-org/your-repo/master/your_extension-v0.1.0-x86_64-apple-darwin.tar.gz ",
+ "sha256": "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890"
+ },
+ "apple-darwin-aarch64": {
+ "url": "https://raw.githubusercontent.com/your-org/your-repo/master/your_extension-v0.1.0-aarch64-apple-darwin.tar.gz ",
+ "sha256": "abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890"
+ }
+ }
+ }
+ }
+ }
+}
+```
+
+The extension must be published as a tarball (`.tag.gz`), which contains a directory named `<EXTENSION_NAME>-v<EXTENSION_VERSION>-<CPU_ARCHITECTURE>-<PLATFORM>`, which contains a binary named `<EXTENSION_NAME>`, here is minimal example:
+
+```bash
+mkdir test_extension-v0.1.0-x86_64-unknown-linux-gnu
+cat > test_extension-v0.1.0-x86_64-unknown-linux-gnu/test_extension << "EOF"
+#!/usr/bin/env bash
+
+echo Hello, World!
+EOF
+tar -czf test_extension-v0.1.0-x86_64-unknown-linux-gnu.tar.gz test_extension-v0.1.0-x86_64-unknown-linux-gnu
+# you'll also need to generate a sha256 hash of the tarball and put it into the registry file
+shasum -a 256 test_extension-v0.1.0-x86_64-unknown-linux-gnu.tar.gz 
+```


This is a little too much to put in the dfx changelog. A better place for it would be a document in https://github.com/dfinity/dfx-extensions/docs which describes how to set up your own extension registry and publish extensions from it. Then, we can link to that document from here.

ericswanson-dfinity · 2023-07-20T21:33:09Z

CHANGELOG.md

@@ -4,6 +4,80 @@

 ## DFX

+### feat: dfx extension can now be installed from external registries
+
+Modifies the `dfx extension install` command to take a flag `--registry` that allows installing extensions from external registries. Here is an example:


Suggestion:

The dfx extension install command now supports --registry <url to registry json> for installing an extension from an external registry. Example usage:

dfx extension install your_extension --registry https://raw.githubusercontent.com/your-org/your-repo/master/dfx-extensions-registry.json

For more information about hosting external registries, please see https://github.com/dfinity/dfx-extensions/docs/hosting-external-registries.md

ericswanson-dfinity · 2023-07-20T21:46:47Z

src/dfx-core/src/extension/manager/install.rs

 }
 Ok(())
 }
 }

+fn process_external_registry_manifest(


Would you please explain the rationale for having a different scheme for storing/obtaining the manifest for external registries vs the the dfx-extensions repo?

I see in the PR description

dfx extension install foo --registry https://github.com/org/repo/other-compatibility.json

which seems like it was closer to what's going on in dfx-extensions. What didn't work out with that?

Sure, tho I'm not sure if I completely understand the question, but... the manifest from dfinity/dfx-extensions is of a different format than the one specified by the external manifest. As specified in design doc:

In order for extensions stored in external repositories to be consumable by dfx, the repository maintainers must provide a URL to a JSON file with the following structure. This file is an amalgamation of the compatibility.json file and the combined ~~manifest.json~~ extension.json files for each individual extension, as they exist in the DFINITY extension repository.

Here are some of the differences

dfinity/dfx-extensions external registry

checksum well-known location: https://github.com/dfinity/dfx-extensions/releases/<GIT_TAG>/downloads/<ARCHIVE_FILENAME>.sha256 supplied in registry file

tarball download url well-known location: https://github.com/dfinity/dfx-extensions/releases/<GIT_TAG>/downloads/<ARCHIVE_FILENAME> supplied in registry file

supported arch/platform all platforms supported by dfx are supported by official extensions determined by fetching value of .extensions.<EXT>.<VER>.binaries.<ARCH-PLATFORM> from registry file, if key is missing then platform is not supported

extension.json unpacked from tarball extracted from registry file

To process both DFINITY extensions and external extensions using the same logic, would mean that we require extension developers to use only GitHub and mimic dfinity/dfx-extensions release process

smallstepman marked this pull request as ready for review March 2, 2023 17:43

smallstepman requested a review from a team as a code owner March 2, 2023 17:43

smallstepman mentioned this pull request Mar 2, 2023

feat(extensions): add --install-as flag to dfx extension install command #2998

Merged

3 tasks

smallstepman marked this pull request as draft June 19, 2023 10:09

smallstepman mentioned this pull request Jul 18, 2023

ci(release): add checksum file dfinity/dfx-extensions#35

Merged

smallstepman marked this pull request as ready for review July 20, 2023 14:28

smallstepman requested a review from chenyan-dfinity as a code owner July 20, 2023 14:28

feat: install extensions from external registries

da6b933

smallstepman force-pushed the SDK-963-enable-multiple-extension-registries-to-be-defined-for-dfx-extensions branch from 3d49880 to da6b933 Compare July 20, 2023 16:17

lint and fix

9d6b445

smallstepman force-pushed the SDK-963-enable-multiple-extension-registries-to-be-defined-for-dfx-extensions branch from 5b1007f to 9d6b445 Compare July 20, 2023 18:38

lint and changelog

e6f5e1b

smallstepman force-pushed the SDK-963-enable-multiple-extension-registries-to-be-defined-for-dfx-extensions branch from 007d9df to e6f5e1b Compare July 20, 2023 19:09

Marcin Nowak-Liebiediew added 2 commits July 20, 2023 23:18

small improvement to the extension.bash script and changelog

66ab361

Merge branch 'master' into SDK-963-enable-multiple-extension-registri…

6243fb7

…es-to-be-defined-for-dfx-extensions

ericswanson-dfinity reviewed Jul 20, 2023

View reviewed changes

smallstepman mentioned this pull request Oct 25, 2023

feat(extension): add support for custom canister types #3222

Closed

6 tasks

ericswanson-dfinity closed this May 3, 2024

ericswanson-dfinity deleted the SDK-963-enable-multiple-extension-registries-to-be-defined-for-dfx-extensions branch May 6, 2024 21:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(extensions): install from external registries #2968

feat(extensions): install from external registries #2968

smallstepman commented Feb 13, 2023 •

edited

Loading

ericswanson-dfinity Mar 2, 2023

ericswanson-dfinity Jul 20, 2023

ericswanson-dfinity Jul 20, 2023

ericswanson-dfinity Jul 20, 2023

ericswanson-dfinity Jul 20, 2023

ericswanson-dfinity Jul 20, 2023

smallstepman Jul 20, 2023 •

edited

Loading

	`dfinity/dfx-extensions`	external registry
checksum	well-known location: `https://github.com/dfinity/dfx-extensions/releases/<GIT_TAG>/downloads/<ARCHIVE_FILENAME>.sha256`	supplied in registry file
tarball download url	well-known location: `https://github.com/dfinity/dfx-extensions/releases/<GIT_TAG>/downloads/<ARCHIVE_FILENAME>`	supplied in registry file
supported arch/platform	all platforms supported by dfx are supported by official extensions	determined by fetching value of `.extensions.<EXT>.<VER>.binaries.<ARCH-PLATFORM>` from registry file, if key is missing then platform is not supported
`extension.json`	unpacked from tarball	extracted from registry file

feat(extensions): install from external registries #2968

feat(extensions): install from external registries #2968

Conversation

smallstepman commented Feb 13, 2023 • edited Loading

Description

How Has This Been Tested?

Checklist:

ericswanson-dfinity Mar 2, 2023

Choose a reason for hiding this comment

ericswanson-dfinity Jul 20, 2023

Choose a reason for hiding this comment

ericswanson-dfinity Jul 20, 2023

Choose a reason for hiding this comment

ericswanson-dfinity Jul 20, 2023

Choose a reason for hiding this comment

ericswanson-dfinity Jul 20, 2023

Choose a reason for hiding this comment

ericswanson-dfinity Jul 20, 2023

Choose a reason for hiding this comment

smallstepman Jul 20, 2023 • edited Loading

Choose a reason for hiding this comment

smallstepman commented Feb 13, 2023 •

edited

Loading

smallstepman Jul 20, 2023 •

edited

Loading