[proposal] Improve version and distribution management of the CLI

[andreaTP]

Hey all 👋 !

I think we touched on this point earlier, the root problem here is that we would like to have a configurable version of the kiota CLI depending on the project we are working on and not rely only on a global installation.
I have noticed that most of the issues reported by colleagues have been caused by the incorrect version of kiota installed/used 😢

As a first step to improve the situation I want to propose leveraging asdf already used by other big projects like Crystal.

Here I prototyped a basic and initial version of an asdf plugin for Kiota: https://github.com/andreaTP/asdf-kiota (tested only on linux and mac).

In a nutshell, the workflow would be something like this:

• globally install the asdf-kiota plugin: asdf plugin add kiota https://github.com/andreaTP/asdf-kiota.git
• install a kiota version: asdf install kiota latest
• set a global version of kiota: asdf global kiota latest
• at a project level you can still use a different version by dropping a file .tool-versions with a content similar to e.g. asdf 1.1.2 (or using asdf local ...)
• you can easily switch Kiota versions even in the same terminal session asdf shell kiota 1.1.2

Now the questions 🙂 :

• what do you think about this approach?
• can we assume that we would support this effort by stabilizing the released zip structure?
• would you like to take over? alternatively, what do you think about moving the repo to: https://github.com/asdf-community/ ?

Let me know your thoughts!

[baywet]

Thanks for the detailed information here.
We've been talking about how would people be able to manage multiple versions of kiota locally without having to go through an installation every time. (a little like nvm or pyenv for example)

Let's start by acknowledging a couple of hypothesis:

• The need to switch between versions should decrease over time as languages mature to stable.
• We don't want to write code for this inside the generator itself if we can avoid it.
• The need to switch between versions would most likely increase when we release a v2.
• We already have some ability to do a version switch through the docker image today, savvy people might use the binaries and symlinks/environment variables.

Most packaging and distribution mechanisms only offer the ability to install one version of the same major at a time:

• chocolatey
• winget (there are ways around that, but people would have to handle the switch manually)
• deb/rpm/...

A couple of of packaging and distribution mechanisms offer this ability

• snap
• brew
• asdf

With all that context in mind, one of the challenges for us (Microsoft) is that we don't want to stretch ourselves too fin with "distributions", so we have to be careful with what we commit to as a first party team.
We've talked about adding support for brew and winget and maybe snap in the past without making commitments to see if we get customer demand for it.

Back to asdf, there are probably two models which could work here: community maintained "package" or a Microsoft maintained package. I'll need to bring this back to my team. The fact that asdf doesn't support windows would be a major contention point for us in terms of prioritization. Additionally, I'd like to make sure we have a clear understanding of the different ecosystems before we make any commitments. Are there any other packaging and distribution mechanisms that I might have forgotten about at this point?

[andreaTP]

Thanks for the answer @baywet !
Let me go through it.

Let's start by acknowledging a couple of hypothesis:

• The need to switch between versions should decrease over time as languages mature to stable.

I'm not 100% sold on this point as it greatly depends on the maturity of the specific language and it greatly impacts the stability of the produced output when you are, i.e. shipping it as a library. This goes hand in hand with:

• The need to switch between versions would most likely increase when we release a v2.

I strongly agree with this.

• We don't want to write code for this inside the generator itself if we can avoid it.

This is an option I am still considering, especially given the fact that kiota.lock already contains the version of the tool used. A possibility that can be explored, if desired, is to tweak the main entrypoint of the cli to implement a logic as such:

• the version used in the lock file is equal/compatible with the current -> proceed with the standard command
• if the version is different download the expected version and invoke the original command on the just downloaded binary

this will look very much like this approach.

• We already have some ability to do a version switch through the docker image today, savvy people might use the binaries and symlinks/environment variables.

True, but container images are usually a pretty hard requirement, especially in CI/CD pipelines, and because of the overhead.

Are there any other packaging and distribution mechanisms that I might have forgotten about at this point?

Just a few notable mentions for completeness:

• sdkman is pretty popular in Java land
• Scoop only for windows

Let me also add a note on JReleaser which makes it easy to release for various distribution mechanisms. The tool is great and used in various projects (even outside java land) and the author is usually eager to help upstream projects adopt it.

On the subject in general, I don't have a super strong opinion and I'm happy to second whatever guidance comes from you on this aspect to be able to ditch those scripts I'm using around 🙂 .

Regarding asdf in particular feel free to take your time and discuss it internally, I opened this issue to check your interest but I can totally live with a community-maintained "plugin" if there are no objections on your side.

[baywet]

Thanks for the additional information.

if the version is different download the expected version and invoke the original command on the just downloaded binary

That's an interesting take on the problem. We took a different stance on this one "if the version is not the same, it most likely means the user wants to update their client". And kiota just upgrades the client (dependencies still need to be updated).

Updating (or installing) another version in place is a tricky endeavor because we'd effectively need to "understand" how kiota was installed at the first place to avoid messing up the environment. It also supposes our CLI structure will stay the same across versions which is a bigger commitment than we can make at this time.

On the subject in general, I don't have a super strong opinion and I'm happy to second whatever guidance comes from you on this aspect to be able to ditch those scripts I'm using around

Is there any specific reason why you're still on 0.11.1 which is a rather "old" version at this point?

I can totally live with a community-maintained "plugin" if there are no objections on your side.

We want the community around kiota to thrive and try to support it the best we can, whether that means integrating "new features" in the main code base or supporting projects federated around it. Blocking any initiative is not at all our intent aside from malicious/unethical activities or things that'd drastically change the vision of the project itself.

As a side note, on the first party distributions mechanisms, we're also trying to keep the distribution for kiota, hidi and the microsoft graph CLI aligned as they are fundamentally "CLIs" and this decreases our engineering efforts.

[andreaTP]

Thanks for the additional context!

That's an interesting take on the problem. We took a different stance on this one "if the version is not the same, it most likely means the user wants to update their client". And kiota just upgrades the client (dependencies still need to be updated).

I just want to rise the flag that this approach is highly subject to causing accidental breaking changes, and makes reproducible builds a bit more tricky for the users.

Updating (or installing) another version in place is a tricky endeavor because we'd effectively need to "understand" how kiota was installed at the first place to avoid messing up the environment.

I don't think this is a tricky endeavor as there are a few "well-known" approach to it (i.e. using a ${HOME}/.kiota/<version>/) that are used by many tools.

It also supposes our CLI structure will stay the same across versions which is a bigger commitment than we can make at this time.

As of today we only need the kiota executable(given the fallbacks done with Source generators), which is great IMHO.
It would be awesome to keep this minimal requirement ...

Is there any specific reason why you're still on 0.11.1 which is a rather "old" version at this point?

just an old gist 🙂 , in most cases I'm on main but I also released the kiota-maven-plugin for 1.1.2 already.

Re. asdf I'll wait for a nudge from your side 👍

As a side note, on the first party distributions mechanisms, we're also trying to keep the distribution for kiota, hidi and the microsoft graph CLI aligned as they are fundamentally "CLIs" and this decreases our engineering efforts.

Do you have a short list of additional requirements for those applications? As I'm not really familiar with them ...

[baywet]

I don't think this is a tricky endeavor as there are a few "well-known" approach to it (i.e. using a ${HOME}/.kiota//) that are used by many tools.

Right but we don't control all those. dotnet tool installs under ~\.dotnet\tools\..., vs code under ~\.vscode\extensions\..., choco C:\chocholatey\... (unless the user has configured something else, which means we need to read the chocolatey's configuration)....

Do you have a short list of additional requirements for those applications? As I'm not really familiar with them

Additional requirements in what sense? They are all built with dotnet. And distributed as: GitHub release assets, dotnet tools, docker images like kiota today. Kiota is the only one with a vs code extension, and we don't foresee shipping the other two as an extension anytime soon.

[andreaTP]

Don't wanna be pedantic here but the "global" installation of kiota doesn't need to be the same as a local/derived one.
I don't think we need to understand how kiota was installed in the first place, we can just decide to use a specific download/execution logic.

Additional requirements in what sense?

Do they need any OS dependency? Additional files to be around on the FS, env vars etc. etc.

[baywet]

Sure we could decide that, that then means that if people uninstall kiota through their distribution mechanism, they might be left with a backlog of versions at that place because "we won't know it got uninstalled". We could augment some of those uninstalls with post cleanup scripts, but not all of those distribution mechanisms support it. Probably a tradeoff we can live with though.

The requirements are pretty much the same as kiota itself, they are built the same way. There is a configuration file that might need a source generator like you did for kiota. The CLI also has a token cache, but that's already under the home directory.

[aalmiray]

👋 from JReleaser. Happy to help with any questions you may have regarding the tool should you decide to consider it.

[baywet]

@darrelmiller will come and comment with the outcome of our discussion. In the meantime I thought I'd let you know we logged that #2598

[andreaTP]

Polite ping on this @darrelmiller 🙏
If there is no news I'll try to move the asdf-kiota plugin to the asdf-community org and let you decide on the "official" distribution mechanisms at your own pace.

[andreaTP]

Quick update, I moved the asdf-kiota plugin to the asdf-community organization and added asdf as an option in the installation mechanisms in the docs.

[baywet]

Thanks for the continuous work on this. It might be worth also putting a quick PR together to add an entry to the changelog for the upcoming release.

[andreaTP]

@baywet thanks a lot for your continuous help and support.
In this case, despite the fact that we got something together for ASDF, I do believe that the original question is still valid and we need to:

• choose the "official" distribution mechanisms
  -implement them

Or, do you have another issue(maybe internal) for tracking progress on this?

[baywet]

Thanks for your patience on the matter.
We might get winget for free since the Microsoft Graph CLI will be working on this in the near future and it should simply be some copy/pasta on the pipeline definitions.
Which would probably rule out chocolatey for windows.

For linux, asdf is a great way forward, and we're going to pause packaging efforts for the time being (snap/deb/rpm) to see what requests we get from the community.

For MacOS, same rationale applies so we're not going to invest in brew right now.

This is why we don't have any outstanding issue, we'd like the community to voice their demands.

I hope that makes sense, let us know if we missed something.

[andreaTP]

The ASDF integration is, by far, sub-optimal.
I have implemented it to prevent a show stop here and to show one of the possibilities but has a number of issues:

• doesn't work on win
• is not widespread
• it's community maintained
• it's subject to GH rate limiting

I think that having a standard and supported distribution mechanism(per platform) would greatly improve the getting started experience and increase the number of users of Kiota.
Currently, there is no "good way", nor standard to enforce a Kiota version.
This is, likely, going to cause quite a bunch of issues in the mid-term as we experienced in RH, going through the adoption phase.

[baywet]

And I'm guessing that most of your context is linux based?

[andreaTP]

Linux and Macs around ... yes

[baywet]

from your perspective, do you have a lot of usage of homebrew on linux?

[andreaTP]

No, ppl are using yum, dnf and apt on Linux from my personal experience

[baywet]

Microsoft hosts this package feed that supports multiple formats (deb/rpm), package managers and distributions. And I've found the onboarding documentation (you won't be able to access that). The security requirements are similar to other publishing workflows. We'd need to ramp up on how to make deb/rpm packages, decide which distributions we want to support etc... It's still a significant undertaking on that side. But it should be simpler than trying to publish to each distribution's official feed (archive.ubuntu.com, etc...)

Couple of questions for you:

• how often do people add new package feeds to their OS?
• have you seen anyone who's not a dotnet developer but want kiota installing dotnet framework and then the kiota dotnet tool?

[andreaTP]

Answering from the camping, so I might be less accurate...

how often do people add new package feeds to their OS?

Rarely in my experience, and is a little awkward having to add a new feed.
Still, if it can be used for more packages, it might be ok

have you seen anyone who's not a dotnet developer but want kiota installing dotnet framework and then the kiota dotnet tool?

I have never seen a non-dotnet developer installing tools through the dotnet cli/framework.
I was not even aware of the fact that it was possible and built in (looks similar to Go /bin installations).