-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider releasing linux assets with .tar.gz
instead of .zip
+ misc improvements
#2145
Comments
@polarathene , thanks for detailed, considered request. I'll definitely look into your suggestions to improve the publishing/distribution of qsv. I agree that
Just the same, as FYI, we primarily target the latest Ubuntu LTS on the x86_64 platform, which is glibc-based, as that is our standard deployment platform for our CKAN PaaS service. I don't get to exercise the musl build as much, so we depend on community feedback to improve it. |
Thanks for sharing those insights, very informative 😁
You could probably workaround that with an alternative host for releases? I know some other projects like Caddy have open-source plans with Cloudsmith (linked to related Github Action) for publishing releases either as packages or raw files/archives. EDIT: Ah, seems like The intent was Cloudsmith would be the place your versioned archives would be stored for self-update functionality, and GH releases would be more akin to other projects GH releases 😅
I've not inspected to check if it's decompressing the entire archive, but I really don't see that being a pragmatic concern for most? With
I think that's more convenient on your end than the users though? Most only need a single variant, but instead need to download the whole archive redundantly to get that. No bandwidth fees with Github on your end, so I can understand why that's fine. For users it also minimizes searching through the list of links to find one they're interested in, although with automation or self-update that's often a 1 time only benefit.
I assume only when it's pulling an update? Not the original archive? I didn't check that, but since I renamed the archive to make it simpler to extract
I understand the value of self updating for some users, but it's not something I want myself, it rarely is on linux (we have package managers for this purpose). Last thing I would want is to run some project that had some I am aware of some distros having packages (often by community) that may pull from a pre-built release from official channels rather than doing a local build on the client as opposed to more official package repos per distro where they're built from source. Likewise in my case with Docker, adding
You can accomplish the same with other formats, but I understand the choice with zip and self-updating features. My intention isn't to burden you with further maintenance, just to raise awareness of where some minor friction is for a different type of user. EDIT: FWIW
Static musl builds work on glibc systems (you usually can't have proper static glibc). For My comment that you responded to though was about leveraging Zig to compile both glibc and musl builds from the same build host with the added benefit of not requiring any qsv/.github/workflows/rust-musl.yml Lines 34 to 37 in 2652d76
Your glibc min version will be from the build host: qsv/.github/workflows/rust.yml Line 20 in 2652d76
Presently With Zig (via So there's potential with zig to simplify your workflows / CI. |
Thanks @polarathene for your detailed feedback. I'll take your recommendations under further consideration as we refine the publishing workflow. And thanks for pointing out that I should explicitly use As for qsv, my number one goal for the project is to be the fastest csv data-wrangling kit - thus the aggressive MSRV policy, taking advantage of the latest language features, the latest dependencies, etc. One big ticket item that I haven't taken on with a big performance payoff is profile guided optimization (#1448). Given the number of binaries/platforms we support, I can only do that for select platforms (starting with qsv pro). I'll leave it to package maintainers should they choose to distribute qsv to fine-tune it to their requirements. As for self-update, it's gated behind the And even if you choose to use the pre-builts in your Docker image, you can set the Finally, self-update is not automatic. You have to explicitly opt-in to update. |
Is your feature request related to a problem?
On linux the
.zip
archive format is not as desirable for publishing releases (GH release assets)..zip
:vs
.tar.gz
:Additional justification:
tar
and related codec support there (gzip
/xz
) than it is for zip support. This is a minor issue, but not too uncommon with container environments for example (and perhaps some CI?).unzip
is often the package/command used to handle the zip format on linux, but it does not support a stream from stdin, the zip file must be downloaded to disk first, then extracted, followed by optionally removing the redundant archive.funzip
which can handle stdin, but this is not compatible for zip archives with multiple files like the QSV releases have.Describe the solution you'd like
Publishing assets for linux with
.tar.gz
as most projects do on GH releases for this platform.Also consider:
latest
release URL to be used. More details here.lto = "thin"
, as the published assets are built with features dropped due to CI memory limits (lto = true
requiring over 10GB memory to build from source): Asset does not include all features #1102 (comment)Describe alternatives you've considered
No major issues, for manual downloads I can grab the latest URL from the release process with a few more clicks, then add a package in environments that need it. It would be nicer to not need to think about it though :)
I had also tried to leverage Cargo Binstall but got an error due to the
.zip
format not being as well supported.cargo binstall qsv
would automate pulling the latest release from your GH releases which works just as well for me if that functioned properly (note that you can also further improve your support for that utility by adding some metadata to yourCargo.toml
).At one point I did try building from source (this was problematic due to build requirements being ridiculous to try a CLI program out, over 10GB of memory was used).
Additional context
At this point changing the asset name (either by extension or complimentary version omission) is technically a "breaking" change that'd affect any automated processes (once they version bump at least). Technically since QSV is still on
0.x.y
, wheneverx
increases breaking changes are permitted, so that is your call 👍GoReleaser could help with the archive by platform difference if you were to pursue this change.
UPDATE: I see that your CI is reliant upon the release tag in the asset name, so that may be a blocker for dropping the version name:
qsv/.github/workflows/publish-linux-qsvpy-glibc-231-musl-123.yml
Lines 166 to 172 in 2652d76
BTW, you could probably simplify the build process a bit (especially for the lower glibc requirement and musl build support) by using Zig (see
cargo-zigbuild
).The text was updated successfully, but these errors were encountered: