specific nightly tags

[clux]

Hi there.

Is there any chance of having specifically tagged images rather than just a floating nightly tag? It's always too scary for me to rely on a docker image with a floating tag anywhere.

Having tags such as nightly-2018-02-23 would be very helpful!

This is the system used on muslrust and associated travis logic as an example.

[sfackler]

We don't currently do this on the assumption that the Docker Hub will not be happy with us creating a new ~250MB image every day. If there's no problem there then it seems reasonable.

[clux]

I'd assume they would've said something about it to all the other languages that does this if it was a problem. For instance, node has over 1200 tags there (src repo), multiple ones pushed daily. That's also really useful, if you needed a specific variant / flavour. For a language like rust, it'd be nice if good images seemed official and didn't come from random people like me :-)

From what I can tell their "policy" is no limits, which makes sense in some ways. Downloads are going to be more costly than uploads to docker hub, and people generally aren't going to download more than the one/two they need even if more are available. Storage wise, who knows how they deal with it. My repo alone pushes >100GB a year. Maybe they'll implement a fall-off.

Anyway, it's still way better if this was done on an official account than duplicated around. There's nothing stopping people from forking this and pushing daily (for free, like I do), but that might be considered a rather selfish use of resources. Still, that's what happens when there's no official ones around that we can rely upon. Just look at how many different musl rust builders exist.

[sfackler]

Where do 1200 tags come from for node? I'm not seeing that many, and of the ones that are there, many refer to the same image (e.g. wheezy, 9-wheezy, 9.6-wheezy, 9.6.1-wheezy). I also don't see which of those would be pushed daily. Official images are supposed to be reproducible, and unless the Node folks are opening a new PR every day to docker-library/official-images, any rebuilds would be triggered for other reasons, like changes to the base image. It looks like their release cadence is ~weekly https://github.com/docker-library/official-images/pulls?q=is%3Apr+label%3Alibrary%2Fnode+is%3Aclosed.

It does seem like we wouldn't be violating Hub policy though!

[clux]

Maybe their cycle is not daily, but there's around 20 new tags pushed in the last 24 hours (though as you say, some duplication for convenience). You can see all their 1214 currently available tags via the API:

curl -sSL https://registry.hub.docker.com/v1/repositories/node/tags | jq ".[].name" -r 

[BlinkyStitt]

I would also like to see tags for previous nightly releases. For now I copied the RUN lines from this Dockerfile into my application's Dockerfile except I set RUST_VERSION to "nightly-2018-10-31".

That works for now, but it got me thinking; could we make an image that uses ONBUILD to fetch a specific version? I think that would work for the way I'm currently developing.

[tyranron]

Same here. It would be nice to rollback to some specific day-version in case regression happens (like #12).

[tyranron]

@sfackler would the contribution be appreciated? If yes, do you have any thoughts/ideas/plans about organizing this already?

[zroug]

I would be surprised if Docker Hub doesn't use some form of deduplication to store the images and the difference between two versions of an image is probably not very big. Internally they do already keep the images for some amount of time. For example a few weeks ago I used rustlang/rust@sha256:0b7f81da7fd40fb78d824cacc2a710fde544a6f0799d59365b16867e78b07af8.

The image is still available today, so adding tags would only make it explicit how long you can pull an image.

docker pull rustlang/rust@sha256:0b7f81da7fd40fb78d824cacc2a710fde544a6f0799d59365b16867e78b07af8

[jhpratt]

Still no plans to provide this? I was just looking into converting a web server to Docker, but certainly won't be able to without the ability to select a specific nightly. Sure, I can ensure it works with the server, but that's hardly possible with CI involved. I need reproducibility.

[tyranron]

For those who needs this there is a mirror repo https://hub.docker.com/r/instrumentisto/rust

With a bit of hook/post_push magic it mirrors images of this repo and preserves every nightly version as nightly-2019-08-12.

[jorgecarleitao]

Please reconsider this, even if one tag at the end of the month or something.

For projects that rely on the nightly builds, such as anything using rocket, a CI pipeline is unable to cache the docker image and re-use it across builds because FROM rust-nightly changes, invalidating all future layers. This leads to a significant amount of compilation time.

I have been using

# the image that we use to build the backend
# Rocket requires nightly build
FROM rustlang/rust:nightly-buster AS backend-builder

# Compile a dummy program, so that the dependencies are compiled and cached on a layer
# see https://stackoverflow.com/a/58474618/931303 for details
RUN echo "fn main() {}" > dummy.rs
## if cargo.lock or cargo changes, we trigger a build of all dependencies.
COPY backend/Cargo.lock backend/Cargo.toml /
RUN sed -i 's@src/[email protected]@' Cargo.toml
RUN cargo build --release
RUN sed -i '[email protected]@src/main.rs@' Cargo.toml

# compile the actual program
COPY backend/src/ src/

RUN cargo build --release

FROM debian:buster-slim

COPY --from=backend-builder target/release/... .

to avoid re-building dependencies and reduce image size, but this code is pretty much useless with a moving tag from rustlang/rust:nightly-buster.

[tyranron]

@jorgecarleitao consider instrumentisto/rust image, it has all the nightly versions preserved, and it's nothing but just a direct retaging of these official images.

We're using them for quite a long time without any problems.