Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert using Docker Engine's squash functionality #113

Open
ronaldtse opened this issue Aug 27, 2021 · 20 comments
Open

Revert using Docker Engine's squash functionality #113

ronaldtse opened this issue Aug 27, 2021 · 20 comments

Comments

@ronaldtse
Copy link
Contributor

We only want to squash up to a certain layer to facilitate re-use of the base images, not squash all the way to the top.

@phuonghuynh
Copy link

@ronaldtse from the document, --squash is always create new layer by squashing all layers in the image. There is no option to select which layers need to squash for now,

https://docs.docker.com/engine/reference/commandline/build/#squash-an-images-layers---squash-experimental

Could we consider to use multi-stage build instead ?

@ronaldtse
Copy link
Contributor Author

@phuonghuynh yes we should use a multi-stage build. We want to base our image "off" the existing base ubuntu container, i.e. we only want to squash the layers we build on top of the base Ubuntu container.

It was previously doable with our docker-squash image (https://github.com/riboseinc/docker-squash-container) but this may no longer be current.

@phuonghuynh
Copy link

@ronaldtse we already use it, https://github.com/metanorma/metanorma-docker/blob/master/Dockerfile.ubuntu.in#L4

So I think we will add new make command to let user select with or without --squash option

@ronaldtse
Copy link
Contributor Author

@phuonghuynh the reason I want to have 2 stages squash is so that people who already have ubuntu do not need to download the full container. Can we do this?

@phuonghuynh
Copy link

Let me check

@phuonghuynh
Copy link

@phuonghuynh the reason I want to have 2 stages squash is so that people who already have ubuntu do not need to download the full container. Can we do this?

So people who already pull Ubuntu stage do not need to pull it again, docker client will only need to pull the cli stage?

@ronaldtse
Copy link
Contributor Author

@phuonghuynh correct!

@ronaldtse
Copy link
Contributor Author

Ping @phuonghuynh

@phuonghuynh
Copy link

Yes, I am on it

@CAMOBAP
Copy link
Contributor

CAMOBAP commented Sep 8, 2022

@ronaldtse as far as I understand this isn't valid anymore, because we don't squash images, right?

@ronaldtse
Copy link
Contributor Author

The best practice approach now is to use multi-stage builds to just copy the necessary files. However this approach requires detailed testing.

Perhaps we should try another base image like distroless?

https://symflower.com/en/company/blog/2022/complete-guide-on-shrinking-container-images/

@ronaldtse
Copy link
Contributor Author

Actually I finished reading the article and it does not provide much info that we don't already know. I think if we have tebako, then there is a "pre-expanded" mode, that would be the best.

Ping @maxirmx

@maxirmx
Copy link
Contributor

maxirmx commented Sep 9, 2022

This topic confuses me a lot: tamatebako/tebako#74 (comment)
We can have pre-expanded mode, of course. But in this case we do not need DwarFS and all the pain it creates

@ronaldtse
Copy link
Contributor Author

I'm not sure if I'm suggesting the right thing.

The issue here is that we have two ways of building a "package", as we have two deliverables: a tebako executable and a docker image.

The way we build the docker image seems to contain some files we don't need.

The way we build the tebako image seems more streamlined.

There are pros and cons of both, but in any case we need both deliverables.

If it is possible to streamline the build method of tebako to also generate a docker image, then it would simplify a lot our maintenance burden.

@ronaldtse
Copy link
Contributor Author

Simply put, the base case is to just add the tebako executable in the docker image. Then we only build once and can calls create the docker image.

But since the docker environment has less constraints (it has a r/w FS, it is allowed to have a much larger size), we could try to optimize the space/execution time requirements for the docker environment.

@maxirmx
Copy link
Contributor

maxirmx commented Sep 9, 2022

I would suggest then that we

  • build ruby with statically linked libraries and install it to prisitine environment
  • deploy metanorma in vendor mode
  • copy this ruby instance with environment to docker

there will be single layer

@ronaldtse
Copy link
Contributor Author

@maxirmx look forward to when we can do this... would you be able to help?

@maxirmx
Copy link
Contributor

maxirmx commented Oct 12, 2022

I believe that adding tebako executable to docker image will affect size ot speed.
As shown below tebako will provide exactly the same bundle

Img-1

@maxirmx
Copy link
Contributor

maxirmx commented Oct 12, 2022

Another big layer installs jvm, python and inskape.
Img-2

@maxirmx
Copy link
Contributor

maxirmx commented Oct 12, 2022

Please note also that image efficiency is estimated at 98%.
So squashing the layers may potentially save 27Mb out of 1.5Gb. Download time will increase because of smaller concurrency factor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants