Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ci): disable Docker build cache mode=max option due to disk space problem #5005

Closed
wants to merge 4 commits into from

Conversation

youtalk
Copy link
Member

@youtalk youtalk commented Jul 18, 2024

Description

Unfortunately, even with the above two corrections, it was found that the cuda job still fails due to disk space issues.

https://github.com/autowarefoundation/autoware/actions/runs/9981995778/job/27586843983#step:5:18544

#145 114.8 fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "mkdir: cannot create directory ‘/root/.ansible/tmp/ansible-tmp-1721262994.3001075-4771-138480599394249’: No space left on device\n", "unreachable": true}

Therefore, this PR temporarily removes the mode=max option from the cache-to setting of docker-build-and-push workflows to reduce the amount of cache export.

Ref. https://docs.docker.com/build/cache/backends/#cache-mode

Tests performed

https://github.com/autowarefoundation/autoware/actions/runs/9983663011

Effects on system behavior

Not applicable.

Interface changes

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

  • There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

youtalk added 3 commits July 17, 2024 21:18
Signed-off-by: Yutaka Kondo <[email protected]>
Signed-off-by: Yutaka Kondo <[email protected]>
Signed-off-by: Yutaka Kondo <[email protected]>
@youtalk youtalk self-assigned this Jul 18, 2024
@youtalk youtalk added type:ci Continuous Integration (CI) processes and testing. type:containers Docker containers, containerization of components, or container orchestration. labels Jul 18, 2024
@oguzkaganozt
Copy link
Contributor

Switching to a different build farm other than default github runners might be better workaround to get the most out of the docker caching. @mitsudome-r

@youtalk youtalk closed this Jul 18, 2024
@youtalk youtalk deleted the disable-mode-max branch July 18, 2024 12:45
@youtalk youtalk restored the disable-mode-max branch July 18, 2024 12:45
@youtalk youtalk reopened this Jul 18, 2024
@xmfcx xmfcx added the tag:run-health-check Run health-check label Jul 18, 2024
@xmfcx
Copy link
Contributor

xmfcx commented Jul 18, 2024

Switching to a different build farm other than default github runners might be better workaround to get the most out of the docker caching. @mitsudome-r

@oguzkaganozt GitHub public runner are an invaluable resource for us, they provide us with 20 machines with 4vCPU with reasonable specs.

I realized that we had the artifacts within the images just today. Autoware CI doesn't even make use of them. Once we remove them from the current image and split them into something else, space won't be an issue for a long time.

See:

@youtalk youtalk closed this Jul 23, 2024
@youtalk youtalk deleted the disable-mode-max branch July 23, 2024 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tag:run-health-check Run health-check type:ci Continuous Integration (CI) processes and testing. type:containers Docker containers, containerization of components, or container orchestration.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants