-
Notifications
You must be signed in to change notification settings - Fork 270
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MountTmpfsAtTemp
doesn't seem to always have the desired effect
#1326
Comments
@DrJosh9000 Yes I've tested that and it looks like it works! Thanks for the quick fix :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
MountTmpfsAtTemp=false
doesn't always seem to take effect. A large proportion of instances come up withtmpfs
still mounted to/tmp
, and then get terminated.This causes a lot of instance churn on scale-up.
I suspect there is something else running on startup that writes to the
/tmp
dir at around the same time thatbk-mount-instance-storage.sh
is run, blocking the unmount operation and causing it to fail. Is there anything in the AMI that might do that?note: it's entirely possible that there's something in our custom AMI that we build atop of the buildkite-provided one that is responsible for this. If you can't reproduce this, that is still good info that we need to audit our startup processes
Steps To Reproduce
Steps to reproduce the behavior:
v6.21.0
MountTmpfsAtTemp
to falseInService
state, but get terminated after ~1 minute.Expected behavior
when
MountTmpfsAtTemp
is set to false, the instance runssystemctl mask --now tmp.mount
on startup, which correctly unmounts tmpfs from the/tmp
directory. We observe this in a small number of instances that come up:Actual behaviour
On a large number of instances, instead we see the following:
Stack parameters:
ami-04ca34320055d861c
The text was updated successfully, but these errors were encountered: