Annotator instances shutting down because of inadequate disk

[gfr10598]

Monitoring shows (for some deployments) that instances are being killed because of too little disk space. Minimum seems to be around 1GB. Could this be caused by excessive logging?

resource.type="gae_app"
resource.labels.module_id="annotator"
logName="projects/mlab-sandbox/logs/appengine.googleapis.com%2Fvm.syslog"
"unhealthy"

Dec 19 22:17:01 aef-annotator-20181219t200546-3dwf vm_check_disk_space.sh: Free disk space (883 MB) is lower than threshold value 998 MB. Reporting instance permanently unhealthy. Note: different instances will have a different threshold value.

[gfr10598]

After cleaning up some spammy logs, the problem seems to be much improved. We should probably just be aware of this problem, and try to avoid spamming the logs too severely.

Alternatively, we could certainly allocate more disk, but this might just hide a problem that it would be better to avoid.

[gfr10598]