Skip to content

Commit

Permalink
Reduces terminationGracePeriodSeconds from 180s to 100s (#894)
Browse files Browse the repository at this point in the history
The 180s termination grace period was to account for the very long time it
could take for mlab-ns to notice that pod was shutting down. With Locate and
Heartbeat, this process only takes 10s. Nothing changes about the configuration
here, except removing unnecessary wait time. Tests still have 30s to complete,
and pusher still has 60s to push data to GCS as fast as it can.

Resolves m-lab/ops-tracker#1742
  • Loading branch information
nkinkade authored Aug 7, 2024
1 parent 9963d57 commit 3bce113
Showing 1 changed file with 5 additions and 6 deletions.
11 changes: 5 additions & 6 deletions k8s/daemonsets/templates.jsonnet
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,11 @@ local PROJECT_ID = std.extVar('PROJECT_ID');
//
// * k8s sends SIGTERM to container
// * container enables lame duck status
// * monitoring reads lame duck status (60s max)
// * mlab-ns updates server status (60s max)
// * heartbeat notices lame duck status and notifies Locate (10s max)
// * all currently running tests complete. (30s max)
// * give everything an additional 30s to be safe
// * 60s + 60s + 30s + 30s = 180s grace period
local terminationGracePeriodSeconds = 180;
// * give Pusher an additional 60s to upload all data
// * 10s + 30s + 60s = 100s grace period
local terminationGracePeriodSeconds = 100;

local uuid = {
initContainer: {
Expand Down Expand Up @@ -845,7 +844,7 @@ local Experiment(name, index, bucket, anonMode, datatypes=[], datatypesAutoloade
dnsConfig: {
nameservers: ['8.8.8.8', '8.8.4.4'],
},
// Only enable extended grace period where production traffic is possible.
// Apply extended grace period, except for mlab-sandbox
[if std.extVar('PROJECT_ID') != 'mlab-sandbox' then 'terminationGracePeriodSeconds']: terminationGracePeriodSeconds,
},
},
Expand Down

0 comments on commit 3bce113

Please sign in to comment.