Invoker doesn't recognize pods stuck in "Pending" state #7
Description
We had an issue in our deployment where the infrastructure ran out of resources. As a result, pods could not be created and all new pods were stuck in the "pending" state. The invoker log showed the following error:
[2018-03-14T09:21:45.694Z] [ERROR] [#sid_121] [KubernetesClient] Failed create pod for 'wskinvoker-00-54-whisksystem-invokerhealthtestaction0': class io.fabric8.kubernetes.client.KubernetesClientTimeoutException - Timed out waiting for [60000] milliseconds for [Pod] with name:[wskinvoker-00-54-whisksystem-invokerhealthtestaction0] in namespace [dh-stage-ow].
There was never an attempt for the invoker to kill the pending pod. This resulted in hundreds of pods left in the "pending" state and required me to manually clean them up.
If a pod times out during creation, the invoker should attempt to kill that pod to avoid leaving orphaned pods.