Failed to pull image "busybox"

[Vadim-Zenin]

After last fluentd-cloudwatch update I have this in logs.

`kubectl get events -A | grep -i "error|warning|failed" | grep fluentd-cloudwatch

Failed to pull image "busybox": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

amazon-cloudwatch 12m Warning Failed pod/fluentd-cloudwatch-h8q6w Error: ErrImagePull
amazon-cloudwatch 12m Warning Failed pod/fluentd-cloudwatch-h8q6w Error: ImagePullBackOff
amazon-cloudwatch 19m Warning Failed pod/fluentd-cloudwatch-qd8hh Failed to pull image "busybox": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit`

fluentd-cloudwatch could not start anymore.

Could you include variable to use either public AWS ECR repository or private AWS ECR repository?
If the variable was not set use by default busybox docker hub repository.
If the variable was set use repository provided in the variable.

[Bryksin]

Yes would be awesome if any ref to public images could be replaced by variable
if the variable was not set, fine, use a public image from Docker Hub, however, if the variable was set with a value containing ref to image from company/private repo (AWS ECR, Nexus, Artifactory...) use the value from the variable

[fanuch]

I have a plan I have yet to try.

High level

1. Find a public cache for busybox other than Docker
2. Replace InitContainer with new link

1. Find a public cache for busybox other than Docker

I found an official repository on ECR Public Gallery for busybox
https://gallery.ecr.aws/docker/library/busybox

I could set up a mirror that I would need to maintain, but this looks in line with the official busybox on Dockerhub

2. Replace InitContainer with new link

Kubernetes official documentation shows you can update the image of the daemonset in place and the default strategy is already RollingUpgrade:

kubectl set image \
    ds/fluentd-cloudwatch \
    busybox=public.ecr.aws/docker/library/busybox:latest \
    -n amazon-cloudwatch

But we don't want to update the image, we need to update the Image under the InitContainer.

This StackOverflow answer says that we can update the image under the InitContainer by referencing the InitContainer name as per the Kubernetes PR that implemented this change.

In my case there are two

• copy-fluentd-config
• update-log-driver

Which means my command will be:

kubectl set image \
    ds/fluentd-cloudwatch \
    copy-fluentd-config=public.ecr.aws/docker/library/busybox:latest \
    -n amazon-cloudwatch

# and

kubectl set image \
    ds/fluentd-cloudwatch \
    update-log-driver=public.ecr.aws/docker/library/busybox:latest \
    -n amazon-cloudwatch