Stuck in setup #65
Assignees
Labels
bug
Something isn't working
Comments
|
Hi thank you for the issue 👍 This happens when the job resource is modified from other operators or in some way. That's mostly true if you see the diff hash log/event. If the job resource is not the same as it was before, we delete the job and recreate it. Because jobs can't be patched and so we delete and recreate it. The first issue happens when the setup job is found but the name of the setup container inside the job is not shopware-setup. You can't modify the name, because it's hard coded in the code so I guess the issue is a timing issue with the api and the operator. We can fix this in the operator by ignoring fields for the hash generation. So if you can tell me what get's attached after the job is created, I can fix it. Annotations should be fine, but labels for example would trigger a recreate of the job. |
|
Thanks for your reply. I got the definitions of the resources. I cannot see that any other operator would modify the job. It is deleted almost instantly after creation (aprox. 4-5 sec). The created Pod is also being terminated almost instantly as the Job is deleted. apiVersion: batch/v1
kind: Job
metadata:
annotations:
batch.kubernetes.io/job-tracking: ""
shopware.com/last-config-hash: 24b308ee1d48a04d07293a69ed85e4f4
creationTimestamp: "2024-11-27T18:29:00Z"
deletionGracePeriodSeconds: 0
deletionTimestamp: "2024-11-27T18:29:00Z"
finalizers:
- foregroundDeletion
generation: 2
labels:
store: my-shop
type: setup
name: my-shop-setup
namespace: my-shop
ownerReferences:
- apiVersion: shop.shopware.com/v1
blockOwnerDeletion: true
controller: true
kind: Store
name: my-shop
uid: 0cd29409-5b25-4e5e-ac71-89e86a48dc7b
resourceVersion: "513749357"
uid: 87de84cf-7c20-42d9-b5b1-c20202fb4cf4
spec:
backoffLimit: 6
completionMode: NonIndexed
completions: 1
parallelism: 1
selector:
matchLabels:
controller-uid: 87de84cf-7c20-42d9-b5b1-c20202fb4cf4
suspend: false
template:
metadata:
creationTimestamp: null
labels:
controller-uid: 87de84cf-7c20-42d9-b5b1-c20202fb4cf4
job-name: my-shop-setup
store: my-shop
type: setup
spec:
containers:
...apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2024-11-27T18:33:51Z"
deletionGracePeriodSeconds: 30
deletionTimestamp: "2024-11-27T18:34:21Z"
generateName: my-shop-setup-
labels:
controller-uid: 2c455c57-04c6-4657-8d3a-ef259f660179
job-name: my-shop-setup
store: my-shop
type: setup
name: my-shop-setup-hd8rl
namespace: my-shop
ownerReferences:
- apiVersion: batch/v1
blockOwnerDeletion: true
controller: true
kind: Job
name: my-shop-setup
uid: 2c455c57-04c6-4657-8d3a-ef259f660179
resourceVersion: "513751839"
uid: 22f56eb5-a237-4119-8f3a-094767a66f45
spec:
containers:
- args:
- /setup
command:
- sh
- -c
env:
...
image: ghcr.io/shopware/shopware-kubernetes:latest
imagePullPolicy: IfNotPresent
name: shopware-setup
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/caddy
name: my-shop-caddy-config
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-nd4gj
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: ...
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
shareProcessNamespace: true
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: my-shop-caddy-config
name: my-shop-caddy-config
- name: kube-api-access-nd4gj
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-11-27T18:33:51Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-11-27T18:33:51Z"
message: 'containers with unready status: [shopware-setup]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-11-27T18:33:51Z"
message: 'containers with unready status: [shopware-setup]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-11-27T18:33:51Z"
status: "True"
type: PodScheduled
containerStatuses:
- image: ghcr.io/shopware/shopware-kubernetes:latest
imageID: ""
lastState: {}
name: shopware-setup
ready: false
restartCount: 0
started: false
state:
waiting:
reason: ContainerCreating
hostIP: ...
phase: Pending
qosClass: BestEffort
startTime: "2024-11-27T18:33:51Z" |
|
Updated shopware operator logs in the issue description |
|
The operator is changing the setup job because the resource is modified in some way. I can't see anything directly what cause this problem, but I will focus on the sub issue to make it visible. Sorry for that. |
|
Just for clearification, a local cluster without the operator? Which chart are you referring to? I'm not able to find one where the operator isn't used. Thanks for your help in advance :) |
|
Hi, we provider a helm-chart which you can use for testing. Can you maybe give us some more information about the current environment you are running the workload in? |
|
Exactly this helm chart is used to deploy the shop (incl. operator) and it was setup according to docs (see values in issue description). Environment is a k3s, self hosted. Furthermore I want to use the s3 backend provided by rook-ceph, which is used as storage backend, as I don't want to deploy multiple services providing a s3 (e. g. minio) |
|
Thank you, I will try to recreate the issue in k3s. |
|
Hi I am still on trying to recreate the issue. Are the secrets that are used for the store generated by other operators or did you create them manually? |
|
So I did manage to let the k3s setup run on my local machine and unfortunately it worked like expected. I used minio in this example but set the credentials by a test values yaml file: After some debugging and testing my first assumption was wrong, the job resource doesn't get changed because that is also not possible for most of the values. If you change for example some env variables for the store resource, the hash is different and that's when the job gets deleted and then created. Why this is happening I can only guess. Important would be that we are talking of the same helm-chart and the same operator. Operator image of the pod One Idea to see if something is changing the resource you can watch the resources like this: Make sure to scale the shopware operator deployment down to 0, because the operator is changing the status of the resource from time to time. And if you run this command: you should see different hashes each time the job get's created. This is not tested because it's working locally for me but should confirm my suspicions. |
|
Thanks for your support. I can provide the following answers:
The secret for the store is created manually.
I can confirm that the same image is being used.
When the operator is not running, there are no changes to the store resource.
The hash of the job is the same while it's being recreated multiple times: |
|
Ok for me it's a bit strange, because the calculation of the hash looks good. So the crd is not getting changed at all, which is good. We pushed a new image which logs the reason why the object will be patched by the operator: If you change the deployment of the operator to this image you should get more logs like this one: On thing to make sure that the operator is deleting the job would be, that you remove the permission for the deletion of the operator. Then we should also see an error if the operator is trying to delete it. The object |
|
I collected the following logs with the new image. |
|
Ok found it, the This is why it gets deleted and recreated. The Annotation |
|
Can confirm that it works with the added annotation. (The annotation has to be added to the store crd directly, its currently missing in the helm chart template.) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Steps to reproduce
Deployment of a Shop through shopware helm chart
What should happened?
The setup job should pass
What actually happened?
The setup job is instantly deleted (and the pod killed) after creation, leaving the operator stuck in the setup stage. Repeating every 10 seconds
Relevant log output
(Namespace, deployment name and URLs are changed)
2024-11-27 18:41:05 [INFO] setup starting manager 2024-11-27 18:41:05 [INFO] controller-runtime.metrics Starting metrics server 2024-11-27 18:41:05 [INFO] starting server {"kind": "health probe", "addr": ":8081"} 2024-11-27 18:41:05 [INFO] controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false} I1127 18:41:05.986837 1 leaderelection.go:250] attempting to acquire leader lease my-shop/d79142e5.shopware.com... I1127 18:41:21.032202 1 leaderelection.go:260] successfully acquired lease my-shop/d79142e5.shopware.com 2024-11-27 18:41:21 [DEBUG] events shopware-operator-5b8dfc5f8f-ctmz8_b033fa7b-3577-4bce-9362-4281a886aaaf became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"my-shop","name":"d79142e5.shopware.com","uid":"6c33bfe3-8dca-4893-897d-88b1e00ec547","apiVersion":"coordination.k8s.io/v1","resourceVersion":"513755298"}, "reason": "LeaderElection"} 2024-11-27 18:41:21 [INFO] Starting EventSource {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "source": "kind source: *v1.Store"} 2024-11-27 18:41:21 [INFO] Starting EventSource {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "source": "kind source: *v1.Secret"} 2024-11-27 18:41:21 [INFO] Starting EventSource {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "source": "kind source: *v1.Deployment"} 2024-11-27 18:41:21 [INFO] Starting EventSource {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "source": "kind source: *v1.Job"} 2024-11-27 18:41:21 [INFO] Starting EventSource {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "source": "kind source: *v1.Secret"} 2024-11-27 18:41:21 [INFO] Starting Controller {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store"} 2024-11-27 18:41:21 [INFO] Starting workers {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "worker count": 1} 2024-11-27 18:41:21 [INFO] Do reconcile on store because db secret has changed {"Store": "my-shop"} 2024-11-27 18:41:21 [INFO] my-shop Do reconcile on store {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276", "state": "setup"} 2024-11-27 18:41:21 [INFO] my-shop reconcile app secrets {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276", "state": "setup"} 2024-11-27 18:41:21 [DEBUG] events Update Store my-shop setup job in namespace my-shop. Diff hash {"type": "Normal", "object": {"kind":"Store","namespace":"my-shop","name":"my-shop","uid":"0cd29409-5b25-4e5e-ac71-89e86a48dc7b","apiVersion":"shop.shopware.com/v1","resourceVersion":"513754547"}, "reason": "Diff setup job hash"} 2024-11-27 18:41:21 [INFO] my-shop Wait for setup to finish {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276", "state": "setup"} 2024-11-27 18:41:21 [INFO] Reconcile finished {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276"} 2024-11-27 18:41:21 [INFO] Found container for job `shopware-setup` {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276"} 2024-11-27 18:41:21 [INFO] Job not terminated still running {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276"} 2024-11-27 18:41:21 [INFO] Update store status {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "8f480550-5d3b-4023-b9e1-8a1570c38276", "status": {"state":"setup","conditions":[{"type":"setup","lastTransiti onTime":null,"lastUpdatedTime":"2024-11-27T18:41:21Z","message":"Waiting for setup job to finish (Notice sidecars are counted). Active jobs: 0, Failed jobs: 0","status":"True"}]}} 2024-11-27 18:41:21 [INFO] my-shop Do reconcile on store {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "5a9d1b84-b146-4eb1-9f1b-aa6568c7de81", "state": "setup"} 2024-11-27 18:41:21 [INFO] my-shop reconcile app secrets {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "5a9d1b84-b146-4eb1-9f1b-aa6568c7de81", "state": "setup"} 2024-11-27 18:41:21 [DEBUG] events Update Store my-shop setup job in namespace my-shop. Diff hash {"type": "Normal", "object": {"kind":"Store","namespace":"my-shop","name":"my-shop","uid":"0cd29409-5b25-4e5e-ac71-89e86a48dc7b","apiVersion":"shop.shopware.com/v1","resourceVersion":"513754547"}, "reason": "Diff setup job hash"} 2024-11-27 18:41:31 [INFO] my-shop Wait for setup to finish {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "5a9d1b84-b146-4eb1-9f1b-aa6568c7de81", "state": "setup"} 2024-11-27 18:41:31 [INFO] Reconcile finished {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "5a9d1b84-b146-4eb1-9f1b-aa6568c7de81"} 2024-11-27 18:41:31 [ERROR] job not found in container: shopware-setup {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "5a9d1b84-b146-4eb1-9f1b-aa6568c7de81", "error": "job not found in container: shopware -setup"} github.com/shopware/shopware-operator/internal/job.IsJobContainerDone /home/runner/work/shopware-operator/shopware-operator/internal/job/util.go:89 github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).stateSetup /home/runner/work/shopware-operator/shopware-operator/internal/controller/status.go:232 github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).reconcileCRStatus /home/runner/work/shopware-operator/shopware-operator/internal/controller/status.go:63 github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).Reconcile.func1 /home/runner/work/shopware-operator/shopware-operator/internal/controller/store_controller.go:107 github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).Reconcile /home/runner/work/shopware-operator/shopware-operator/internal/controller/store_controller.go:132 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 2024-11-27 18:41:31 [INFO] Update store status {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "5a9d1b84-b146-4eb1-9f1b-aa6568c7de81", "status": {"state":"setup","conditions":[{"type":"setup","lastTransiti onTime":null,"lastUpdatedTime":"2024-11-27T18:41:31Z","message":"Waiting for setup job to finish","reason":"job not found in container: shopware-setup","status":"Error"}]}} 2024-11-27 18:41:31 [INFO] my-shop Do reconcile on store {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "705040e8-3343-4517-a1d1-f55d68f24e7e", "state": "setup"} 2024-11-27 18:41:31 [INFO] my-shop reconcile app secrets {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "705040e8-3343-4517-a1d1-f55d68f24e7e", "state": "setup"} 2024-11-27 18:41:31 [DEBUG] events Update Store my-shop setup job in namespace my-shop. Diff hash {"type": "Normal", "object": {"kind":"Store","namespace":"my-shop","name":"my-shop","uid":"0cd29409-5b25-4e5e-ac71-89e86a48dc7b","apiVersion":"shop.shopware.com/v1","resourceVersion":"513755310"}, "reason": "Diff setup job hash"} 2024-11-27 18:41:41 [INFO] my-shop Wait for setup to finish {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "705040e8-3343-4517-a1d1-f55d68f24e7e", "state": "setup"} 2024-11-27 18:41:41 [INFO] Reconcile finished {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "705040e8-3343-4517-a1d1-f55d68f24e7e"} 2024-11-27 18:41:41 [INFO] Update store status {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "705040e8-3343-4517-a1d1-f55d68f24e7e", "status": {"state":"setup","conditions":[{"type":"setup","lastTransiti onTime":null,"lastUpdatedTime":"2024-11-27T18:41:41Z","message":"Waiting for setup job to finish","status":"True"}]}} 2024-11-27 18:41:41 [INFO] my-shop Do reconcile on store {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "9eed825f-148b-448a-bffd-46027bb35b3b", "state": "setup"} 2024-11-27 18:41:41 [INFO] my-shop reconcile app secrets {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop"}, "namespace": "my-shop", "name": "my-shop", "reconcileID": "9eed825f-148b-448a-bffd-46027bb35b3b", "state": "setup"} 2024-11-27 18:41:41 [DEBUG] events Update Store my-shop setup job in namespace my-shop. Diff hash {"type": "Normal", "object": {"kind":"Store","namespace":"my-shop","name":"my-shop","uid":"0cd29409-5b25-4e5e-ac71-89e86a48dc7b","apiVersion":"shop.shopware.com/v1","resourceVersion":"513755386"}, "reason": "Diff setup job hash"}Your custom resource
The text was updated successfully, but these errors were encountered: