sync path configuration would overwrite path of existing Kustomization

[geoffhendrey]

Hi - I am trying to setup flux so that the flux GitRepositoryObjects and Kustomization objects in the flux-system folder point to a second repo where I define my environments. The app being deployed in the environments is just @stefanprodan's podinfo app.

flux bootstrap github --owner=geoffhendrey --repository=flux-system --private=false --personal=true --path=flux-system-overlays/env1 --branch=master --verbose

When I try to do this, I get: ✗ sync path configuration ("./flux-system-overlays/env1") would overwrite path ("./environment-overlays/env1") of existing Kustomization

yet it seems as though podinfo is properly deployed: `Every 2.0s: flux get kustomizations GHENDREY-M-G6C7: Sun Jun 13 22:57:57 2021

NAME READY MESSAGE REVISION SUSPENDED
flux-system True Applied revision: master/d75f0a9ef55af3eb000a108c426b552c69e86724 master/d75f0a9ef55af3eb000a108c426b552c69e86724 False
podinfo True Applied revision: master/685371108de7b5772c8500293ebe62f258bbc783 master/685371108de7b5772c8500293ebe62f258bbc783 False`

in the logs I see 2021-06-14T05:59:24.160Z info GitRepository/podinfo.flux-system - Reconciliation finished in 361.1029ms, next run in 30s so I believe reconciliation is fine.

I need help understanding why this error (mentioned earlier in the posd) is shown in the bootstrap output: ✗ sync path configuration ("./flux-system-overlays/env1") would overwrite path ("./environment-overlays/env1") of existing Kustomization

The repository I am using for flux-system is here: https://github.com/geoffhendrey/flux-system
The repo that is pointed to by the Kustomization controller in said flux-system repo, is here: https://github.com/geoffhendrey/flux-environments

[stefanprodan]

You can't change the path of a bootstrapped cluster, run flux uninstall, delete the ./flux-system-overlays/env1 from your repo and start over.

[geoffhendrey]

I am trying to setup a repo for flux-system that points to a second repo that holds the GitRepository object and Kustomization object for the podinfo repo. If I delete ./flux-system-overlays/env1 then I will be removing the information that tells flux-system about this second repo. Is there any guidance on how I can setup flux this way? Is it a matter of using flux create sources git rather than bootstrapping?

[stefanprodan]

You bootstrap once per cluster then you can add as many repos as you want to that cluster, please see the get started guide: https://fluxcd.io/docs/get-started/

[geoffhendrey]

The bootstrap command results in a cluster whose GitRepository branch and path point to the same repo that you bootstrapped into. The problem I foresee with this is that it makes it hard for us to take a branch off of the master flux-system folder and then merge that branch back to master. If we take a branch and test it, then the branch also would have to have its GirRepository object pointed at its test branch. Then when we merge back to master it essentially corrupts master. We have hundreds of clusters and my hope was to setup kustomize on the flux-system repo itself so that I could do things like adjust the sync interval in a base folder and have it affect all the clusters. So I split flux-system repo into a separate repo from the GitRepository objects that sync the apps themselves. This allows the flux-system cluster to be branched for test, and merged back in without any problems. What is the recommended way to do this?

[geoffhendrey]

also, thank you for the link to get started. I did start with that, and as I mentioned we perceived a problem with being able to do gitOps on the bootstrapped flux-system itself.

[ichandan16]

Hi @stefanprodan , I am facing similar issue and trying to figure out what is the right way to do it.

1. Created a directory structure: /opt/fluxcdwork/myfluxrepo as my base path for all the flux related work.

2. Created a path named ./clusters/my-cluster inside myfluxrepo.
   

3. Bootstrapped and it was successful. It is working as expected.

flux bootstrap gitlab 
      --owner=ckp1608 
      --repository=myfluxrepo --
      --branch=main
      --path=clusters/my-cluster 
      --token-auth --personal

4. Now, I am trying to boostrap another git repo in another folder. The directory path structure is: /opt/fluxcdwork/flux-fleets.
5. Running the bootstrap command as below. Getting error:

flux bootstrap gitlab \
  --owner=ckp1608 \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/test-cluster \
  --personal

sync path configuration ("./clusters/test-cluster") would overwrite path ("./clusters/my-cluster") of existing Kustomization.

Why this is happening? In this case there is no conflicts between linux directory path and the git repo. How flux is referring my existing cluster path?

Of course, the common components are in my kind k8s cluster where the previous bootstrap command installed the flux components.

I read the documentation but not getting an idea, how to manage multiple repos after only one bootstrap on same k8s cluster.
Appreciate your suggestions.

[scotlyt]

@ichandan16

I just encountered this problem and I believe @stefanprodan is correct. The problem for me was that I was trying to move repos on what is managing the cluster's flux deployment. If you already have a repo that is managing the flux-system for a cluster, it won't allow you to bootstrap it again on the system level.

So for context, I had my local minikube cluster setup and it was using github as the source of truth for the flux-system

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=fleet-infra-demo \
  --branch=main \
  --path=./clusters/my-cluster \
  --personal --components-extra=image-reflector-controller,image-automation-controller

This installs all the components onto whatever is my current context was, which was a minikube cluster locally.

I wanted to move this over to my internal gitlab server (I was starting with the tutorials and following the guides to get myself familiar with what it was doing behind the scenes for the tokens and what needs to be passed & set) so I wanted to bootstrap it again:

flux bootstrap gitlab \
  --owner=iac \
  --hostname=gitlab.*.* \
  --ssh-hostname=gitlab-ssh.*.* \
  --path=clusters/scotlocal \
  --context=minikube \ 
  --repository=fleet-infra \
  --deploy-token-auth \
  --branch=master --components=source-controller,helm-controller,kustomize-controller --components-extra=image-reflector-controller,image-automation-controller

But received the same error message:

Please enter your GitLab personal access token (PAT):
► connecting to https://gitlab.*.*
► cloning branch "master" from Git repository "https://gitlab.*.*/iac/fleet-infra.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
✔ reconciled components
► checking to reconcile deploy token for source secret
✔ configured deploy token "flux-system-master-flux-system-./clusters/scotlocal" for "https://gitlab.*.*/iac/fleet-infra"
► determining if source secret "flux-system/flux-system" exists
► generating source secret
► applying source secret "flux-system/flux-system"
✔ reconciled source secret
✗ sync path configuration ("./clusters/scotlocal") would overwrite path ("./clusters/my-cluster") of existing Kustomization

It's because my minikube local still had the flux system from the previous bootstrap, you need to delete & flux uninstall (with minikube as my kubectl context) so only a single repo is managing the flux-system, that's how I'm interpreting the issue.

[spuxx1701]

Sorry for necroing this thread, but I'm trying to reorganize my k8s cluster and wanted to give it a proper name. I tried running flux uninstall but even after doing that and waiting for the flux-system namespace to be deleted, I still get the same message. @stefanprodan I can't find flux-system-overlays. Is there any other location where the sync path might still be cached?

[stefanprodan]

The path is set inside gotk-sync.yaml in your Git repo, so I don't see how after running uninstall, bootstrap would fail for a new path.

[spuxx1701]

I feel stupid. 🤦🏼‍♂️ Somehow I missed those when running search/replace. Thanks for making me take another look.