Sync EKS clusters in different AWS accounts

[resworld]

Hello there,

We have tried to use aws-cloud-map-mcs-controller-for-k8s tool, but we like to have sync between two EKS clusters that are running in different AWS accounts (Not sure if that one is possible). VPC peering is enabled and nACL rules are checked - there's no restrictions between two accounts so we can access services running outside EKS in account_A from account_B (for example). When we have run the tool in both clusters and export a service in cluster running in account_B, we can see service's namespace is added in AWS cloud-map in account that is running, but cluster in account_A couldn't see any imported service:

# cluster in account_B:
$ kubectl get ServiceExport -A
NAMESPACE            NAME    AGE
cross-cluster-test   nginx   46h
===
# cluster in account_A:
$ kubectl get ServiceImport -A
No resources found

Is there any other config I need to do? Maybe sync two cloud-maps between accounts? I just followed the steps in Readme file, and I don't see any errors in cloud-map-mcs-controller-manager pod.

Thank you

[astaticvoid]

Hi, thanks for trying that out.
Yes, import and export is possible across clusters irrespective of EKS accounts or on-prem clusters as long as there is network connectivity between them and they are syncing with the same Cloud Map AWS account.
The controller in the importing cluster in account_A needs to be able to authenticate with the Cloud Map account that is being exported to from the cluster in account_B.
For import to begin in the second cluster you will have to create a namespace in the cluster with the same name as being exported. This is because we do not attempt to import everything in the Cloud Map account, as there may be other registered resources that are unrelated to a multicluster Kubernetes setup.
For routing between clusters to work to the imported service you will need coredns with the multi cluster plugin installed, which is being developed under a separate project. The plugin has an outstanding issue with routing to services which have different service & target ports that is currently being worked on.
We will be publishing a blog post in the coming days that outlines an EKS multicluster setup with coredns multicluster integration.
Hope that helps. It's an interesting use case, let me know if you get it working or if there are any further roadblocks.

[astaticvoid]

If you'd like to try out multicluster routing using the current state of the coredns plugin without building it, we are hosting an experimental image. 1.8.6 is currently tested and known to work (with the defect of same service/target port required as mentioned), and requires EKS 1.21. The 1.8.4 experimental build is a work in progress at the moment.

You can add the experimental plugin to the cluster with the following command:

kubectl set image --namespace $NAMESPACE deployment.apps/coredns coredns=ghcr.io/aws/aws-cloud-map-mcs-controller-for-k8s/coredns-multicluster/coredns:v1.8.6

[bendu]

For an EKS cluster in a different account than Cloud Map. I suggest taking a look at this post from AWS. This allows a pod in a EKS cluster in one AWS account to assume a role in a different AWS account. Once set up, you can annotate the controller's service account so it can access the other AWS account containing Cloud Map.

[resworld]

Thank you all for the help and suggestions. I have tried to do same with two EKS clusters in single account, but different regions and facing same issue - obviously because CloudMap isn't global resource so I need to follow post for cross account iam roles that @bendu mentioned.

Anyway, first we like to see if aws-cloud-map-mcs-controller is the thing that we need to achieve our goal, and that why I've tried something quicker that should work - creating 2 EKS clusters in single account and region. Just to make them use mcs controller as it is described in the documents.

So currently I am having same EKS (1.19) clusters created in same account. I've did installation in both clusters:

kubectl apply -k "github.com/aws/aws-cloud-map-mcs-controller-for-k8s/config/controller_install_release"

and in one of the clusters (let's say CLUSTER1) I've did:

kubectl create namespace nginx-test
kubectl create deployment nginx --image=nginx -n nginx-test
kubectl expose deployment nginx --port=80 -n nginx-test

This brings me very dummy nginx service in nginx-test namespace. I have applied following ServiceExport's file:

kind: ServiceExport
apiVersion: multicluster.x-k8s.io/v1alpha1
metadata:
  namespace: nginx-test
  name: nginx

After all that steps I am able to see namespace (nginx-test) created in AWS Cloud map and service can be seen in CLUSTER1 -> kubectl get ServiceExport -A , but problem is that is missing in CLUSTER2 -> kubectl get ServiceImport -A

In the cloud-map-mcs-controller-manager's pod manager container of CLUSTER1 I can see successfully creation of ServiceExport with no error (I can share logs if needed), but still no luck in CLUSTER2 to see that service imported. Is there anything else I need to create or do in CLUSTER2?

I am sorry if this isn't correct place to put my question, if I need to open another discussion/issue, please let me know. Thank you!

[resworld]

I found what was missing - I need to create the namespace in CLUSTER2. Now it looks I can see it. Thank you.

[bendu]

If you need cross region cluster service discovery, you can modify the pod definition to set the AWS_REGION environment variable for one of the controllers to override the default region inferred from the EKS cluster. As usual, you'll still need to set up some kind of connectivity (VPC peering, security groups, etc.) between the clusters.