"Legacy" Lambda Layers keep failing deployments

[hisham]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

20.12.1

Amplify CLI Version

12.12.4

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

no

Describe the bug

amplify push fails from time to time with error Resource is not in the state stackUpdateComplete. When I look at the error detail in cloudformation, the error is:
The following resource(s) failed to create: [LambdaLayerVersion675d0075, LambdaLayerPermissionAwsAccounts914282159778e0e2307fLegacy406].

and Resource handler returned message: "1 validation error detected: Value '914282159778e0e2307f' at 'principal' failed to satisfy constraint: Member must satisfy regular expression pattern: \d{12}|\*|arn:(aws[a-zA-Z-]*):iam::\d{12}:root (Service: AWSLambdaInternal; Status Code: 400; Error Code: ValidationException; Request ID: f11c4ba0-5e26-4eca-a520-39c4ed933b61; Proxy: null)" (RequestToken: b59897b3-cc95-1d5c-3e8a-7c499e5b1214, HandlerErrorCode: GeneralServiceException)

The issue is these "legacy" layers in my *Layer-awscloudformation-template.json file:

    "LambdaLayerPermissionAwsAccounts914282159778e0e2307fLegacy406": {
      "Type": "AWS::Lambda::LayerVersionPermission",
      "Properties": {
        "Action": "lambda:GetLayerVersion",
        "LayerVersionArn": "arn:aws:lambda:us-east-1:914282159778:layer:essappCliLambdaLayer-hishamdev:406",
        "Principal": "914282159778e0e2307f"
      }
    },

But these layers are not legacy at all. They've been deployed in the last few months. I workaround this issue by deleting the layer manually and removing the "Legacy" entries in the cloudformation file like the one above. However, this error is now affecting the latest layer I have, so if I delete it, my lambdas will not function correctly.

Expected behavior

amplify push should just work and update layers

Reproduction steps

Unclear what the repro steps are, but this issue has been happening now in pretty much every push in one specific environment in my stack.

Project Identifier

3def1a2bd59d61900f734d59f169a578

Log output

# Put your logs below this line

Additional information

This issue is also discussed in #8525 but the resolution there does not work.

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[hisham]

I worked around this issue btw via amplify update function and re-confirming the permissions for the troublesome layer version...

[hisham]

this keeps happening pretty much anytime I push a new layer version....