Failed to provision with mapt

[lilyLuLiu]

I failed to provision with mapt. The error log is as follow:

{"level":"debug","msg":"running 'mapt aws rhel create'","time":"2024-07-23T07:26:35Z"}
{"level":"debug","msg":"context initialized for mapt8511f98f","time":"2024-07-23T07:26:35Z"}
{"level":"debug","msg":"checking stack spotOption-crc-qe-virtualized-rhel-971822005","time":"2024-07-23T07:26:35Z"}
{"level":"debug","msg":"managing stack spotOption-crc-qe-virtualized-rhel-971822005","time":"2024-07-23T07:27:37Z"}
{"level":"info","msg":"Updating (spotOption-crc-qe-virtualized-rhel-971822005):","time":"2024-07-23T07:31:44Z"}
{"level":"info","msg":"","time":"2024-07-23T07:31:44Z"}
{"level":"info","msg":" +  pulumi:pulumi:Stack crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005 creating (0s) ","time":"2024-07-23T07:31:44Z"}
{"level":"debug","msg":"grouped prices map[{ us-east-1a 0 0 0}:[{0xc000828330 c5d.metal Red Hat Enterprise Linux 0xc000828320 2024-07-23 04:16:37 +0000 UTC {}} {0xc000828350 c5.metal Red Hat Enterprise Linux 0xc000828340 2024-07-23 04:02:00 +0000 UTC {}} {0xc000828520 c5n.metal Red Hat Enterprise Linux 0xc000828510 2024-07-23 00:17:52 +0000 UTC {}}] { us-east-1b 0 0 0}:[{0xc0008282d0 c5.metal Red Hat Enterprise Linux 0xc0008282c0 2024-07-23 06:17:39 +0000 UTC {}} {0xc000828390 c5d.metal Red Hat Enterprise Linux 0xc000828380 2024-07-23 02:47:35 +0000 UTC {}} {0xc0008284c0 c5n.metal Red Hat Enterprise Linux 0xc0008284b0 2024-07-23 02:33:01 +0000 UTC {}}] { us-east-1c 0 0 0}:[{0xc000828290 c5n.metal Red Hat Enterprise Linux 0xc000828280 2024-07-23 06:32:36 +0000 UTC {}} {0xc0008282b0 c5d.metal Red Hat Enterprise Linux 0xc0008282a0 2024-07-23 06:17:53 +0000 UTC {}} {0xc000828400 c5n.metal Red Hat Enterprise Linux 0xc0008283a0 2024-07-23 02:46:39 +0000 UTC {}} {0xc000828500 c5.metal Red Hat Enterprise Linux 0xc0008284f0 2024-07-23 01:47:31 +0000 UTC {}}] { us-east-1d 0 0 0}:[{0xc0008282f0 c5d.metal Red Hat Enterprise Linux 0xc0008282e0 2024-07-23 06:02:50 +0000 UTC {}} {0xc000828310 c5n.metal Red Hat Enterprise Linux 0xc000828300 2024-07-23 05:17:20 +0000 UTC {}} {0xc000828370 c5.metal Red Hat Enterprise Linux 0xc000828360 2024-07-23 03:32:00 +0000 UTC {}}] { us-east-1f 0 0 0}:[{0xc000828420 c5n.metal Red Hat Enterprise Linux 0xc000828410 2024-07-23 02:46:39 +0000 UTC {}} {0xc0008284e0 c5.metal Red Hat Enterprise Linux 0xc0008284d0 2024-07-23 02:02:36 +0000 UTC {}}]]","time":"2024-07-23T07:31:45Z"}
{"level":"info","msg":"@ updating................................................................................................................................................................................................","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":" +  pulumi:pulumi:Stack crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005 creating (189s) error: an unhandled error occurred: go inline source runtime error, an unhandled error occurred: runtime error: invalid memory address or nil pointer dereference","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":" +  pulumi:pulumi:Stack crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005 **creating failed** 1 error","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"Diagnostics:","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"  pulumi:pulumi:Stack (crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005):","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"    error: an unhandled error occurred: go inline source runtime error, an unhandled error occurred: runtime error: invalid memory address or nil pointer dereference","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"Resources:","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"    + 1 created","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"Duration: 3m10s","time":"2024-07-23T07:34:53Z"}
{"level":"info","msg":"","time":"2024-07-23T07:34:53Z"}
{"level":"error","msg":"failed to run update: exit status 255\ncode: 255\nstdout: Updating (spotOption-crc-qe-virtualized-rhel-971822005):\n\n +  pulumi:pulumi:Stack crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005 creating (0s) \n@ updating................................................................................................................................................................................................\n +  pulumi:pulumi:Stack crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005 creating (189s) error: an unhandled error occurred: go inline source runtime error, an unhandled error occurred: runtime error: invalid memory address or nil pointer dereference\n +  pulumi:pulumi:Stack crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005 **creating failed** 1 error\nDiagnostics:\n  pulumi:pulumi:Stack (crc-qe-virtualized-rhel-971822005-spotOption-crc-qe-virtualized-rhel-971822005):\n    error: an unhandled error occurred: go inline source runtime error, an unhandled error occurred: runtime error: invalid memory address or nil pointer dereference\n\nResources:\n    + 1 created\n\nDuration: 3m10s\n\n\nstderr: \n","time":"2024-07-23T07:34:53Z"}
cat: /opt/storage/971822005/host: No such file or directory
cat: /opt/storage/971822005/username: No such file or directory
id_rsa

[anjannath]

@lilyLuLiu hi, can you please share the mapt command you ran, please also make sure you have the AWS_DEFAULT_REGION env variable set

[lilyLuLiu]

Hi, the error happened in new cluster's pipeline with running https://github.com/redhat-developer/mapt/blob/main/tkn/infra-aws-rhel.yaml task.
The most like problem is some address needs to be added into egress. But I can't find any clue in the log.

[adrianriobo]

yeah, we are having issues while connecting to the AWS API due to some egress rules policies on the cluster; so basically here, an according to this use case:

• In case of an error on that connection this should exit properly (maybe this is related on how spot price history is picked (not sure if with Pulumi or direct call to the AWS API, in any case the connection error should exist properly)

• Beyond the error it seems some data aggregation is panic ing due to some empty array (it seems like), probably if first error exit the execution this panic will not occur, but in any case we have a missing check for a value somewhere on the spot price module

[lilyLuLiu]

After debug, I found the address 169.254.169.254 is connected during the provision process.
After add this address into egress, the issue is solved.

[adrianriobo]

ACK

I will keep this one open, as we still need to add some checks to ensure a proper error is shown in case of wrong connectivity when login against the cloud provider