Redeployment for azure nva-into-existing-hub fails

[LukasAuSc]

Hi,

firstly thank you for the implementations for different cloud providers and languages.
I am currently struggling with an strange issue for deploying the terraform integration on azure, nva-into-existing-hub.

The first deployment works perfectly fine, in under 10 min, but afterwards it tries to deploy again, and fails with the following error:

2024-11-05T12:36:35.614+0100 [ERROR] provider.terraform-provider-azurerm.exe: Response contains error diagnostic: @module=sdk.proto tf_proto_version=5.4 tf_req_id=864f4aa4-0ec7-b9db-478b-4d5b999de45d tf_resource_type=azurerm_managed_application @caller=/home/runner/work/terraform-provider-azurerm/terraform-provider-azurerm/provider/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" diagnostic_severity=ERROR tf_rpc=ApplyResourceChange
  diagnostic_summary=
  | failed to create Application (Subscription: "***************"
  | Resource Group Name: "***************"
  | Application Name: "***************"): polling after CreateOrUpdate: polling failed: the Azure API returned the following error:
  |
  | Status: "Failed"
  | Code: "RoleAssignmentExists"
  | Message: "The role assignment already exists."
  | Activity Id: ""
  |
  | ---
  |
  | API Response:
  |
  | ----[start]----
  | {"id":"***************","name":"***************","resourceId":"***************","status":"Failed","startTime":"2024-11-05T11:21:21.1914472Z","endTime":"2024-11-05T11:36:34.8203636Z","error":{"code":"RoleAssignmentExists","message":"The role assignment already exists."}}
  | -----[end]-----
   tf_provider_addr=provider timestamp="2024-11-05T12:36:35.614+0100"
2024-11-05T12:36:35.616+0100 [ERROR] vertex "azurerm_managed_application.nva" error: failed to create Application (Subscription: "***************"
Resource Group Name: "***************"
Application Name: "***************"): polling after CreateOrUpdate: polling failed: the Azure API returned the following error:

Status: "Failed"
Code: "RoleAssignmentExists"
Message: "The role assignment already exists."
Activity Id: ""

What I was able to figure out, is that with the first deployment the automatic created UAI vwan-managed-identity, is being created and assigned as Managed Application Operator Role to the NVA.
With the second deployment, the above error comes up and the deployment fails after >15min. When I delete the role assignment, the deployment works again smoothly, but if deploying it again, same error.

If you can help me or point me in the right direction, I would greatly appreciate it. Since I can not see into the application, unfortuantely I am stuck here.

[chkp-natanelm]

Hi @LukasAuSc,
Could you clarify what you mean by “deploy again”? Did you destroy the previous deployment and start over, or did you simply modify the parameters and re-run the Terraform template?
Thanks

[LukasAuSc]

Hi @chkp-natanelm, thank you for the quick response.
deploy again really just meaning, run the same apply command than before, not even modifying the parameters.
First it recognizes something as changed in the parameter_values, which looks like this

  # azurerm_managed_application.nva will be updated in-place
  ~ resource "azurerm_managed_application" "nva" {
        id                          = "***************"
        name                        = "***************"
      ~ parameter_values            = (sensitive value)
        tags                        = {}
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

(shortened for readability), and then when running it, it fails with the above mentioned error.

And when I delete the role assignment, I can redeploy without issue. It looks to me like the role assignment is recreated with every apply step, even if it was created before. If it would be terraform code, I would say it has not been saved in the state, but I do not know how the managed application works.

[chkp-natanelm]

Hi @LukasAuSc,
Thank you for the feedback.
We will investigate it, and update the issue with the progress.

[chkp-olgami]

Hi @LukasAuSc
Could you clarify the purpose of running terraform apply again without changing any parameters? Additionally, have there been any other modifications to the Terraform configuration? I’m unable to reproduce the issue on my end.

[LukasAuSc]

Hi @chkp-olgami,
we are running all our configuration inside pipelines, and those must be idempotent, to allow further additions to the pipelines and repositories.
I have adjusted your templates to our code structure, but as far as I know there have been no other changes.

For reference, here are our used files:
main.tf

# ------------------
# Resource Groups
# ------------------

resource "azurerm_resource_group" "firewall" {
  name     = module.meta.resource_group_name.network_firewall
  location = module.meta.locations.weu
}

# ------------------
# Check Point NVA
# based on https://github.com/CheckPointSW/CloudGuardIaaS/tree/master/terraform/azure/nva-into-existing-hub
# ------------------

resource "azurerm_marketplace_agreement" "accept-marketplace-terms" {
  publisher = "checkpoint"
  offer     = "cp-vwan-managed-app"
  plan      = "vwan-app"
}

resource "azurerm_resource_provider_registration" "solutions" {
  name = "Microsoft.Solutions"
}

resource "azurerm_managed_application" "nva" {
  depends_on = [
    azurerm_marketplace_agreement.accept-marketplace-terms,
    azurerm_resource_provider_registration.solutions,
  ]

  name                        = module.meta.resource["${local.module_identifier_nva}-${var.purpose_nva}"].name
  location                    = azurerm_resource_group.firewall.location
  resource_group_name         = azurerm_resource_group.firewall.name
  kind                        = "MarketPlace"
  managed_resource_group_name = module.meta.resource_group_name.network_nva

  plan {
    name      = "vwan-app"
    product   = "cp-vwan-managed-app"
    publisher = "checkpoint"
    version   = "1.0.15"
  }

  parameter_values = jsonencode({
    location = {
      value = azurerm_resource_group.firewall.location
    },
    hubId = {
      value = data.azurerm_virtual_hub.this.id
    },
    osVersion = {
      value = var.os-version
    },
    LicenseType = {
      value = var.license-type
    },
    imageVersion = {
      value = element(local.image_versions, length(local.image_versions) - 1)
    },
    scaleUnit = {
      value = var.scale-unit
    },
    bootstrapScript = {
      value = var.bootstrap-script
    },
    adminShell = {
      value = var.admin-shell
    },
    sicKey = {
      value = var.sic-key
    },
    sshPublicKey = {
      value = var.ssh-public-key
    },
    BGP = {
      value = var.bgp-asn
    },
    NVA = {
      value = module.meta.resource["${local.module_identifier_nva}-${var.purpose_nva}"].name
    },
    customMetrics = {
      value = var.custom-metrics
    },
    hubASN = {
      value = data.azurerm_virtual_hub.this.virtual_router_asn
    },
    hubPeers = {
      value = data.azurerm_virtual_hub.this.virtual_router_ips
    },
    smart1CloudTokenA = {
      value = var.smart1-cloud-token-a
    },
    smart1CloudTokenB = {
      value = var.smart1-cloud-token-b
    },
    smart1CloudTokenC = {
      value = var.smart1-cloud-token-c
    },
    smart1CloudTokenD = {
      value = var.smart1-cloud-token-d
    },
    smart1CloudTokenE = {
      value = var.smart1-cloud-token-e
    },
    publicIPIngress = {
      value = (var.new-public-ip == "yes" || length(var.existing-public-ip) > 0) ? "yes" : "no"
    },
    createNewIPIngress = {
      value = var.new-public-ip
    }
    ipIngressExistingResourceId = {
      value = var.existing-public-ip
    }
  })

}

dependencies.tf

# ------------------
# Hub
# ------------------

data "azurerm_virtual_hub" "this" {
  name                = module.meta.resource["${local.module_identifier_virtual_hub}-${var.purpose_hub}"].name
  resource_group_name = module.meta.resource_group_name.network_vwan
}

# ------------------
# Check Point NVA
# based on https://github.com/CheckPointSW/CloudGuardIaaS/tree/master/terraform/azure/nva-into-existing-hub
# ------------------

data "external" "az_access_token" {
  program = [
    "az", "account",
    "get-access-token",
    "--resource=https://management.azure.com",
    "--query={accessToken: accessToken}",
    "--output=json"
  ]
}

data "http" "image-versions" {
  method = "GET"
  url = (
    "https://management.azure.com/subscriptions/${module.meta.tenant_config[lower(var.environment)][lower(var.project_config.project_name_short)]}/providers/Microsoft.Network/networkVirtualApplianceSKUs/checkpoint${var.license-type == "Full Package (NGTX + S1C)" ?
    "-ngtx":
    var.license-type == "Full Package Premium (NGTX + S1C++)" ? "-premium" : ""}?api-version=2020-05-01"
  )

  request_headers = {
    Accept          = "application/json"
    "Authorization" = "Bearer ${local.access_token}"
  }
}

and locals.tf

locals {
  module_identifier_nva         = "Nva"
  module_identifier_virtual_hub = "Vhub"
  access_token                  = data.external.az_access_token.result.accessToken

  # Validate that new-public-ip is false when existing-public-ip is used
  is_both_params_used     = length(var.existing-public-ip) > 0 && var.new-public-ip == "yes"
  validation_message_both = "Only one parameter of existing-public-ip or new-public-ip can be used"
  _                       = regex("^$", (!local.is_both_params_used ? "" : local.validation_message_both))

  image_versions = tolist([
    for version in jsondecode(data.http.image-versions.response_body).properties.availableVersions :
    version if(
      substr(version, 0, 4) == substr(lower(length(var.os-version) > 3 ?
        var.os-version :
        "${var.os-version}00"), 1, 4
    ))
  ])

}

We have a meta module, that delivers uniform naming for resources, otherwise it should be the same as in the template.

[chkp-olgami]

Hi @LukasAuSc
We currently don’t support any customizations to our Terraform template.
However, I can share that we are working on improving our Terraform structure to make it more flexible and support modifications.
This enhancement is on our roadmap for 2025, but I don’t have an exact timeline to share at the moment.

I’ll tentatively close this issue for now. If you need further assistance or details, please reach out to Check Point support.

Thank you for your understanding and cooperation.

[LukasAuSc]

Short Update from my side, thanks to @chkp-yairra, who helped me with troubleshooting the usage of the official module. Now the deployment and redeployment works.