Merged PR 280: feat(iot-ops): add Bicep deployment scripts support for Azure Key Vault integration

This PR introduces significant enhancements to Bicep blueprint modules, deployment script management, and Key Vault integration for Azure IoT Operations. The changes include new blueprint modules, expanded parameters, improved outputs, and refactored resource and secret management for deployment automation and security.

- **feat(blueprints)**(_bicep_): added new blueprint modules for only-cloud-single-node-cluster and only-edge-iot-ops
  - Introduced `blueprints/only-cloud-single-node-cluster/bicep/main.bicep` and
    `blueprints/only-edge-iot-ops/bicep/main.bicep` for targeted deployments
  - Added corresponding `types.core.bicep` files for type safety and parameter consistency
- **feat(blueprints)**(_bicep_): expanded parameters and outputs in full-single-node-cluster and
  full-multi-node-cluster
  - Added IoT Operations, trust, Key Vault, and deployment script parameters
  - Enhanced outputs to expose extension, instance, and data flow resource details
- **feat(cncf-cluster)**(_bicep_): refactored cluster and Key Vault parameters, improved script and
  secret handling
  - Unified and clarified parameters for Arc onboarding, Key Vault, and deployment scripts
  - Updated modules to use new parameter names and improved environment variable handling
  - Added support for storing and retrieving deployment and K3s tokens as Key Vault secrets
- **feat(security-identity)**(_bicep_): added deployment identity outputs and resources
  - Created new managed identity for deployment automation
  - Exposed deployment identity details in outputs
- **feat(iot-ops)**(_bicep_): improved deployment script and secret management
  - Added parameters and logic for deployment scripts and Key Vault integration
  - Enhanced support for trust issuer and secret sync configuration
- **chore**: updated .gitignore to exclude generated PR files
- **fix(blueprints)**: enabled event grid and event hub data flows in Terraform blueprints
- **chore**: various documentation and linting improvements

## Notes

- Several parameters for deployment scripts are currently disabled and set as variables, pending
  support for AZ CLI 2.71+ in DeploymentScripts (expected post May 4).
- The new blueprints allow for more granular deployment scenarios, including cloud-only and
  edge-only setups.

## Important

- Ensure that Key Vault and managed identities are properly provisioned before running deployment
  scripts.
- Review new and updated parameters in blueprint and component modules for required values.

## Follow-up Tasks

- Update deployment scripts to leverage enabled parameters once AZ CLI 2.71+ is supported in
  DeploymentScripts
- Review and update documentation for new blueprint usage and parameterization

🚀 - Generated by Copilot

Related work items: #447

Author: agreaves-ms

.github/copilot-instructions.md

@@ -1,62 +1,20 @@
 # Copilot Instructions
 
-This is a large multi-faceted repo, so as you work in it, help the user ramp up by:
-
-- Add links and short tips and explanations where it would help the user learn the organization of the repo or the structure of the deployed solution.
-- Never guess or assume default options or parameters. Instead ask the user questions needed, provide a suggested set of defaults, but confirm these with the user (with alternatives where appropriate).
-
-## Deploy the Solution (via a Blueprint)
-
-- You can assist deploying the solution by running through the instructions from [the Blueprint README](../blueprints/README.md). (Use this carefully step-by-step and refer back to it as you go). Guidance for running the deployment:
-
-- You will be using the Azure CLI to build a `terraform.tfvars` using Azure CLI to get the required info
-- For a learning/sandbox deployment, the 'single-full-cluster' deployment is a good default, but please ask.
-- For `resource-prefix` always ask the user what to use here (this ensures multiple deployments in the same RG won't conflict).
-- The code in the repo works for a normal deployment. Do not modify any repo files during an assisted deployment. If there are issues with Azure Login/AAD, they are likely config issues.
-- If running a CLI operation with an input prompt (e.g. Terraform operation with args that aren't in tf vars), fill these into the CLI so the shell operation doesn't prompt.
-
-## Build a new Component
-
-- Components are defined in the [src](../src/) directory. [Components README](../src/README.md). To create a new component, follow the examples in this folder.
-
-## Build a new Blueprint
-
-- Blueprints are defined in the [blueprints](../blueprints/) directory. [Blueprints README](../blueprints/README.md). Use an existing Blueprint as a starting point, usually [full-single-snode-cluster](../blueprints/full-single-node-cluster/terraform/) and add/remove as needed.
-- When building a new Blueprint, use the existing building blocks in `/src`.
-
-## Terraform
-
-- Always do a `terraform plan` before running `apply`.
-- Always run `terraform fmt` to maintain consistent code formatting for project files (.gitignored files like tfvars files excluded).
-- Organize infrastructure resources into reusable modules.
-- Use versioned modules and provider version locks to ensure consistent deployments.
-- Use `terraform state` commands to inspect, move, or remove resources in the state when necessary.
-- Run `terraform refresh` to ensure that state reflects the current infrastructure.
-- When building reusable modules (in the `/src` folder), always write tests in a `tests` directory of the main terraform module.
-
-## Markdown
-
-- Follow Markdown [megalinter](../.mega-linter.yml) rules for all markdown files.
-- Use proper header formatting: blank line before and after headers
-- Use proper list formatting:
-  - Use `-` for unordered lists
-  - Use `1.` for ordered lists
-  - Use `>` for blockquotes
-  - Use `*` for emphasis (italics) and `**` for strong emphasis (bold)
-  - Blank line before and after lists
-- Use proper table formatting:
-  - Include a header row with column names
-  - Use the `|` character to define columns
-  - Include a separator row with dashes (e.g., `|------|------|`)
-  - Tables are exempted from line length limits, but should remain readable
-- Use proper code block formatting:
-  - Use triple backticks (```) for code blocks
-  - Always specify the language for syntax highlighting (e.g., ` ```hlc `, ` ```python `, ` ```bicep ` and so on)
-  - Use indentation for nested code blocks
-  - Blank line before and after code blocks
-- The repo uses MARKDOWN_MARKDOWN_TABLE_FORMATTER to automatically format tables
-- Follow consistent header formatting with proper spacing (one space after `#`)
-- Use ordered lists (`1.`, `2.`, etc.) for sequential steps and unordered lists (`-`) for non-sequential items
-- Code blocks should specify the language for proper syntax highlighting
-- Links should use reference-style format when referencing the same URL multiple times
-- Avoid HTML elements except those explicitly allowed in `.markdownlint.json` (`details` and `summary`)
+- You are RESPONSIBLE for ALWAYS reading in AT LEAST 1000+ (thousand) lines at a time for prompt and instruction files.
+- You will ALWAYS follow the `prompt-instruction-rules`.
+
+## Prompt and Instruction Rules
+
+<!-- <prompt-instruction-rules> -->
+- You will ALWAYS read and follow instructions from ./instructions/general.instructions.md
+- You will ALWAYS read and follow instructions from ./instructions/structure.instructions.md
+- You will ALWAYS read and follow instructions from ./prompts/getting-started.prompt.md WHEN prompted with anything about getting started or quick start.
+- You will ALWAYS read and follow instructions from ./prompts/deploy.prompt.md WHEN prompted with anything about deployment.
+- You will ALWAYS read and follow instructions from ./prompts/pr.prompt.md WHEN prompted with anything about pull request generation or pr creation.
+- You will ALWAYS read and follow instructions from ./instructions/bicep.instructions.md WHEN prompted with anything about bicep.
+- You will ALWAYS read and follow instructions from ./instructions/csharp-tests.instructions.md WHEN prompted with anything about C# tests or csharp tests.
+- You will ALWAYS read and follow instructions from ./instructions/csharp.instructions.md WHEN prompted with anything about C# or csharp.
+- You will ALWAYS read and follow instructions from ./instructions/markdown.instructions.md WHEN prompted with anything about markdown.
+- You will ALWAYS read and follow instructions from ./instructions/python-script.instructions.md WHEN prompted with anything about python.
+- You will ALWAYS read and follow instructions from ./instructions/terraform.instructions.md WHEN prompted with anything about terraform.
+<!-- </prompt-instruction-rules> -->

.github/prompts/bicep.prompt.md …ithub/instructions/bicep.instructions.md.github/prompts/bicep.prompt.md renamed to .github/instructions/bicep.instructions.md .github/prompts/bicep.prompt.md renamed to .github/instructions/bicep.instructions.md

@@ -1,18 +1,21 @@
-# Bicep Infrastructure as Code (IaC) Conventions and Best Practices
+---
+applyTo: '**/*.bicep*'
+---
+# Bicep IaC Conventions and Best Practices
 
 You are an expert in Bicep Infrastructure as Code (IaC) with deep knowledge of Azure resources. When writing or evaluating Bicep code, always follow the conventions in this document.
 
 ## Repository Structure
 
 This repository is organized with the following key directories:
 
-1. **Source Components** (`/src`) - Individual reusable infrastructure components:
+1. **Source Components** (`src/`) - Individual reusable infrastructure components:
    - `000-cloud/` - Cloud-based resources (Resource Groups, Identity, Storage, etc.)
    - `100-edge/` - Edge-based resources (IoT Operations, CNCF clusters, etc.)
    - `500-application/` - Application resources and source code
    - `900-tools-utilities/` - Tools and utilities (YAML, Helm charts, etc.)
 
-2. **Blueprints** (`/blueprints`) - End-to-end solutions that combine components:
+2. **Blueprints** (`blueprints/`) - End-to-end solutions that combine components:
    - `full-single-node-cluster/` - Single-node AIO deployment
    - `full-multi-node-cluster/` - Multi-node HA AIO deployment
    - `only-output-cncf-cluster-script/` - Script-only deployment
@@ -25,7 +28,7 @@ Component Modules are the top-level, standalone modules that provide a specific
 
 **Characteristics of Component Modules:**
 
-- Located directly under a component directory in `/src` (e.g., `/src/100-edge/110-iot-ops/bicep/`)
+- Located directly under a component directory in `src/` (e.g., `src/100-edge/110-iot-ops/bicep/`)
 - Exposed to be called from **Blueprint** modules
 - Represent a complete, self-contained functional unit (e.g., Resource Group, CNCF Cluster, IoT Ops)
 - Can use their own Internal Modules but NEVER reference other Component Modules
@@ -36,7 +39,7 @@ Component Modules are the top-level, standalone modules that provide a specific
 **Example Component Module path:**
 
 ```plain
-/src/100-edge/110-iot-ops/bicep/
+src/100-edge/110-iot-ops/bicep/
 ```
 
 ### Internal Modules
@@ -45,17 +48,16 @@ Internal Modules are subordinate, reusable modules that are only used within the
 
 **Characteristics of Internal Modules:**
 
-- Located in the `modules` subdirectory of a Component Module (e.g., `/src/100-edge/110-iot-ops/bicep/modules/iot-ops-instance.bicep`)
+- Located in the `modules` subdirectory of a Component Module (e.g., `src/100-edge/110-iot-ops/bicep/modules/iot-ops-instance.bicep`)
 - NEVER called directly from Blueprints or other Component Modules
-- Only called from their parent Component Module
 - Implement specific functionality within the scope of their parent Component
 - Have narrower scope and focused responsibility
 - Include metadata name and description at the top of the file
 
 **Example Internal Module path:**
 
 ```plain
-/src/100-edge/110-iot-ops/bicep/modules/iot-ops-instance.bicep
+src/100-edge/110-iot-ops/bicep/modules/iot-ops-instance.bicep
 ```
 
 ### CI Bicep Directories
@@ -64,7 +66,7 @@ CI Bicep directories are wrapper directories used for CI/CD integration that lev
 
 **Characteristics of CI Bicep Directories:**
 
-- Located at `<component>/ci/bicep/` (e.g., `/src/100-edge/110-iot-ops/ci/bicep/`)
+- Located at `<component>/ci/bicep/` (e.g., `src/100-edge/110-iot-ops/ci/bicep/`)
 - Contains simple code that calls the parent Component Module with default/test configurations
 - Used for individual component testing and verification in CI/CD pipelines
 - Acts as a thin wrapper for the component, not setting up test resources
@@ -145,6 +147,39 @@ Each blueprint includes:
 
 ## Bicep Coding Conventions
 
+- ALWAYS use the same API version for the same resource, the `@2023-01-31` part is the API version
+- ALWAYS use the existing API version for the same resource
+- ALWAYS use the latest API version for the same resource, as an example:
+  - If the following code exists in the codebase:
+
+  ```bicep
+  resource sseIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' = {
+  }
+
+  // Somewhere else in the codebase...
+  resource aioIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
+  }
+  ```
+
+  - Then the code should be updated to the following:
+
+  ```bicep
+  resource sseIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' = {
+  }
+
+  // Somewhere else in the codebase...
+  resource aioIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2024-11-30' = {
+  }
+  ```
+
+- ALWAYS search the codebase for existing Bicep resources to use as a reference when editing
+  - WHEN no reference exists, it is important to always use that latest Bicep ARM API reference and version
+    - Use VS Code's API Tooling for getting the ARM reference and version information
+    - Fallback to navigating and completely reading in the web pages (excluding images) for the following template, `https://learn.microsoft.com/azure/templates/{provider-namespace}/{resource-type}` for the latest Bicep ARM API reference and version
+      - e.g., `https://learn.microsoft.com/azure/templates/microsoft.managedidentity/userassignedidentities`
+- ALWAYS after making edits, verify VS Codes's information, warnings, and errors for Bicep
+  - ALWAYS make any and all corrections, if unable, fallback to these prompt instructions
+
 ### General Conventions
 
 - ALWAYS use `kebab-case` for file and folder names
@@ -292,7 +327,7 @@ resource example '[email protected]' = if (shouldCreateExample) {
 */
 
 module exampleInternalModule 'modules/example-internal-module.bicep' = {
-  name: '${deployment().name}-exampleInternalModule'
+  name: '${deployment().name}-eim0'
   params: {
     aioPlatformConfig: aioPlatformConfig
     common: common
@@ -321,12 +356,12 @@ output storageAccountName string = exampleInternalModule.outputs.storageAccountN
 
 IMPORTANT RULES:
 
-- Component Modules (e.g., `/src/100-edge/110-iot-ops/bicep/`) NEVER reference other Component Modules
+- Component Modules (e.g., `src/100-edge/110-iot-ops/bicep/`) NEVER reference other Component Modules
 - Component Modules NEVER reference other Component Modules' Internal Modules
 - Component Modules ONLY reference their own Internal Modules
-- Internal Modules (e.g., `/src/100-edge/110-iot-ops/bicep/modules/iot-ops-instance.bicep`) NEVER reference other Component Modules
+- Internal Modules (e.g., `src/100-edge/110-iot-ops/bicep/modules/iot-ops-instance.bicep`) NEVER reference other Component Modules
 - Internal Modules NEVER reference other Component Modules' Internal Modules
-- Blueprint Modules (e.g., `/blueprints/full-multi-node-cluster/bicep/`) ONLY reference Component Modules, NEVER Internal Modules
+- Blueprint Modules (e.g., `blueprints/full-multi-node-cluster/bicep/`) ONLY reference Component Modules, NEVER Internal Modules
 
 ### Parameters Conventions
 
@@ -375,15 +410,15 @@ Example:
 
 ```bicep
 module roleAssignment 'modules/role-assignment.bicep' = if (shouldAssignKeyVaultRoles) {
-  name: '${deployment().name}-roleAssignment'
+  name: '${deployment().name}-ra0'
   params: {
     keyVaultName: sseKeyVaultName
     sseUserAssignedIdentityName: sseIdentityName
   }
 }
 
 module iotOpsInstance 'modules/iot-ops-instance.bicep' = {
-  name: '${deployment().name}-iotOpsInstance'
+  name: '${deployment().name}-ioi1'
   params: {
     aioDataFlowInstanceConfig: aioDataFlowInstanceConfig
     aioExtensionConfig: aioExtensionConfig
@@ -433,7 +468,7 @@ output aioMqBrokerId string = shouldDeployResourceSyncRules ? mqBroker.id : ''
 - DO use the safe access operator `.?` when accessing potentially null properties
 - DO create default objects with default values for complex types
 - DO use conditional deployments with `if` statements for optional resources
-- DO use `${deployment().name}` prefix for module names to ensure uniqueness
+- DO use `${deployment().name}` prefix for module names and acronym followed by order number for uniqueness
 - DO use section headers with `/*` comments to organize your code
 - DO include comments for important logic or implementation details
 - DO validate inputs when appropriate, especially for security-sensitive parameters
@@ -455,9 +490,9 @@ output aioMqBrokerId string = shouldDeployResourceSyncRules ? mqBroker.id : ''
 Before making ANY changes to Bicep code, ask yourself:
 
 - [ ] Am I working on a Component Module, an Internal Module, or a Blueprint?
-  - Component Module: Located directly under `/src/000-grouping/000-component/bicep/`
-  - Internal Module: Located under `/src/000-grouping/000-component/bicep/modules/module-name.bicep`
-  - Blueprint: Located under `/blueprints/blueprint-name/bicep/`
+  - Component Module: Located directly under `src/000-grouping/000-component/bicep/`
+  - Internal Module: Located under `src/000-grouping/000-component/bicep/modules/module-name.bicep`
+  - Blueprint: Located under `blueprints/blueprint-name/bicep/`
 - [ ] Am I following the correct file structure for this module type?
 - [ ] Will my module references follow the module relationship rules?
 - [ ] Are my parameters organized with appropriate groupings?

.github/prompts/csharp-tests.prompt.md …nstructions/csharp-tests.instructions.md.github/prompts/csharp-tests.prompt.md renamed to .github/instructions/csharp-tests.instructions.md .github/prompts/csharp-tests.prompt.md renamed to .github/instructions/csharp-tests.instructions.md

@@ -1,5 +1,10 @@
+---
+applyTo: '**/*.cs,**/*.csproj'
+---
 # C# (csharp) Tests
 
+Only use this section when the prompt is specifically asking for C# tests. This section is only for C# tests and not
+for C# code.
 You are an expert in the latest C#, .NET, and testing. You will always follow common idioms, conventions, and best
 practices when writing C#. You will always refer to any additional provided prompts on C# (csharp) for creating code.
 

.github/prompts/csharp.prompt.md …thub/instructions/csharp.instructions.md.github/prompts/csharp.prompt.md renamed to .github/instructions/csharp.instructions.md .github/prompts/csharp.prompt.md renamed to .github/instructions/csharp.instructions.md

@@ -1,3 +1,6 @@
+---
+applyTo: '**/*.cs,**/*.csproj'
+---
 # C# (csharp)
 
 You are an expert in the latest C# and .NET, you will always follow common idioms, conventions, and best practices when
@@ -20,7 +23,7 @@ be used as an example).
 ## C# (csharp) Code Conventions and Styles
 
 Before writing any new C#, you will always make sure that your knowledge is up-to-date by referring to the
-websites outlined in the [Latest C# (csharp)](#latest-c-csharp) section of this document.
+websites outlined in the Latest C# (csharp) section of this document.
 
 Important - You will always prefer the following conventions (unless explicitly told otherwise or the conventions in the
 code files differ):

.github/instructions/general.instructions.md

@@ -0,0 +1,7 @@
+---
+applyTo: '**'
+---
+# Project Instructions
+
+- You will ALWAYS follow the `prompt-instruction-rules`.
+- BEFORE/DURING/AFTER ANY edits/responses you will ALWAYS verify that you have read all the content for instruction or prompt files.

.github/instructions/markdown.instructions.md

@@ -0,0 +1,24 @@
+---
+applyTo: '**/*.md'
+---
+# Markdown Instructions
+
+<!-- <markdown-instructions> -->
+- Before any edits you will read required linting rules from ../../.mega-linter.yml.
+- You will always edit following the `.mega-linter.yml` rules.
+- You will always edit with `markdown-proper-formatting` formatting.
+<!-- </markdown-instructions> -->
+
+## Proper Formatting Instructions
+
+<!-- <markdown-proper-formatting> -->
+- Headers always have a blank line before and after.
+- Titles always have a blank line after #.
+- Unordered lists always use `-`.
+- Ordered lists always use `1.`.
+- Lists always have a blank line before and after.
+- Code blocks always use triple backticks with the language specified.
+- Tables always have a header row, separator row, and `|` for columns.
+- Links always use reference-style for repeated URLs.
+- You will only ever allow `details` and `summary` HTML elements.
+<!-- </markdown-proper-formatting> -->

.github/prompts/python-script.prompt.md …structions/python-script.instructions.md.github/prompts/python-script.prompt.md renamed to .github/instructions/python-script.instructions.md .github/prompts/python-script.prompt.md renamed to .github/instructions/python-script.instructions.md

@@ -1,10 +1,15 @@
+---
+applyTo: '**/*.py'
+---
 # Python Script Conventions and Best Practices
 
-You are an expert in Python scripting with deep knowledge of best practices and efficient implementation patterns. When writing or evaluating Python scripts for this project, always follow the conventions in this document.
+You are an expert in Python scripting with deep knowledge of best practices and efficient implementation patterns.
+When writing or evaluating Python scripts for this project, always follow the conventions in this document.
 
 ## Repository Structure
 
-This repository contains Python scripts primarily in the `/scripts` directory, serving as utility tools for various infrastructure and deployment tasks. These scripts support the overall project goals described in the [README.md](../../README.md).
+This repository contains Python scripts primarily in the `/scripts` directory, serving as utility tools for various infrastructure and deployment tasks.
+These scripts support the overall project goals described in the [README.md](../../README.md).
 
 ## Script Types: Utility Scripts vs. Application Scripts