Implementation of FinOps Toolkit / Cost Reporting

[AErmie]

As part of achieving cost optimization, cost reporting, and financial responsibility, we need to look into if the FinOps Toolkit is applicable to implement.

Additionally, there are 2 FinOps Toolkit implementation options: FinOps Hub versus individual Power BI reports and Azure Workbooks.

Note

The automation of deploying the FinOps Toolkit currently is only supported with PowerShell or Bicep. Terraform is not supported / provided.

UPDATE
Per the following GitHub Issue (Build a Terraform Module for FinOps Hub), there is a Terraform module in development.

Acceptance Criteria

Generated by Zenhub AI

• Scenario: Implementation of FinOps Toolkit
• Given the FinOps Toolkit is applicable for cost reporting and optimization
• When implementing the FinOps Toolkit using either FinOps Hub or individual Power BI reports and Azure Workbooks
• Then the implementation should provide cost reporting, optimization, and financial responsibility capabilities
• Scenario: Automation of deploying the FinOps Toolkit
• Given the automation of deploying the FinOps Toolkit is supported with PowerShell or Bicep
• When using PowerShell or Bicep to deploy the FinOps Toolkit
• Then the deployment should be successful and provide cost reporting, optimization, and financial responsibility capabilities

[AErmie]

FinOps Toolkit Deployment Notes

FinOps Hub

• The only deployment options are ARM or Bicep, no Terraform support (yet)
  • There is an open GitHub Issue (Build a Terraform Module for FinOps Hub) that is in progress, but it has few updates, and the milestone keeps being pushed
• While attempting to deploy via the Azure Portal ARM option, the deployment partially failed because the KeyVault configuration violates the following Policies:
  • Azure Key Vault should use RBAC permission model
  • Key vaults should have deletion protection enabled

Note

Created a Policy exception within the Enforce recommended guardrails for Azure Key Vault policy initiative. This exception was targeted at the specific Resource Group (ie. bcgov-managed-lz-forge-finops ), and only for the 2 policies mentioned within the initiative.

[!TIP] TO DO
Opened the following GitHub Issue: Update Key Vault to Support RBAC Permissions and Delete Protection

PowerBI Reports

• Backfilling data needs to be run month-by-month (ie. StartDate and EndDate need to be in the same month, cannot span multiple months)

Cost Summary

• The START HERE in the PowerBI report fails to load properly
  • Reference the screenshots on the PowerBI - How to setup page for guidance
    • Note: In the screenshot the instructions say to copy the HubUrlForPowerBI value, but there is no output property with this label. There is, however, a storageUrlForPowerBI property.
    • This value is actually the Azure Storage's Data Lake Storage endpoint, with /ingestion appended
• PowerBI report shows the error "Access to the resource is forbidden" for the CostDetails and Prices models
  • NOTE: Using the Storage Account SAS approach worked
  • UPDATE: Another attempt worked without using the SAS

[!TIP] TO DO
Opened the following GitHub Issue: [Documentation] Storage Blob Data Reader Role Required to Resolve Access to the resource is forbidden Error

Update: According to the following GitHub Issues (Error to Connect the report, Access to the resource is forbidden when connecting to stroage account with powerBI), the Storage Blob Data Reader role is required, even if you're a Reader, Contributor, or Owner on the storage account. Added this permission and reset the credentials, and the report loaded the data without error.

Governance

• The START HERE in the PowerBI report fails to load properly
  • Error: Access to the resource is forbidden.

    • HubScopes, HubSettings, SqlDatabases

    • Checking the Tables, it shows "Information is required about data privacy". Presented with an option to set the Privacy Level. For more information, refer to the PowerBI Desktop - Privacy Levels documentation.

      • Set to Organizational for the Storage endpoint

    • Under Transform Data > Data Source Settings ... set Privacy Level to Organizational for the Azure Resource Graph data source

      • Still encountered errors concerning the HubScopes, HubSettings, and SqlDatabases
      

[!TIP] TO DO
Opened the following GitHub Issue: [Documentation] Storage Blob Data Reader Role Required to Resolve Access to the resource is forbidden Error

Rate Optimization

• The START HERE in the PowerBI report fails to load properly
• Everything loads, though there doesn't seem to be any data populating (probably since this report is about reservations, savings plans, and Azure Hybrid Benefit)

Workload Optimization

• The START HERE in the PowerBI report fails to load properly
  • Error: Access to the resource is forbidden.
    • HubScopes, HubSettings, SqlDatabases

[!TIP] TO DO
Opened the following GitHub Issue: [Documentation] Storage Blob Data Reader Role Required to Resolve Access to the resource is forbidden Error

Data Ingestion

• The START HERE in the PowerBI report fails to load properly
  • Error: Access to the resource is forbidden.
    • HubScopes, HubSettings, SqlDatabases

[!TIP] TO DO
Opened the following GitHub Issue: [Documentation] Storage Blob Data Reader Role Required to Resolve Access to the resource is forbidden Error

FinOps Workbooks

• The only deployment options are ARM or Bicep, no Terraform support yet

Optimization

• No issues with loading data into Workbook

Governance

• No issues with loading data into Workbook

[AErmie]

Manually applied the "Inherit a tag from the subscription" policy in FORGE to the Landing Zone root level.

Note

Reference documentation:

• Assign policy definitions for tag compliance
• InheritTag_AddOrReplace_FromSubscription.json

Within that policy assignment, we can specify exactly which tag(s) we want it to inherit (it's not an all-or-none approach). So I tested with just the account_coding tag.

Since there were existing resources, I had to manually created/trigger a Remediation Task (as the automatic one didn't work for some reason). The task ran fairly quickly, and as you can see, remediated appropriately The failed resources are VM extensions, as the VM apparently needs to be running for the tag to be applied (to the extension, not the VM itself).

[AErmie]

Checking the Cost Management export the following day (after applying the policy), the x_TagsDictionary column still did not show/list the account_coding tag.

However, in the Tags column (within the same table), it does show the account_coding tag being captured (and thus available).

Modified the PromotedTags step and added account_coding, so the tag would be automatically extracted into its own column, which makes it easier to use as a filter.

Now that tag/filed can be added to the PowerBI report as a custom filter.

[AErmie]

Additionally, now that this tag is being inherited, it means it because an optional filter in the Optimization, and Governance Workbooks.

Further, this is now also available in the Cost Management Group By value list.

[AErmie]

To facilitate long-term operations and use of the FinOps Toolkit, we want to use the Configure managed exports method in LIVE. To do this effectively, we need to configure the Managed Identity (used to generate the exports), with Enterprise or Department Reader permissions at the Enterprise Agreement level.

This will then simplify configuring the scope in the settings.json file to the EA department level, versus having to list each Subscription (and update that scope list every time a new Subscription is created).

"scopes": [
  {
    "scope": "/providers/Microsoft.Billing/billingAccounts/1234567/departments/56789"
  }
]

We will recommend that a separate EA Department be created for the Public Cloud Platform team's Azure implementation, before implementing the FinOps Toolkit in LIVE.

[ThibaultBC]

Blocked until Enterprise Agreement access is granted

[AErmie]

ACM Cost Allocation Rules

After meeting with Jonathan McCaig (who now has access to create ACM Cost Allocation Rules), it seems this feature is not mature enough.

When using a Cost Allocation Rule, it will transfer/drain all costs from the source subscription, into the target subscription(s). This will cause challenges for us, as some costs for shared services (ie. Firewall) are being handled though a specific team. Therefore we cannot use Cost Allocation Rules until they provide more granularity of the source.

EA Department

There is already an EA department (OCIO) that encompasses the Public Cloud Azure subscriptions. We will use this as our target department for managed exports.

Follow-Up

Warren suggests that we have an Entra ID Security Group created with EA Department Reader permissions, so that if/when we need to re-deploy the FinOps Toolkit, we can add the Managed Identity to the appropriate Group, and grant it the access required.

Update

Enterprise Agreement roles can only be assigned to Users, and Service Principals, not Groups. Therefore we will have to wait to deploy the FinOps Toolkit in LIVE.

Additionally, ideally, we could use an existing Service Principal (ie. the one that has the Subscription Creator role), and just add the Department Reader permissions to that. However, currently the FinOps Toolkit does not support using a pre-created Managed Identity. There is a GitHub Issue ([Hubs] Use of pre created Entra ID SPN to deploy the resources) about this feature request.

[AErmie]

Cleaned up existing resources, and deployed the latest version of the FinOps Toolkit (v0.7).

[NOTE]
There are several new deployment configuration options! Such as:

• Azure Data Explorer
• Infrastructure Encryption
• Private Networking

Unfortunately, although selecting the "public" networking option, the resources for private networking were deployed, including VNet, NSG, Private Endpoints, Private DNS Zones!

Created the following GitHub Issue: FinOps Toolkit v0.7 Deploys Private Networking Resources When Public Networking Option Selected

Re-Deploy

Deleted the entire Resource Group, re-created it (and re-applied the Azure Policy exception), and attempted to re-deploy v0.7 of the FinOps Toolkit.

Re-deployment failed due to the Key Vault name already existed (due to purge/delete protection). Also, the private networking resources were also created again, despite selecting the public network option.

[AErmie]

Update

Per the following GitHub Issue (FinOps Toolkit v0.7 Deploys Private Networking Resources When Public Networking Option Selected), this is "by design". Apparently, "Our intent is for private endpoints to be the only option going forward."

Limitations

The following is the limitations of using the FinOps Toolkit within our CAF-structured environment. Until these are resolved/addressed, we cannot deploy the FOT:

• No support for using an existing VNet
• No support for using existing Private DNS Zones
• Deployed KeyVault does not support RBAC access control
• It does not support using an existing Managed Identity

[wrnu]

We have decided that the FinOps tool kit needs to mature a bit more before we use it. We also have a meeting with the developer next week to express our needs and experiences

[AErmie]

FinOps Toolkit Implementation Planning

A few notes to consider:

• Since we plan to connect it to the EA Department, then we should deploy it to the LIVE Management subscription
• Since it will contain LIVE cost data
  • We should include deploying an Azure Data Explorer (used for scalable data analytics)
    • Cost starts at $120 for a single-node cluster + $10 per million in monitored spend
      • Monitored spend refers to how much cost data is stored based on desired retention. For instance, $1 million per month in spend for 13 months is $13 million in monitored spend. The basic deployment with Data Explorer would be $250/mo - $120/mo for a single node cluster plus $10 times 13 for Data Factory and storage costs.
  • We should chose ZRS storage for high availability/redundancy
  • Data retention by default is 13 months
    • Should this be longer? Are there specific data retention requirements for BC Gov?
• We will need to allocate a /26 address space for the VNet that is deployed (which we will have to manually attach to the vWAN Hub)
• For Executive use of PowerBI reports and dashboard, we will also need to deploy a PowerBI Data Gateway

[AErmie]

FinOps Toolkit Networking Meeting Notes

• deployment pieces around existing private networking
• The Data Explorer and Script subnets have to be allocated to each respective component (ie. ADX Kusto cluster, and Azure Container Instances for the deployment scripts). We cannot use a shared subnet (ie. the subnet for Private Endpoints)
• The deployment scripts are used to securely connect to the storage and the config file, and securely connect to the data explorer cluster
• Azure Cost Management Exports write to the Data Lake Storage Account, this is why the storage account firewall is configured with default deny with a bypass for trusted Azure Services
• EventGrid is triggered when ACM writes to the Storage, and goes to Data Factory and triggers the ETL jobs
• When attaching the FTK VNet to the Hub, you need to add 2 DNS records for each of the private endpoints (ie. ADLS, Kusto)
  • privatelink.dfs.core.windows.net (ADLS) and privatelink.region.core.windows.net (Kusto)
• 4x Private DNS Zones are deployed and bound to the VNet/subnets that the FTK deploys. It's all internal, and needed for the deployment scripts to work
  • Name resolution outside of the FTK is not needed (you need access in though)
• For a private network implementation, to access the data (for PowerBI reports via PowerBI.com), you need a PowerBI Data Gateway that can route to the Storage Account
  • PowerBI can deploy a managed Data Gateway into a VNet that you specify, or you can use a self-hosted where you deploy the Data Gateway onto a VM (reverse proxy to the PowerBI.com service)
  • Alternatively, you can use the Kusto-based reports for the same thing, or a managed private link for Microsoft Fabric
• Version 0.8.0 of the FTK should have security enhancements based on a threat model that was conducted by Microsoft internally
• Work-around for using an existing Managed Identity, would be to pre-create the Identity within the Resource Group where the FTK is created; then Bicep (once the deployment accepts custom naming), will see that it already exists
• The ADX implementation is the future for the FTK, due to scalability issues with storage-based reports

Architecture Diagram