Skip to content

Heat fails to authenticate with Keystone when using self-signed TLS certs. #1157

@vish6760

Description

@vish6760

Describe the bug
When deploying heat with openstack-helm, the default manifests configure the heat service to use publicURL and public interface when authenticating with Keystone. This works fine with Let’s Encrypt or other publicly trusted CAs, but fails when the Keystone public endpoint is using a self-signed TLS certificate. Since pods don’t mount the CA, SSL validation fails with SSLError, and this results in 500 Internal Server Error from Heat client.

To Reproduce
Steps to reproduce the behavior:

  1. Updated the required override config for heat in /etc/genestack/helm-configs/heat/heat-helm-overrides.yaml.
  2. Run the Heat deployment Script /opt/genestack/bin/install-heat.sh
  3. Check that all the heat pods are running.
  4. Run debug while validating the Heat service.

Expected behavior
Heat should be able to authenticate with Keystone even when Keystone is using a self-signed cert.

Server (please complete the following information):

  • OS: Ubuntu 22.04

Additional context
Observed Behavior:

  1. All heat pods are running.
  2. Run the debug while validating the heat service.
# kubectl --namespace openstack exec -ti openstack-admin-client -- \
  openstack --os-interface internal orchestration service list --debug
<truncated-output>
Instantiating orchestration client: <class 'heatclient.v1.client.Client'>
REQ: curl -g -i -X GET http://heat-api.openstack.svc.cluster.local:8004/v1/e22812fc202b4f2abf5ddf39757cd3f0/services -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-heatclient" -H "X-Auth-Token: {SHA256}f0820602a9575d0f610cc5c8feacdd9dec641a8a542370730906e6ba5a64c991" -H "X-Region-Name: dev"
Starting new HTTP connection (1): heat-api.openstack.svc.cluster.local:8004
http://heat-api.openstack.svc.cluster.local:8004 "GET /v1/e22812fc202b4f2abf5ddf39757cd3f0/services HTTP/1.1" 500 198
RESP: [500] Connection: close Content-Length: 198 Content-Type: application/json x-openstack-request-id: req-7d96b8c2-05f7-4054-bd4d-6b8f02ec182b
RESP BODY: {"code": 500, "title": "Internal Server Error", "explanation": "The server has either erred or is incapable of performing the requested operation.", "error": {"type": "SSLError", "traceback": null}}
GET call to orchestration for http://heat-api.openstack.svc.cluster.local:8004/v1/e22812fc202b4f2abf5ddf39757cd3f0/services used request id req-7d96b8c2-05f7-4054-bd4d-6b8f02ec182b
ERROR: Internal Error
Traceback (most recent call last):
  File "/var/lib/openstack/lib/python3.12/site-packages/cliff/app.py", line 410, in run_subcommand
    result = cmd.run(parsed_args)
             ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/osc_lib/command/command.py", line 38, in run
    return super(Command, self).run(parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/cliff/display.py", line 117, in run
    column_names, data = self.take_action(parsed_args)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/heatclient/osc/v1/service.py", line 37, in take_action
    services = heat_client.services.list()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/heatclient/v1/services.py", line 33, in list
    return self._list(url, "services")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/heatclient/common/base.py", line 113, in _list
    body = self.client.get(url).json()
           ^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/keystoneauth1/adapter.py", line 393, in get
    return self.request(url, 'GET', **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/openstack/lib/python3.12/site-packages/heatclient/common/http.py", line 322, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPInternalServerError: ERROR: Internal Error
clean_up ListService: ERROR: Internal Error
END return value: 1
command terminated with exit code 1
  1. Exec into the heat-api pod and check /etc/heat/heat.conf. Under the [keystone_authtoken] section, notice that the interface is set to public.
  2. I notice that the endpoint_type is set to publicURL and interface is set to public in the default heat overrides https://github.com/rackerlabs/genestack/blob/main/base-helm-configs/heat/heat-helm-overrides.yaml#L93
conf:
  heat:
    DEFAULT:
      num_engine_workers: 2
      server_keystone_endpoint_type: public
    clients_keystone:
      endpoint_type: publicURL
    ----
    keystone_authtoken:
      auth_type: password
      auth_version: v3
      interface: public
  1. In my lab, I fixed the issue by overriding these values: endpoint_type to internal and interface to internal, and reinstalling the heat.
(genestack) root@controller-01:~# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack --os-interface internal orchestration service list
+------------------------------+-------------+--------------------------------------+-------------+--------+----------------------------+--------+
| Hostname                     | Binary      | Engine ID                            | Host        | Topic  | Updated At                 | Status |
+------------------------------+-------------+--------------------------------------+-------------+--------+----------------------------+--------+
| heat-engine-6764f5d475-nhtv4 | heat-engine | 30b0b08b-d9b9-49a4-9ed3-47aab0575f7d | heat-engine | engine | 2025-08-28T11:45:07.000000 | up     |
| heat-engine-6764f5d475-nhtv4 | heat-engine | 631e6d92-d66f-4223-b57e-cef47c0910d8 | heat-engine | engine | 2025-08-28T11:45:07.000000 | up     |
| heat-engine-6764f5d475-znbcz | heat-engine | 38446092-28ee-40f9-8216-003436381178 | heat-engine | engine | 2025-08-28T11:45:06.000000 | up     |
| heat-engine-6764f5d475-znbcz | heat-engine | 67f49d6e-6a79-47a7-8b52-d25e40ef771e | heat-engine | engine | 2025-08-28T11:45:06.000000 | up     |
+------------------------------+-------------+--------------------------------------+-------------+--------+----------------------------+--------+

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions