gitlab inventory group_vars not loading satellite groups

[mboeker]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

This is my gitlab setup:
inventory/group_vars/all.yml
inventory/group_vars/home.yml

In Satellite, I have a group called 'home'
This group contains server 'server1.home.local'

I created a playbook that outputs all groups a server is in, and outputs a variable defined in all.yml, and a variable defined in home.yml.

TASK [Gathering Facts] *********************************************************
task path: /runner/project/playbooks/_testing.yml:6
ok: [server1.home.local]

TASK [Check connection] *************
task path: /runner/project/playbooks/_testing.yml:10
ok: [server1.home.local] => {"changed": false, "ping": "pong"}

TASK [Show groups] *************************************************************
task path: /runner/project/playbooks/_testing.yml:14
ok: [server1.home.local] => (item=home) => {
    "msg": "Found group: home"
}

TASK [debug all vars] *******************************************************
task path: /runner/project/playbooks/_testing.yml:19
ok: [server1.home.local] => {
    "msg": "Timezone: Europe/Vienna"
}

TASK [debug home vars] *******************************************************
task path: /runner/project/playbooks/_testing.yml:24
fatal: [server1.home.local]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ntp_server' is undefined. 'ntp_server' is undefined\n\nThe error appears to be in '/runner/project/playbooks/_testing.yml': line 24, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: debug home vars\n    ^ here\n"}
...ignoring

PLAY RECAP *********************************************************************
server1.home.local          : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=1   

This host is not in the gitlab inventory, it is only in the Satellite inventory.
Timezone is defined in all.yml, ntp_server is defined in home.yml.
You can see it loads group_vars/all.yml, but it is not loading group_vars/home.yml.

AWX version

24.6.1

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

RHEL 9.6

Web browser

Edge

Steps to reproduce

Deploy AWX
Link it to a gitlab project with various group_vars
Link it to a Satellite with hosts and group names matching group_vars
Define things in group_vars/all.yml
Define things in group_vars/GROUPNAME.yml

Vars from all.yml are loaded, GROUPNAME.yml are not loaded.

Expected results

AWX should load the variables corresponding to the group names the server identifies (in my example "home")

Actual results

Only group_vars/all.yml is loaded, group_vars/GROUPNAME.yml is not read

Additional information

No response

[thedoubl3j]

@mboeker thanks for reporting this issue, looking over it I have a few questions.

• are you only loading variables via files? nothing configured in the satellite plugin or anything?
• looking at your playbook output, I think there might be a potential variable precedence issue because how group vars work in an inventory is different from a playbook (docs

if this playbook is being run in awx, could you turn the verbosity up on the JT and paste the output here? (cleanse of all personal data).

awx var precedence if you prefer visual.

my gut instinct is that this looks like a variable issue and not a satellite plugin issue/awx API issue but happy to dig a bit more to confirm.

[mboeker]

Hi @thedoubl3j
Thank you for your input!
I don't know about a Satellite plugin, I just use Satellite as an inventory source.
I have been using Ansible for a long time, AWX only a short while, but this exact config works fine in Ansible CLI. If I put the host in the ansible file-based inventory under the group 'home' then it loads the variables from home.yml correctly.
In the playbook output it actually appears the same, 'home' is listed in the groups regardless of Satellite or Ansible Inventory, but if it's defined in the Satellite Inventory then it does not load home.yml.
I verified this by adding the host to the ansible inventory, then all vars were defined. As soon as I removed it again the group var was no longer defined.
I also cranked loglevel up to 5, aside from ten pages of connection info before the ping it's identical.
Here is the playbook output if my static ansible inventory includes '[home]\nserver1.home.local' (pseudo text):

TASK [Check connection] *************
task path: /runner/project/playbooks/_testing.yml:10
ok: [server1.home.local] => {"changed": false, "ping": "pong"}

TASK [Show groups] *************************************************************
task path: /runner/project/playbooks/_testing.yml:14
ok: [server1.home.local] => (item=home) => {
    "msg": "Found group: home"
}

TASK [debug all vars] *******************************************************
task path: /runner/project/playbooks/_testing.yml:19
ok: [server1.home.local] => {
    "msg": "Timezone: Europe/Vienna"
}

TASK [debug home vars] *******************************************************
task path: /runner/project/playbooks/_testing.yml:24
ok: [server1.home.local] => {
    "msg": "ntp_server: 10.0.0.2"
}

Annoyingly, even after removing the host from the ansible inventory, AWX remembered the group variable. I played around with the "Overwrite" and "Overwrite variables" options in the combined inventory sources to clear this out, and I'm wondering if that's the source of the issues. I now have both inventory sources set to "Overwrite" ON and "Overwrite variables" OFF -- it's now correctly aware that the host is only defined in Satellite, and it is loading all.yml, but not the groups (so same as before).

[mboeker]

UPDATE: I deleted my last comment because the results were bogus, turns out I did have that server in my gitlab inventory.
I've been sanitizing my output too much, I am not actually using the same server for all of these tests.

I tried using a Constructed Inventory using 1. Satellite and 2. Gitlab, I had no luck at all with that

Now my configuration is as follows:
Single inventory, two sources
Gitlab: overwrite OFF, overwrite vars ON
Satellite: overwrite OFF, overwrite vars OFF

In my gitlab Inventory I have three vars defined:
group_vars/all.yml: ansible_inv_all_var: present
group_vars/home.yml: ansible_inv_group_var: present
host_vars/v8718t.home.local: ansible_inv_host_var: present

The playbook tasks are:
List all groups
Output ansible_inv_all_var
Output ansible_inv_group_var
Output ansible_inv_host_var

The playbook output:

ok: [v8718t.home.local] => (item=home) => {
    "msg": "Found group: home"
}

TASK [debug ansible_inv_all_var] ***********************************************
ok: [v8718t.home.local] => {
    "msg": "ansible_inv_all_var: present"
}

TASK [debug ansible_inv_group_var] *********************************************
ok: [v8718t.home.local] => {
    "msg": "ansible_inv_group_var: present"
}

TASK [debug ansible_inv_host_var] **********************************************
fatal: [v8718t.home.local]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible_inv_host_var' is undefined. 'ansible_inv_host_var' is undefined\n\nThe error appears to be in '/runner/project/playbooks/_testing.yml': line 18, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n  - name: debug ansible_inv_host_var\n    ^ here\n"}
...ignoring

This host is ONLY listed in Satellite.

[mboeker]

UPDATE:
I changed the ansible_inv_group_var value from "present" to "from_group" and the task output did not change, it still just says "present" which means AWX is caching that somehow, presumably from a previous test with a different configuration.

I also tried putting this task in there:

  - name: Load group vars
    include_vars: "{{ playbook_dir }}/../inventory/group_vars/{{ item }}.yml"
    loop: "{{ group_names }}"
    ignore_errors: yes

The task says that it successfully loaded home.yml, but the ansible_inv_group_var value is still "present"

[mboeker]

To ensure I am not dealing with cached variables I have updated the playbook.
Now the different yaml files have var2 defined. I am still using the Load group vars task. It reports successfully loading the files, but it actually only loads the host_vars, not the group vars.
Output summarized:

list groups -> OK (lists home)
output host_var2 -> FAIL
output group_var2 -> FAIL
output all_var2 -> OK
load var files -> OK (loads host, group, and all)
output host_var2 -> Now OK!
output group_var2 -> Still FAIL
output all_var2 -> Still OK

[thedoubl3j]

@mboeker

I don't know about a Satellite plugin, I just use Satellite as an inventory source.

plugin == inventory source (I misused the term here, the inventory source is backed by a satellite dynamic inventory plugin)

And sweet, I was just about to ask about running things on the CLI. Ok, so awx is doing something wonky. removing triage label as I think we have narrowed in on the issue. I will need to see if this can be reproducible on devel as well. not sure when I can get around to that.

[mboeker]

If there's anything I can do to help please let me know.
I'm still trying to come up with a workaround.
Going to no longer test for (or refer to) all.yml since that is always loaded.
During one of my many tests the playbook did actually load both the group and host variables, unfortunately I didn't record exactly what I had configured that time.
What I do know is this:

- name: test inventory vars
  hosts: all
  vars_files:
    - /runner/project/inventory/group_vars/home.yml

-> This one DOES load the variables in home.yml!

  - name: Load group vars
    include_vars: "{{ playbook_dir }}/../inventory/group_vars/{{ item }}.yml"
    loop: "{{ group_names }}"
    ignore_errors: yes

-> This one lists home, and it says it loaded home.yml, but it DOES NOT actually load the variables.

[mboeker]

Also, this block makes AWX load the host_vars:

  - name: Load host vars
    include_vars:
      file: "{{ playbook_dir }}/../inventory/host_vars/{{ inventory_hostname }}.yml"
    ignore_errors: yes

However still don't have a way to get it to load group_vars.

[mboeker]

Another update to group_vars:
When I run these two tasks:

  - name: Load group vars
    include_vars:
      file: "{{ playbook_dir }}/../inventory/group_vars/{{ item }}.yml"
    loop: "{{ group_names }}"
    register: group_vars_result
    ignore_errors: yes

  - name: Show group vars load results
    debug:
      var: group_vars_result
-   
  - name: Check if group variable exists
    debug:
      msg: "ansible_inv_group_var2 is: {{ ansible_inv_group_var2 | default('UNDEFINED') }}"

In that group_vars_result, I do see my variable!

    - ansible_included_var_files:
        - /runner/project/inventory/group_vars/home.yml
      ansible_facts:
        ansible_inv_group_var2: from_home.yml
      failed: false
      changed: false
      item: home
      ansible_loop_var: item

However the very next task checking if ansible_inv_group_var2 exists says it is UNDEFINED.

[mboeker]

I found a workaround that will make group_vars available:

  - name: Load group vars
    include_vars:
      file: "{{ playbook_dir }}/../inventory/group_vars/{{ item }}.yml"
    loop: "{{ group_names }}"
    register: group_vars_result
    ignore_errors: yes

  - name: Apply all loaded group vars as facts
    set_fact:
      "{{ fact_item.key }}": "{{ fact_item.value }}"
    loop: >-
      {{
        group_vars_result.results
        | selectattr('failed', 'defined')
        | rejectattr('failed')
        | selectattr('ansible_facts', 'defined')
        | map(attribute='ansible_facts')
        | map('dict2items')
        | flatten
      }}
    loop_control:
      loop_var: fact_item

This was a suggestion from Claude based on the problem. After that second task, ansible knows that ansible_inv_group_var2: from_home.yml

[mboeker]

Hello! I have a big update.
I switched from the Satellite plugin to the Foreman plugin, and now all group variables are being loaded correctly.
I actually did this because the Satellite Plugin was not giving me other variables from Satellite that I wanted, I did not see host parameters, and AI suggested that I try the Foreman plugin instead.
No workaround needed, I created a group called home2 in Satellite, and added a Satellite host parameter "AWX_TEST1: present" and in my gitlab project I created home2.yml and in there I added "AWX_TEST2: present" and when I run a playbook/template using the gitlab+foreman inventory, both AWX_TEST1 and AWX_TEST2 are correctly defined.