Use of __env__ in file_roots master configuration declaration

[fs-nicola-worthington]

Hi everyone. I'm hoping you can offer some guidance regarding the use of __env__ in the file_roots: directive in the Salt master configuration?

I am using Salt 3007.1 on the latest x86_64 Amazon Linux 2023 directly from the new Broadcom RPM repositories:

# rpm -qa salt\*
salt-3007.1-0.x86_64
salt-master-3007.1-0.x86_64
salt-cloud-3007.1-0.x86_64
salt-syndic-3007.1-0.x86_64
salt-api-3007.1-0.x86_64
salt-ssh-3007.1-0.x86_64
salt-minion-3007.1-0.x86_64

The documentation I have been referring to is:

• new feature: add saltenv directory to file_roots and pillar_roots for dynamic environments #55747
• https://docs.saltproject.io/en/latest/ref/configuration/master.html#roots-master-s-local-file-server
• https://docs.saltproject.io/en/latest/topics/pillar/index.html#dynamic-pillar-environments

We have a number of different teams and engineering staff who need to be able to test their branches of Salt state and pillar changes prior to merging into our Git SCM that contains all our Salt states. Previously we had uses the Salt native GitFS functionality, but due to the enormous size of our repositories and large number of branches, the performance was unusable (the GitFS implementation was an order of magnitude slower than just pointing to plain directories on the local filesystem). For that reason, we now have a GitLab CI job that just pushes the new branch changes to the Salt master filesystem. This works well, and fast.

In the past we were using Salt 3004, which did not support the use of __env__ in the file_roots: or pillar_roots: directives. This means that we has to list every single saltenv and pillarenv combination in our Salt master configuration file, and all the associated different filesystem paths.

We have read that as of Salt 3005, use of __env__ became supported in these directives.

However, we have found that it only seems to reliably work for the pillar_roots: directive.

When we use it in the file_roots:, we receive errors complaining of non-existant saltenvs.
For example salt-call state.apply test=true pillarenv=team-sloth-testing saltenv=team-sloth-testing would result in an error like Nonexistent saltenv 'team-sloth-testing' found in include of 'defaults.centos' within SLS 'team-sloth-testing:defaults'. ...but as soon as we put the hard-coded environment name and path into the file_roots: directive like we used to, it starts working fine again.

file_roots:
  base:
    - /srv/salt
    - /srv/salt-branches/base/states
  __env__:
    - /srv/salt
    - /srv/salt-branches/__env__/states
# For some reason the above __env__ declaration does not appear to work in the manner described in the documentation. It results in many errors like this:
# Nonexistent saltenv 'team-sloth-testing' found in include of 'defaults.centos' within SLS 'team-sloth-testing:defaults'
#  team-sloth-testing:
#    - /srv/salt
#    - /srv/salt-branches/team-sloth-testing/states

pillar_roots:
  base:
    - /srv/pillar
    - /srv/salt-branches/base/pillar
  __env__:
    - /srv/pillar
    - /srv/salt-branches/__env__/pillar

Is there something obviously wrong we are doing here?

Have is misunderstood the documentation?

Very much hoping someone can offer their insight and assit here.

Thanks all!

[fs-nicola-worthington]

A follow up -- this is obviously some failing in our janky Salt configuration. Testing out the most minimal test case does not reproduce the problem -- it works as documented.

For anyone else who was seeing anything similar:

[root@salt-devel ~]# cat /etc/salt/master.d/custom.conf
auto_accept: True
top_file_merging_strategy: same
state_top_saltenv: base
state_verbose: true
state_output: changes

fileserver_backend:
  - roots

file_roots:
  __env__:
    - /test-srv/salt/__env__

pillar_roots:
  __env__:
    - /test-srv/pillar/__env__

[root@salt-devel ~]# find /test-srv/ -type f -exec grep -Hn . '{}' \;
/test-srv/salt/base/top.sls:1:"{{ saltenv }}":
/test-srv/salt/base/top.sls:2:  '*':
/test-srv/salt/base/top.sls:3:    - example
/test-srv/salt/base/example.sls:1:example_wall:
/test-srv/salt/base/example.sls:2:  cmd.run:
/test-srv/salt/base/example.sls:3:    - name: echo "base/example.sls saltenv={{ saltenv }} pillarenv={{ opts['pillarenv'] }} {{ salt['pillar.get']('my_message', 'fallback-value') }}" | wall
/test-srv/salt/base/example.sls:4:    - runas: root
/test-srv/salt/base/example.sls:5:    - shell: /bin/bash
/test-srv/salt/foo/top.sls:1:"{{ saltenv }}":
/test-srv/salt/foo/top.sls:2:  '*':
/test-srv/salt/foo/top.sls:3:    - example
/test-srv/salt/foo/example.sls:1:example_wall:
/test-srv/salt/foo/example.sls:2:  cmd.run:
/test-srv/salt/foo/example.sls:3:    - name: echo "foo/example.sls saltenv={{ saltenv }} pillarenv={{ opts['pillarenv'] }} {{ salt['pillar.get']('my_message', 'fallback-value') }}" | wall
/test-srv/salt/foo/example.sls:4:    - runas: root
/test-srv/salt/foo/example.sls:5:    - shell: /bin/bash
/test-srv/pillar/base/top.sls:1:"{{ saltenv }}":
/test-srv/pillar/base/top.sls:2:  '*':
/test-srv/pillar/base/top.sls:3:    - example
/test-srv/pillar/base/example.sls:1:my_message: "This is the base example pillar"
/test-srv/pillar/foo/top.sls:1:"{{ saltenv }}":
/test-srv/pillar/foo/top.sls:2:  '*':
/test-srv/pillar/foo/top.sls:3:    - example
/test-srv/pillar/foo/example.sls:1:my_message: "This is the foo example pillar"

[root@salt-devel ~]# salt-call state.apply

Broadcast message from root@salt-devel (somewhere) (Thu De

base/example.sls saltenv=base pillarenv=None This is the base example pillar

local:
----------
          ID: example_wall
    Function: cmd.run
        Name: echo "base/example.sls saltenv=base pillarenv=None This is the base example pillar" | wall
      Result: True
     Comment: Command "echo "base/example.sls saltenv=base pillarenv=None This is the base example pillar" | wall" run
     Started: 16:38:08.124912
    Duration: 173.29 ms
     Changes:
              ----------
              pid:
                  181891
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 173.290 ms

[root@salt-devel ~]# salt-call state.apply pillarenv=foo

Broadcast message from root@salt-devel (somewhere) (Thu De

base/example.sls saltenv=base pillarenv=foo This is the foo example pillar

local:
----------
          ID: example_wall
    Function: cmd.run
        Name: echo "base/example.sls saltenv=base pillarenv=foo This is the foo example pillar" | wall
      Result: True
     Comment: Command "echo "base/example.sls saltenv=base pillarenv=foo This is the foo example pillar" | wall" run
     Started: 16:38:20.734755
    Duration: 105.111 ms
     Changes:
              ----------
              pid:
                  182066
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 105.111 ms

[root@salt-devel ~]# salt-call state.apply pillarenv=foo saltenv=foo

Broadcast message from root@salt-devel (somewhere) (Thu De

foo/example.sls saltenv=foo pillarenv=foo This is the foo example pillar

local:
----------
          ID: example_wall
    Function: cmd.run
        Name: echo "foo/example.sls saltenv=foo pillarenv=foo This is the foo example pillar" | wall
      Result: True
     Comment: Command "echo "foo/example.sls saltenv=foo pillarenv=foo This is the foo example pillar" | wall" run
     Started: 16:38:27.567775
    Duration: 107.754 ms
     Changes:
              ----------
              pid:
                  182234
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 107.754 ms

[root@salt-devel ~]# salt-call state.apply pillarenv=base saltenv=foo

Broadcast message from root@salt-devel (somewhere) (Thu De

foo/example.sls saltenv=foo pillarenv=base This is the base example pillar

local:
----------
          ID: example_wall
    Function: cmd.run
        Name: echo "foo/example.sls saltenv=foo pillarenv=base This is the base example pillar" | wall
      Result: True
     Comment: Command "echo "foo/example.sls saltenv=foo pillarenv=base This is the base example pillar" | wall" run
     Started: 16:38:34.967799
    Duration: 113.895 ms
     Changes:
              ----------
              pid:
                  182399
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 113.895 ms

[root@salt-devel ~]# salt-call state.apply saltenv=foo

Broadcast message from root@salt-devel (somewhere) (Thu De

foo/example.sls saltenv=foo pillarenv=None This is the base example pillar

local:
----------
          ID: example_wall
    Function: cmd.run
        Name: echo "foo/example.sls saltenv=foo pillarenv=None This is the base example pillar" | wall
      Result: True
     Comment: Command "echo "foo/example.sls saltenv=foo pillarenv=None This is the base example pillar" | wall" run
     Started: 16:38:43.559109
    Duration: 115.911 ms
     Changes:
              ----------
              pid:
                  182559
              retcode:
                  0
              stderr:
              stdout:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 115.911 ms