feat: New environment variable obfuscation functionality

[matglas]

What this PR does / why we need it

Implementing obfuscation of environment variables. Capturing secret values like tokens and api keys is a security risk and attestation should not hold that information. Although it is important to know which values are used to adjust the behavior for scripts.

Which issue(s) this PR fixes (optional)

Fixes:

• [Feat]: Extensibility for the environment attestor #275

Acceptance Criteria Met

• Docs changes if needed
• Testing changes if needed
• All workflow checks passing (automatically enforced)
• All review conversations resolved (automatically enforced)
• DCO Sign-off

Special notes for your reviewer:

The default list can be extended with more variable that should be hidden. This can be extended in the cli with flags that add more items to the list. I think that the default obfuscation list does not need to be 'overridden' but only needs extension.

It also obfuscates based on glob. So if an key would be added like REASON_* it will pick up all REASON_TEST etc. variables. There are a few glob keys that are added by default.

*_TOKEN
SECRET_*
*_API_KEY
*_PASSWORD
*_JWT

Final solution

New flags:

• --env-filter-sensitive-vars
• --env-disable-default-sensitive-vars
• --env-add-sensitive-key can be repeated multiple times.
• --env-exclude-sensitive-key can be repeated multiple times.

Blocklist is now filter list. It describes better what it does.
Obfuscation is the default behavior.
Variables can be obfuscated or filtered based on glob patterns e.g. *SECRET

There is a default list of globs which is quite restrictive by default allow users to explicitly use exclude-sensitive-key to exclude it from filtering or obfuscation.

*TOKEN*
*SECRET*
*API_KEY*
*PASSWORD*
*JWT*

[matglas]

@jkjell would you be able to do a review?

[matglas]

With this latest change I go from default blocking behavior to default obfuscate behavior. And the default list
is the same for blocking and obfuscating. Also the blocking will also use glob patterns now.

Introducing several arguments:

--attestor-environment-block-sensitive-vars
Switch from obfuscate to blocking variables which removes them from the output completely.

--attestor-environment-disable-default-sensitive-vars
Disable the default list of sensitive vars and only use the items mentioned by --attestor-environment-sensitive-key.

--attestor-environment-sensitive-key can be repeated
Add keys to the list of sensitive environment keys.

I believe that as we are still in the v0.x.y versions it should not be a problem to change the default
behavior from blocking to obfuscating. The variables will still not be exposed and switching to blocking
is still possible.

@mikhailswift I need a bit of help on the --attestor-environment-sensitive-key and it looks like you worked on the registry.StringSliceConfigOption. But its not capturing the values correctly somehow. Are you able to help me a bit here?

[matglas]

Forget the part about the registry.StringSliceConfigOption. I had a misconfiguration in my debug setup. But I did found another missing part after that 😄 . The function works now properly. It only needs more work on the tests.

[matglas]

My last changes fixed the tests. I also renamed block to filter. Which seems to me that it is a much more logical.

[matglas]

It looks like we have a challenge with the linux tracing. It also contains all the env vars in the proc info when used. So the behavior needs to pass the same flags there too.

[kairoaraujo]

LGTM.
Waiting for others feedback

[matglas]

@ChaosInTheCRD and I had a conversation on Slack about the implementation because there is a 'hidden' use of Environment inside the commandrun attestor. It used to use the FilterEnvironmentArray but the behavior is completely changing because of the new flags. Because of this commandrun can not use it correctly anymore.

We discussed that is might be best to introduce the current new flags not on the environment attestor level, but move them to the global flags. This way the flags can be used both in Environment and Commandrun Attestor.

Also the filtering and obfuscation methods need to made generic for. So not specifically as part of the attestors.

@kairoaraujo @ChaosInTheCRD if we agree on this I'll implement the change that way. A 👍 is enough.

[axi92]

Nice to see that this gets implemented! 👍
I asked about that in a discussion in-toto/witness#510
I love to see the support for bamboo from Atlassian too.
Since the original list was from a separate repo I posted an issue there Puliczek/awesome-list-of-secrets-in-environment-variables#12. Maybe it could be implemented here as well?

(PS: Is that not more a feature than a chore pr?)

[matglas]

The purpose of the obfuscation is to have a general list of sensitive keys. And they cover the aspect of keys that are matched based on glob. The current that is captured by glob is already covering a lot. Also we do not want to obfuscate all bamboo values I believe.

With the additional flags you can anything that is not covered by these 'sane defaults'. If there is something that is really missing in the sane defaults we could of course add that. See the list here attestation/environment/sensitive_env_vars.go.

[axi92]

Yes, obfuscating all bamboo variables would not make sense. With the list https://github.com/matglas/go-witness/blob/ecacd4ec0ba7a4a12664d3e5d75eedbd433bb9f2/attestation/environment/sensitive_env_vars.go#L24-L28 of defaults are those case sensitive?

Secret and password are already covered right now.
But sshKey and passphrase is missing, if an env variable contains one of those two, they are obfuscated per default on bamboo.

[matglas]

... are those case sensitive?

Yes they are.

Secret and password are already covered right now. But sshKey and passphrase is missing, if an env variable contains one of those two, they are obfuscated per default on bamboo.

These are valuable additions indeed. I'll include them.

[ChaosInTheCRD]

Just as a sanity check, we're ensuring backwards compatibility here right? Again, been out of the loop on this last week or two but just asking the questions to ensure that this was all considered 😄

[matglas]

Yes backward compatibility on the tracing. But with the side note that we now do not filter anymore but use configuration of the EnvironmentCapturer

[matglas]

If you don't see a problem with that we can resolve this conversation.