Run permissions tests from Foreman core

[ekohl]

Opened to showcase an alternative to theforeman/actions#16.

The idea is that plugins should always run a certain test from core. This relies on:

If both pattern and test_files are used, then the list of test files is the union of the two.

It also starts to depend on db:test:prepare since I saw that at least in foreman_tasks.

[evgeni]

I wish this list (or at least its base) could come from Foreman and not be hard-coded here.

[ekohl]

I wish this list (or at least its base) could come from Foreman and not be hard-coded here.

How do you know what is and isn't relevant from Foreman core? Today in Jenkins we run the full Foreman test suite, which makes it a lot slower but you'd test for regressions.

[evgeni]

I wish this list (or at least its base) could come from Foreman and not be hard-coded here.

How do you know what is and isn't relevant from Foreman core? Today in Jenkins we run the full Foreman test suite, which makes it a lot slower but you'd test for regressions.

I don't think there is a way to "know", but we can offer a "recommended minimum"?
So far, when I've seen plugins explicitly calling core tests, it was always the permissions test. So this "recommended minimum" would be that. And we'd write t.test_files = Foreman::Plugin.recommended_core_tests in the default Rakefile, and people can add + [whatever] if they decide they need more?

[ekohl]

OpenSCAP has defined a specific task for it:
https://github.com/theforeman/foreman_openscap/blob/279fa50e568985b099ce8dc9ab216e82096e91e7/lib/tasks/foreman_openscap_tasks.rake#L94-L107

Now one question is: should recommended_core_tests do something smart or is it just a static list?

[evgeni]

Now one question is: should recommended_core_tests do something smart or is it just a static list?

I think a static list is perfectly fine.
(Unless you already an idea how it could be "smart", but I'd not wait for that if not)

[ekohl]

Just a crazy idea: how about some helper in Foreman itself to define the test tasks? Then allow further configuration via a block.

[evgeni]

Crazy? Maybe.

IMHO one thing we should consider is time. Is this something we want to have soon or we can live without while we figure out the best implementation?