spec file preprocessing on rpm level

[clime]

Hello,

since this change https://pagure.io/fesco/issue/2532 will not likely be accepted, I would like to ask if you were willing to accept a pull request to enable spec file preprocessing directly on rpm level. It would add a third type of macro (except %{} and %()), which would work similarly to %() except that the values (after expansion) would be put verbatim into the spec file that goes into an srpm generated by rpmbuild.

Basically, spec files would be always passed through preproc utility first before they would be handed over for further rpm processing. Enabling/disabling of this feature would be only system-wide, not spec-wide as you need to preprocess a template first before parsing it.

This is a way to bring innovation to rpm ecosystem mainly by:

• enabling automatic version, release, changelog, VCS generation e.g. from git metadata
• enabling dynamic source generation e.g. from a git submodule with mirrored upstream sources
• providing a way for automatic generation of requires and build requires by analyzing upstream sources

The only added dependency would the preproc utility, which I would just rewrite from python to C, it is less than 100 lines of code currently so it won't be a problem.

People basically didn't like that rpmbuild -bs would no longer always work on a spec file within Fedora DistGit after accepting the aforementioned change. Having the preprocessor directly in rpm would solve that problem.

[pmatilai]

A spec will no longer be a spec if such a feature was added. I can see how templating can be useful, but I think this would need to be thought of as a different file type - say, .spec.in rather than .spec to make it clear.

Also just noting that #1241 is related, there's an increasing amount of dynamically generated content in specs.

[clime]

A spec will no longer be a spec if such a feature was added. I can see how templating can be useful, but I think this would need to be thought of as a different file type - say, .spec.in rather than .spec to make it clear.

I think it's mostly matter of preference and compatibility. Additional options (except differentiating file type) that come to mind:

• introduce new spec version, make it possible to declare spec version in spec file by e.g. # rpmspec:2 in the first line
• bump rpm major version with introduction of this feature
• maybe something else

Having filetype separated seems quite clean to me although it depends on whether rpm would be able to process such templates/files by default and the extension would trigger preprocessing or if an extra switch would be needed (likely cumbersome).

[elmarco]

I checked quickly preproc and it seems quite limited to me at this point. It allows basic expansion, but no control flow & conditions.

Also I think {{{ }}} is not the best choice, as it conflicts with variables, something like %{{{{ to expand}}}} wouldn't work as expected.

Also, what does it bring compared to %{lua:...} ? (which could run a bash script if necessary)

(btw, preproc as a name for such tool, not the best idea)

[clime]

I checked quickly preproc and it seems quite limited to me at this point. It allows basic expansion, but no control flow & conditions.

I agree it's limited but so far it was enough to provide dynamically generated specs based on git metadata.

Also I think {{{ }}} is not the best choice, as it conflicts with variables, something like %{{{{ to expand}}}} wouldn't work as expected.

I think this case might be quite hypothetical, i.e. that you would want to work with dynamically named rpm variable? You could do {{{ echo %{$<something>} }}} in that case. But it's also possible to make the tags configurable if that was really a problem.

Also, what does it bring compared to %{lua:...} ? (which could run a bash script if necessary)

%{lua:...} scriplets are evaluated during rpm spec evaluation, here we are talking about an earlier step. If the question is why not use lua language rather than bash for evaluation code inside {{{ }}} tags, then I would say, because bash is simpler to use for wider range of people.

(btw, preproc as a name for such tool, not the best idea)

It is supposed to be a generic text-preprocessing tool, hence preproc. Usage generally is cat file.txt | preproc.

[DemiMarie]

If this is to be integrated into RPM, it needs to have clear semantics and escaping rules, and those rules need to be compatible with those of RPM. Code generators targeting RPM already know to escape % wherever it appears. It should also be opt-in.

[elmarco]

I agree it's limited but so far it was enough to provide dynamically generated specs based on git metadata.

ack

Also I think {{{ }}} is not the best choice, as it conflicts with variables, something like %{{{{ to expand}}}} wouldn't work as expected.

I think this case might be quite hypothetical, i.e. that you would want to work with dynamically named rpm variable? You could do {{{ echo %{$<something>} }}} in that case. But it's also possible to make the tags configurable if that was really a problem.

hmm, ok

Also, what does it bring compared to %{lua:...} ? (which could run a bash script if necessary)

%{lua:...} scriplets are evaluated during rpm spec evaluation, here we are talking about an earlier step. If the question is why not use lua language rather than bash for evaluation code inside {{{ }}} tags, then I would say, because bash is simpler to use for wider range of people.

Could you explain a little bit the difference between the preproc time and the %{} expansion time? Doesn't rpm expand %{} repeatedly until no further expansion can be done?

(If there was template control flow it would make sense, because then you would have delimited blocks that you would want to process first, I think).

(my point about calling bash script from lua remains)

(btw, preproc as a name for such tool, not the best idea)

It is supposed to be a generic text-preprocessing tool, hence preproc. Usage generally is cat file.txt | preproc.

Fair enough. I was actually surprised there was no such tool in the coreutils/bash tool belt.

[clime]

Could you explain a little bit the difference between the preproc time and the %{} expansion time? Doesn't rpm expand %{} repeatedly until no further expansion can be done?

( If there was template control flow it would make sense, because then you would have delimited blocks that you would want to process first, I think).

(my point about calling bash script from lua remains)

I mean this is a good question. One reason for having a preprocessing stage is existence of source rpms. Basically both lua scriplets and shell expansions are evaluated once srpm is built and then later when rpm is built from that srpm. The thing is that contexts (i.e. what information is available to the build process) is different each time. For srpm build, there are e.g. git metadata and raw source files, whereas for rpm build, there is no longer any git metadata and sources are packed in exported (e.g. by git archive) tarballs. I.e. lua scriplets and shell expansions that would access the git metadata or some files that aren't tarballed into srpm will succeed first time but fail the second for the rpm build itself. Therefore if such external context is to be used, it should be done so that evaluation is done only once (when building the srpm) and later the same precomputed results are used without any additional evaluation. Koji and Copr (and perhaps other build systems) work this way that they first build and srpm and only later rpm(s) from that srpm and they also store srpms for perhaps later rebuilds. So basically, if we would be building rpms directly from raw sources (e.g. git repos), then usage of lua scriplets or shell expansions would be probably fine for tasks like renderring changelog from git history and similar. Perhaps the usefulness of srpms is what decides whether the additional preprocessing step makes sense or not but this statement might be also a bit far-fetched. E.g. Copr allows you to submit only a spec file and build from it so I could imagine a use-case where you automatically "render" (i.e. run preprocessing) that spec-file locally and then submit it to Copr as a standalone thing. It's true that in this case you wouldn't need preprocessing to be part of rpm.

There are some other differences, e.g. that you can keep variable values across separate {{{ }}} invocations which I think you might not be able to do with shell expansion or the lua scriplets but that's rather an implementation detail i guess.

[clime]

If this is to be integrated into RPM, it needs to have clear semantics and escaping rules, and those rules need to be compatible with those of RPM. Code generators targeting RPM already know to escape % wherever it appears. It should also be opt-in.

What code generators are there currently if I may ask.

[elmarco]

I mean this is a good question. One reason for having a preprocessing stage is existence of source rpms. Basically both lua scriplets and shell expansions are evaluated once srpm is built and then later when rpm is built from that srpm. The thing is that contexts (i.e. what information is available to the build process) is different each time. [..]

Therefore if such external context is to be used, it should be done so that evaluation is done only once (when building the srpm) and later the same precomputed results are used without any additional evaluation. [..]

It doesn't look like %{} are evaluated/expanded at srpm-build time.

However, if we added some srpm time expansion, that would fit your requirement, no?

%{lua(source):
print('%changelog')
exec whatever etc..
}

Common code for generating changelog from git could be added the the lua rpm extension, so packagers could call:
%{lua(source): rpm.git_changelog()}

I just found out there is an %autochangelog now: https://fedoraproject.org/wiki/Changes/rpmautospec, did you know?!

[clime]

I mean this is a good question. One reason for having a preprocessing stage is existence of source rpms. Basically both lua scriplets and shell expansions are evaluated once srpm is built and then later when rpm is built from that srpm. The thing is that contexts (i.e. what information is available to the build process) is different each time. [..]

Therefore if such external context is to be used, it should be done so that evaluation is done only once (when building the srpm) and later the same precomputed results are used without any additional evaluation. [..]

It doesn't look like %{} are evaluated/expanded at srpm-build time.

Try to make spec file with e.g.:

Name: %{lua: print('hello')}

and call rpmbuild --define '_sourcedir .' -bs some.spec. You should see

Wrote: /home/clime/rpmbuild/SRPMS/hello-0.2-2.fc34.src.rpm

which means they are evaluated/expanded.

However, if we added some srpm time expansion, that would fit your requirement, no?

%{lua(source):
print('%changelog')
exec whatever etc..
}

Common code for generating changelog from git could be added the the lua rpm extension, so packagers could call: %{lua(source): rpm.git_changelog()}

If source meant "evaluate at the srpm build-stage only and keep the expanded value in the spec file verbatim", it would probably fit the requirement, yes.

I just found out there is an %autochangelog now: https://fedoraproject.org/wiki/Changes/rpmautospec, did you know?!

Yes, that solution solution relies on build system modifying the spec file based on some regexp parsing of the spec file (i.e. whether it contains %autochangelog) - it's not very clean.

[clime]

It doesn't look like %{} are evaluated/expanded at srpm-build time.

Well...they are evaluated/expanded but the resulting value is not kept in the spec file that goes into srpm. I guess that's what you meant :).

[DemiMarie]

If this is to be integrated into RPM, it needs to have clear semantics and escaping rules, and those rules need to be compatible with those of RPM. Code generators targeting RPM already know to escape % wherever it appears. It should also be opt-in.

What code generators are there currently if I may ask.

There are several tools that automatically generate changelogs. These tools should not need to be altered for the preprocessor.

[DemiMarie]

Also, the output of the external command needs to be escaped for RPM.

[clime]

Also I think {{{ }}} is not the best choice, as it conflicts with variables, something like %{{{{ to expand}}}} wouldn't work as expected.

I was able to fix this particular case in the parser so now %{{{{ echo foo}}}} outputs %{foo} as it should. But probably more testing is needed regarding possible interactions with default rpm constructs.

If this is to be integrated into RPM, it needs to have clear semantics and escaping rules, and those rules need to be compatible with those of RPM. Code generators targeting RPM already know to escape % wherever it appears. It should also be opt-in.

I think semantics should be quite clear - everything inside {{{ <x> }}} is passed to bash interpreter and stdout of the executed command(s) replaces the {{{ <x> }}}. preproc allows you to specify libraries of functions (macros) that should be available inside {{{ }}} tags through -s param. You can pass everything that is to be find in a certain dir (so basically when user installs certain package with preproc macros - they would become immediately available to rpm). You can make a certain variable shared across all subsequent {{{ }}} tags by exporting it (e.g. {{{ export FOO=bar }}} - this can be used to maintain state between the tags)

As for escaping, that I think depends on particular spec-file context. E.g. % in changelog should be escaped but if we were to render some rpm code block dynamically (i.e. something to be executed like %if, %else), then % there shouldn't be quoted. But basically the code executed from within {{{ }}} needs to take care of it. I e.g. also made a (preproc) macro to render changelog from git metadata https://docs.pagure.org/rpkg-util/v3/macro_reference.html#git-changelog-v3 - and this macro does the escaping.

Something %{lua(source):...} is cool too though. It's a smaller syntactic change. There might be some other pros/cons of each approach (bash vs. lua and similar).

[DemiMarie]

If this is to be integrated into RPM, it needs to have clear semantics and escaping rules, and those rules need to be compatible with those of RPM. Code generators targeting RPM already know to escape % wherever it appears. It should also be opt-in.

I think semantics should be quite clear - everything inside {{{ <x> }}} is passed to bash interpreter and stdout of the executed command(s) replaces the {{{ <x> }}}. preproc allows you to specify libraries of functions (macros) that should be available inside {{{ }}} tags through -s param. You can pass everything that is to be find in a certain dir (so basically when user installs certain package with preproc macros - they would become immediately available to rpm). You can make a certain variable shared across all subsequent {{{ }}} tags by exporting it (e.g. {{{ export FOO=bar }}} - this can be used to maintain state between the tags)

Consider a tool that generates changelogs from user-provided input. Changelogs are not executable in any way, so even if they come from untrusted sources, it should be sufficient to ensure they are properly escaped, well-formed, and only contain printable ASCII. A preprocessor like you mentioned breaks this assumption.

[clime]

Consider a tool that generates changelogs from user-provided input. Changelogs are not executable in any way, so even if they come from untrusted sources, it should be sufficient to ensure they are properly escaped, well-formed, and only contain printable ASCII. A preprocessor like you mentioned breaks this assumption.

I don't really understand what assumption is broken, by what way and by which tool. Can you be more specific, please or show an example?

[DemiMarie]

Consider a tool that generates changelogs from user-provided input. Changelogs are not executable in any way, so even if they come from untrusted sources, it should be sufficient to ensure they are properly escaped, well-formed, and only contain printable ASCII. A preprocessor like you mentioned breaks this assumption.

I don't really understand what assumption is broken, by what way and by which tool. Can you be more specific, please or show an example?

Suppose a I have the following (hypothetical) commit message:

Be more careful about cleaning up files

Previously we could accidentally wind up doing {{{ rm -rf /some/important/directory }}}

The tool then generates the following entry in the spec file changelog

- Be more careful about cleaning up files

  Previously we could accidentally wind up doing {{{ rm -rf /some/important/directory }}}

And when the preprocessor runs, the command rm -rf /some/important/directory is executed.

One way to prevent this is to instead use %{{{ and }}} as the delimiters, but leave escaped % characters alone (without unescaping). RPM requires % characters to be escaped, so if the commit message was

Be more careful about cleaning up files

Previously we could accidentally wind up doing %{{{ rm -rf /some/important/directory }}}

it would be converted into

- Be more careful about cleaning up files

  Previously we could accidentally wind up doing %%{{{ rm -rf /some/important/directory }}}

which the preprocessor leaves untouched. The previous nasty message

- Be more careful about cleaning up files

  Previously we could accidentally wind up doing {{{ rm -rf /some/important/directory }}}

is also left untouched.

- Be more careful about cleaning up files

  Previously we could accidentally wind up doing %{{{ rm -rf /some/important/directory }}}

would run rm -rf /some/important/directory, but there is no way to cause the generator to output this string as it contains an unescaped % character.

[clime]

Alright, thanks, makes sense.

Ye, I could make the tags configurable to be e.g. %{{{, }}} or just %{{, }}. I can also provide an option to specify an escape char as % so that preproc really leaves anything %%{{ ... }} untouched - that should still be in the realm of regexes that I am right now using to recognize the tags (i.e. preproc invocations).

But there wouldn't be a perfect syntactic consistency with rpm cause rpm cannot e.g. process %{lua:print('}')} expression whereas preproc can because it ignores everything in double or single quotes.

One other problem is that rpm is happy to leave %<something> unexpanded as long as <something> doesn't exist (i.e. it is not a name of a variable or a bash/lua substitution) and there is no error raised that the expansion isn't available. So if a changelog generating tool was perhaps too smart, it could leave %{{ unescaped with an assumption that {{... won't expand to anything. Likely no tool is doing this but it's a possibility.

So if you want a perfect syntactic consistency with rpm parser rules, the only way is to do this in rpm. If you decide to do this in rpm, it would be nice if you could do this not only for lua but also for bash, e.g. %{bash(src): ...}. That way I could port the macros from rpkg-util (https://docs.pagure.org/rpkg-util/v3/macro_reference.html) into some rpm-macros package and current users could switch to the new rpm syntax when they feel ready. The following features that preproc implements would also be great to have in rpm but perhaps I could find some (at least partial) workarounds during the port:

1. the possibility to maintain state across macro invocations by exporting a variable (e.g. %{bash(src): export FOO=bar } and I would be able to read FOO with bar value in the following %{bash(src): ... } invocations). Possibility to share state is important because preprocessing won't be a single atomic operation and if during parsing and evaluation someone e.g. switches a git branch, part of the spec file might be rendered inconsistently with the other (I am solving this in the preproc git macros by bootstrapping state with git information that I need afterwards).

2. loading a custom (user) macros through some rpmbuild switch (e.g. --load-srcmacros <path>). preproc enables this and some users of rpkg-util (where preprocessing right now resides) use it so it would be nice if there was a clear and full migration path for rpkg-util users.