Implement a declarative autobuild system (prototype) #2620
Conversation
|
Docs of course entirely missing as well, this is just an early rough-cut for feedback. The amhello.spec example + macros.in additions should make it fairly obvious though, it's not exactly rocket science. |
pmatilai
changed the title
|
We should probably put those build macros into separate macro files - one per build type. That makes it easier for people to add their own and to hand them over to other communities, if we want to get rid of them upstream. Also the macros file is too big already. |
|
Sure, I just stuffed them there because... we don't have anything else at the moment. |
|
I think this is going to need a versioned rpmlib capability, at least for SRPMs. |
|
It fails cleanly on the unrecognized Autobuild: tag on versions where not supported. If we added rpmlib() deps for all of those, we'd have hundreds of them by now. rpmlib() deps are primarily about ability to install an rpm (whether source or not), and that's not a problem here. |
|
This needs some design document or at least section in the commit message. This looks awfully like it could also be done with a simple macro or lua script. So we need some explanation why this isn't done that way. |
|
Which is what I said in the first comment:
Using plain macros instead of writing them into files and then parsing is required to handle %prep which is special and using a simple macro just doesn't work, and also to supposedly allow full spec syntax with %if's and the like to be used in the autobuild macros. That doesn't work if we just drop in some macros at the end of the parse, which is the simplest possible approach of course. At this point I'm more interested in the mechanism and what might be missing. Such as the excellent point about BuildRequires from @voxik (in the ticket). |
|
The BuildRequires case is just one example of Preamble content. The dynamic spec feature will soon drop you into the preamble of the main package for each parsed file (PR coming soon). The other option (which is possible now) is to expand a macro directly where the Autobuild directive is encountered - which is also in the main preamble. |
|
Yes, implementing the BuildRequires part isn't hard. For autobuild I do not want "insert arbitrary preamble section here" feature, that's not it's goal at all. Automating package generation is a separate topic. |
|
Oh and FWIW, I initially thought of just dropping these pieces into the specpartdir and let the dynamic spec parsing handle it, but that creates a chicken-egg problem: those generated spec parts require a build to happen first, and that build doesn't happen unless the autobuild stuff is injected into the spec at parse time already. |
|
Yeah, this needs to be non dynamic. But it can use dynamic spec in the implementation of the different build types. So this gets confusing quickly. |
|
Not just can, but it relies on the dynamic spec parsing machinery to handle the out-of-band content at the moment (that bit of refactoring to split out some logic out of parseGeneratedSpecs(), which needs to be split to a separate commit). That's an implementation detail users dont need to care about though. Oh and okay, you mean that it can use dynamic generation in addition to that. Yep. |
pmatilai
force-pushed
the
templ-pr
branch
2 times, most recently
from
5611b97 to
4a879f0
Compare
|
Some git hygiene applied to the previous work + rationale added to the commit message, BuildRequires support added. It probably should be squashed before eventual merging but thought the changes are easier to see this way. Doing the buildrequires like this does feel somehow clunky and ugly but we can't generate preamble content from here AFAICS. |
|
One thing I just realized is that there's no reason for each section to be in a different file. We'll still need a randomized tmp filename for it but it could at least hint at its purpose by autobuild prefix or such. |
We'll need this on other paths than specpartsdir in the next commits. No functional changes.
|
Rebased on current master and updated the test-case to showcase a more real-world use scenario with %build -a. Docs still missing of course, I'll do that next. This is basically fully functional as-in though, the macro alias stuff is more of an added bonus than a strict requirement. |
Remove endless boilerplate to perform the same steps in each package by just declaring which kind of buildsystem the package uses using the new Autobuild: spec tag. Autobuild macros are just regular parametric macros which can be invoked manually from specs too, which should help with reuse and handling complicated cases. As such , this will only work for simple and/or "perfect" upstream projects of course. In reality many/most packages need further tweaking, but this can now be easily done with the new append/prepend modes of the build scriptlets. The implementation looks more complicated than one might expect because we want the full spec processing to take place on these parts, rather than just macro expansion. That covers things like the special facilities of %prep and their side-effects, %if-conditionals, possible error handling and so on. Add automake upstream amhello-1.0.tar.gz as an test-case of a rather hands-free autobuild with added distro specific doc. Fixes: rpm-software-management#1087
|
Also, if we ship any default autobuild macros at all, we'll need to include something for cmake too. Shipping autotools macros but no cmake macros, in a cmake project, would look a bit odd 😅 |
|
It does probably make sense to pull the CMake macros into rpm so we can build ourselves from what is provided by rpm itself... 😅 |
|
It would be nice if we could also use this system to generate a EDIT: It is not lost on me how many times I said |
macros.in
Outdated
| @@ -1349,5 +1349,14 @@ end | |||
| end | |||
| } | |||
|
|
|||
| # example autobuild macros for autotools | |||
| %autobuild_autotools_prep() %autosetup -p1 | |||
There was a problem hiding this comment.
I'm not sure I understand %prep being part of this. It's not going to differ from one build system to another, and including it in the %build -a steps increases the number of spec files that won't be able to use autobuild because they need to perform some sort of pre-build shenanigans. (Running sed on some of the extracted files, extracting additional %SOURCEN tarballs, etc...)
There was a problem hiding this comment.
Sure, the %prep part is here more for demo purposes than something we'd want to ship for real. Instead we should probably just automatically add "%autosetup -p1" %prep section to everything that has sources and does not have a prep. I don't understand the remark about %build -a though. If you need to do pre-build shenanigans, you do it in one %prep -a for the earliest opportunity (assuming automatically added %prep %autosetup) , or in %conf -p/-a or %build -p, as appropriate.
There was a problem hiding this comment.
Actually, there was a reason %prep is a part of this: language ecosystems may have source formats outside the tarball realm, and in those cases we'd rather just have the ecosystem define %prep as they see fit, using their native tools. Ruby is an example here - rpm shouldn't have to know about .gem.
As for the rest, we could indeed just autopopulate %prep with a generic "%autosetup -p1" when Autobuild is set, but I'm not sure it's somehow better than doing it explicitly in the autobuild definition. It's just one line in the definition, and there could be other ecosystem preferences I guess. Thoughts welcome certainly.
There was a problem hiding this comment.
...and that's what the last fixup does: add autobuild defaults that will be applied if there's no build-system specific section. Only probably makes sense for %prep and %clean but the mechanism is generic so supporting the all makes no difference in code.
|
Another note here is that in this draft, the autobuild_* macros are parametric, but that's probably a bad idea - if |
|
Would it make sense to be able to expose some kind of simple way to pass options to the configure stanza? Autotools, CMake, and Meson at least all accept configure options, and it'd make sense to be able to do this without having to override the whole stage. A similar example of something like this is in Flatpak, where you declare the buildsystem type and then declare the options to be passed in. For example Lugaru's manifest sets the buildsystem to "cmake-ninja" and then has a field for setting extra options. |
You can. All build scriptlets are supported. |
|
Some early docs added, fixed typo/thinko wrt %generate_buildrequires name, and dropped the parametric requirement. |
The thought crossed my mind too, we could of course add an optional %{autobuild_mumble_opts} macro that can be used for passing extra options. I'm not sure how much more readable it is than overriding the section though (which is why it's not in the draft): vs The latter is more in line with the notion of declarative, so perhaps that's the way to go. Without taking away the ability to override sections. |
|
Why not define it as a field instead of a macro? |
|
You mean a spec tag, like |
|
There would need to be a way to pass options to all the autobuild macros, not just conf. Because sometimes you need that extra FU=1 SOMEDIR=/there to a make invocation etc, and ... of course the configure / make style line might not be the last thing in the autobuild macro. And so .. maybe these do need to be parametric afterall, to enforce a common convention for passing stuff, so we don't end up with one macro accepting anything passed after it and another one throwing an error. And thanks for the feedback so far everybody! |
I guess, it's just that a spec tag field is standardized, whereas a macro is not. Macro names can be anything at all and there are no real rules or discovery mechanisms for them. And if we're talking about handling the basic case in a reasonably sane way, then being able to pass autobuild configuration options in a uniform fashion would be nice. I'm not totally married to the idea of it being a spec tag though. It just feels like doing so would enforce more consistency. |
|
The macro names here would be simply "%autobuild_build_opts", "%autobuild_conf_opts" and so on, entirely predictable, and enforced by the implementation. But I agree tags are more, hmm, concrete somehow. I'll think about it 👍 |
|
Another remark on the per-section options (configuration being the most important of course) is that whatever the method/syntax, it needs to handle multiple long arguments gracefully. Because those config parameters can get long. Just on a random sampling, xterm in fedora has the following: ...and %configure entries of this length are in no way rare. And what's acceptable as a long formatted line-continuation when passed to %configure like this is not acceptable as a tag. As a macro body its more doable but still ugly - for declarative style they really should be separable entities and not just a heap of text. So, perhaps there should instead be macro(s) that allow you to add options, in a controlled fashion. Something along the lines of: |
|
|
|
Indeed - I was just coming back here to add that alternative 😅 |
|
The downside of tags is that they force the implementation to the C-side of things, whereas I'd rather have this in the (Lua) macro space. OTOH, the outcome has to be the more important factor than implementation preferences. |
|
Yet another random remark is that for this declarative style, "autobuild"-prefix creates annoyingly long names without being particularly descriptive. "Buildsys" comes to mind, but it only saves a couple of letters and I'm not sure it's significantly more descriptive anyway. Dunno. |
|
For a comparison of various alternatives, nothing works better than a real-world test (using an excerpt from Fedora xterm.spec):
2-3, ie the ones most discussed above, are the ones that actually add a lot of technically redundant clutter to the spec. They're are quite readable though, and although it's not evident here, they have the (non-trivial) benefit of being relatively freely placeable within the spec: in many/most cases you'd be able to place the option right alongside eg extra buildrequires for that option. 3) is even more freely placeable than 2. 5 is the only one that really reduces boilerplate and ugly + brittle line-continuations, at the cost of forcing the declarations to one spot again, which is a considerable loss compared to 2-3. It's also way less obviously independent entries and looks more like a continuous text, which makes you think there are line continuations missing or something. 4 is only a win over 1 if one assumes complicated autobuild macros that expand into more than one command so you can't just continue the line as you'd do with eg %configure. So it's a curiously contradicting set of possibilities, and despite being more "wordy", 2-3 have significant advantages over the seemingly more simple versions. |
|
For a real-world examle of the 2-3 benefit, a typical bcond case from Fedora rpm.spec: Existing spec: Using a section override for %conf would be effectively unchanged from existing. Spec tags: 3 would work equally well, just look a little out of place in middle of those spec tags. In some other situation it may be surrounded by other macros and a tag would look out of place, but that's probably an uncommon situation, actually. The above also screams to be Yet another observation is that configuration is the only place where one typically expects "dozens" of options. For actual build/install stages, it's usually more a matter of adding a flag / argument or two to override some specific thing. Or, that's what I thought. Looking at the Fedora spec collection, there's all manner of stuff being passed to both %make_build and %make_install (and never mind everything that falls outside these). So while I initially thought perhaps only configuration deserves the 2-3 style arrangement and others could do with just a simple _opts macro (ie 4), perhaps not so. |
|
So... the conclusion from all the above rant is that there seems to be an actual design wanting to come out:
Too bad there is no option to convert a PR into a discussion because that's what this resembles more now 😄 |
|
Ok, back to drawing board, now with a much nicer plan. Thanks for the very valuable feedback + ideas! |
|
|
Remove endless boilerplate to perform the same steps in each package by just declaring which kind of buildsystem the package uses using the new Autobuild: spec tag.
Autobuild macros are just regular parametric macros which can be invoked manually from specs too, which should help with reuse and handling complicated cases.
As such , this will only work for simple and/or "perfect" upstream projects of course. In reality many/most packages need further tweaking in %install or otherwise. Augmenting those (one way or the other) needs to be implemented to make this real-world usable.
Add automake upstream amhello-1.0.tar.gz as an test-case of a rather hands-free autobuild.
Fixes: #1087