Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

samba: Add autobuild.yml files for dep resolution #2279

Merged
merged 1 commit into from
May 1, 2024

Conversation

ermo
Copy link
Contributor

@ermo ermo commented Apr 17, 2024

Summary
This is the initial attempt at showing how to use the power of autobuild for getting the Solus monorepo under control re. dependency cycles. It is based on this gist.

The purely additive nature of the autobuild.yml configuration files added with this commit, should make it trivial to modify and extend the ignored set of dependencies for a package as our cycle-discovery journey advances.

To gain an appreciation of just how useful this tool might become for us, consider the following:

ermo@solbox:~/repos/getsolus/soluspkgs
$ autobuild -q query src:packages/ samba ldb tdb talloc tevent
talloc tdb cifs-utils notmuch tevent ldb samba acccheck ffmpeg python-pysmbc gvfs kio-extras kodi mpd rhythmbox vlc gnome-control-center nautilus-share nemo-extensions budgie-control-center

The above ^ shows how autobuild might resolve a build + rebuild order for when someone is doing a samba stack upgrade and pushing it to a single builder; like, say, the local packager's machine or the current Solus build infrastructure, which is limited to one builder.

Note that autobuild also already supports committing+pushing to the Solus builder, which implies that it can replace the current build_pr.sh script in short order, if the the appetite to do so is present within Solus as an organisation.

But that's not all!

In fact, autobuild also supports generating a "tiered" builder order (notice the -t argument), which those of you packaging KDE might already be familiar with:

ermo@solbox:~/repos/getsolus/soluspkgs
$ autobuild query -t src:packages/ samba ldb tdb talloc tevent
 (.. some output omitted ..)
 🗸  Successfully parsed state!
 🗸  Found all requested packages in state!
 🗸  Successfully built dependency graph!
 🗸  Build order:
 🗸  Tier 1: talloc tdb
 🗸  Tier 2: cifs-utils notmuch tevent
 🗸  Tier 3: ldb
 🗸  Tier 4: samba
 🗸  Tier 5: acccheck ffmpeg python-pysmbc
 🗸  Tier 6: gvfs kio-extras kodi mpd rhythmbox vlc
 🗸  Tier 7: gnome-control-center nautilus-share nemo-extensions
 🗸  Tier 8: budgie-control-center

Tiers (called "Precedence Constraint Groups" in graph theory literature) effectively show which packages can be built in parallel since they do not interdepend on each other.

With the -q argument like above, this build order could potentially drive a set of builders (like summit does in the Serpent infra currently):

ermo@solbox:~/repos/getsolus/soluspkgs
$ autobuild -qt query src:packages/ samba ldb tdb talloc tevent
talloc tdb
cifs-utils notmuch tevent
ldb
samba
acccheck ffmpeg python-pysmbc
gvfs kio-extras kodi mpd rhythmbox vlc
budgie-control-center gnome-control-center nautilus-share nemo-extensions

I hope this serves to show how useful autobuild can potentially become to Serpent and Solus in terms of automating the discovery of build and rebuild orders.

This commit is mostly intended as a way to showcase what an autobuild-driven packaging process might look like for Solus and Serpent, and as a starting point for the discussion of how to tag "Auxiliary Dependencies" in a way that's compatible with both the Solus and Serpent monorepos and tooling.

See also: #2175 and #2176

autobuild.yaml format design notes here: GZGavinZhao/autobuild#16

Massive props to @GZGavinZhao for his interest in studying the necessary graph theory at my suggestion, and also for his being ready, able, and willing to design and build this tool, and for his willingness to take on board constructive feedback re. the documentation and UX necessary to productise this solution for a wider audience of Solus staff and community packagers.

@ermo ermo marked this pull request as draft April 17, 2024 10:16
@ermo ermo force-pushed the samba-autobuild-cycle-breaking branch from c8c9d87 to dfda866 Compare April 17, 2024 11:42
@ermo ermo added Topic: Tooling Tools and scripts Topic: Platform Integration Integration of various components within Solus Topic: Plumbing Core components Type: Feature Something can be enhanced. and removed Topic: Plumbing Core components labels Apr 17, 2024
@ermo ermo force-pushed the samba-autobuild-cycle-breaking branch 3 times, most recently from 117bdc0 to 4feb8db Compare May 1, 2024 17:23
**Summary**
This is the initial attempt at showing how to use the power of autobuild
for getting the Solus monorepo under control re. dependency cycles.

The purely additive nature of the autobuild.yml configuration files added
with this commit, should make it trivial to modify and extend the ignored
set of dependencies for a package as our cycle-discovery journey
advances.

To gain an appreciation of just how useful this tool might become for us,
consider the following:

```
ermo@solbox:~/repos/getsolus/soluspkgs
$ autobuild -q query src:packages/ samba ldb tdb talloc tevent
talloc tdb cifs-utils notmuch tevent ldb samba acccheck ffmpeg \
python-pysmbc gvfs kio-extras kodi mpd rhythmbox vlc \
gnome-control-center nautilus-share nemo-extensions budgie-control-center
```

The above ^ shows how autobuild might resolve a build + rebuild order for
when someone is doing a samba stack upgrade and pushing it to a single
builder; like, say, the local packager's machine or the current Solus
build infrastructure, which is limited to one builder.

Note that autobuild also already supports committing+pushing to the Solus
builder, which implies that it can replace the current `build_pr.sh`
script in short order, if the the appetite to do so is present within
Solus as an organisation.

But that's not all!

In fact, `autobuild` also supports generating a "tiered" builder order
(notice the `-t` argument), which those of you packaging KDE might
already be familiar with:

```
ermo@solbox:~/repos/getsolus/soluspkgs
$ autobuild query -t src:packages/ samba ldb tdb talloc tevent
 (.. some output omitted ..)
 🗸  Successfully parsed state!
 🗸  Found all requested packages in state!
 🗸  Successfully built dependency graph!
 🗸  Build order:
 🗸  Tier 1: talloc tdb
 🗸  Tier 2: cifs-utils notmuch tevent
 🗸  Tier 3: ldb
 🗸  Tier 4: samba
 🗸  Tier 5: acccheck ffmpeg python-pysmbc
 🗸  Tier 6: gvfs kio-extras kodi mpd rhythmbox vlc
 🗸  Tier 7: gnome-control-center nautilus-share nemo-extensions
 🗸  Tier 8: budgie-control-center
```

Tiers (called "Precedence Constraint Groups" in graph theory literature)
effectively show which packages can be built in parallel since they do
not interdepend on each other.

With the -q argument like above, this build order could potentially drive
a set of builders (like summit does in the Serpent infra currently):

```
ermo@solbox:~/repos/getsolus/soluspkgs
$ autobuild -qt query src:packages/ samba ldb tdb talloc tevent
talloc tdb
cifs-utils notmuch tevent
ldb
samba
acccheck ffmpeg python-pysmbc
gvfs kio-extras kodi mpd rhythmbox vlc
budgie-control-center gnome-control-center nautilus-share nemo-extensions
```

I hope this serves to show how useful autobuild can potentially become
to Serpent and Solus in terms of automating the discovery of build and
rebuild orders.

This commit is mostly intended as a way to showcase what an autobuild-
driven packaging process might look like for Solus and Serpent, and as a
starting point for the discussion of how to tag "Auxiliary Dependencies"
in a way that's compatible with both the Solus and Serpent monorepos and
tooling.

See also: #2175 and #2176

autobuild.yaml format design notes here:

GZGavinZhao/autobuild#16

Massive props to @GZGavinZhao for his interest in studying the necessary
graph theory at my suggestion, and also for his being ready, able, and
willing to design and build this tool, and for his willingness to take on
board constructive feedback re. the documentation and UX necessary to
productise this solution for a wider audience of Solus staff and
community packagers.

Signed-off-by: Rune Morling <[email protected]>
@ermo ermo force-pushed the samba-autobuild-cycle-breaking branch from 4feb8db to 7e00a49 Compare May 1, 2024 17:25
@ermo ermo marked this pull request as ready for review May 1, 2024 17:27
@ermo ermo merged commit c5175af into main May 1, 2024
3 checks passed
@ermo ermo deleted the samba-autobuild-cycle-breaking branch May 1, 2024 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Topic: Platform Integration Integration of various components within Solus Topic: Tooling Tools and scripts Type: Feature Something can be enhanced.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

1 participant