refactor: lockfile #95
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjmaestro
force-pushed
the
refactor-lockfile
branch
2 times, most recently
from
a871658
to
ab87a69
Compare
jjmaestro
force-pushed
the
refactor-lockfile
branch
3 times, most recently
from
e2a2406
to
da44c65
Compare
* separate the script into a template file so it's easier to shellcheck
and syntax highlight in editors.
* shellcheck the script and remove all SC2086 warnings
("Double quote to prevent globbing and word splitting")
* improve the buildozer help messages in the copy.sh script:
* reduce duplication of buildozer command
* add a more clear autofix bazel run command that can be easily
copy-pasted
* change some of the variable names in the copy.sh template for longer,
easier to understand names (repo_name >> name, lock_label >> label)
* move repo_name and workspace_relative_path into variables to reduce
line length and improve readability
While working with some flaky mirrors and trying to figure out why they were failing I found the _fetch_package_index code a bit hard to follow so here's my attempt at streamlining it a bit: * Change the general flow of the for-loop so that we can directly set the reasons for failure in failed_attempts * Remove both integrity as an argument and as a return value since neither is ever used. * return the content of the Packages index instead of the path, since we already have the repository_context. This way we don't need rctx anywhere else. * Reword failure messages adding more context and debug information * Shorter lines and templated strings, trying to make the code easier to read and follow.
* remove the `state` "intermediary `struct`" in `package_resolution.bzl` since it wasn't used / needed. * refactor and move _set_dict from `util.bzl` to a `_package_set` method * use `dict .get()` with default values instead of the "nested `if`s" * renamed `package()` to `package_get` and make it return all package versions when the version is not specified so we can remove `_package_versions` * reordeder `(name, version, arch)` args to match the order of the index keys `(arch, name, version)`
Add testing for package_index mocking the external / side effects (downloads, decompression, etc).
The package resolution debugging that e.g. checks the package dependencies should all be within package_resolution.bzl _resolve_all() and not "leak out" returning the information that's only needed for debugging / logging. Also: * reduce the verbosity of the optional dependencies warning by just printing one message per root package instead of one per package. * break up the long lines to build the error messages and remove the "# buildifier: disable=print".
* move version constraint parsing from package_resolution to its own _parse_version_and_constraint method in version.bzl * refactor _version_relop into a compare method in version.bzl plus a VERSION_OPERATORS dict so that (1) we use the operator strings everywhere and (2) we can use the keys to validate the operators.
Previously we had:
```starlark
pkgindex = package_index.new(rctx, sources = sources, archs = manifest["archs"])
pkgresolution = package_resolution.new(index = pkgindex)
```
And none of the code of package_resolution was used anywhere but in
resolve.bzl and after initializing the `pkgindex`.
Also, it makes sense since we are building the index from the manifest
and once we have the index we use it to resolve the packages and
populate the lock.
Cleanup resolve.bzl and package_index.bzl by moving all of the manifest functionality to a separate manifest.bzl file where we now do all of the work to generate the lock: manifest parsing, validation and the package index and resolution. IMHO this is how it should be because the lock is the "frozen state" of the manifest. * _parse() parses the YAML * _from_dict validates the manifest dict and does the rest of the changes that we need to produce a manifest struct * add extra validation for e.g. duplicated architectures * _lock is the only method that's exposed to the outside and it encapsulates all of the other parts, calling _from_dict and all of the package index and resolution, to produce the lock file. * move get_dupes to util.bzl * refactor the "source" struct into the new manifest where we can now centralize a lot of the structure and logic spread across multiple parts of the code. * remove yq_toolchain_prefix since it's always "yq" and, looking at GH code search, this seems to be a copy-paste leftover from rules_js (or the other way around)... the code is always the same and it never receives a string different from "yq".
* move all of the "package logic" to pkg.bzl
* The v2 lockfile format:
* doesn't need the fast_package_lookup dict because it's already using
a dict to store the packages.
* has the dependencies sorted so the lockfile now has stable
serialization and the diffs of the lock are actually usable and
useful to compare with the changes to the manifest.
* removes the package and dependency key from the lockfile, now it's
done via an external function (make_deb_import_key in
deb_import.bzl)
* Remove add_package_dependency from the lockfile API. Now, the package
dependencies are passed as an argument to add_package. This way, the
lockfile functionality is fully contained in lockfile.bzl and e.g. we
can remove the "consistency checks" that were only needed because
users could forget to add the dependency as a package to the lockfile.
* Ensure backwards-compatibility by internally converting lock v1 to v2.
Also, when a lock is set and it's in v1 format, there's a reminder
that encourages the users to run @repo//:lock to update the lockfile
format.
By separating the migration from the previous commit we get to 1. in the previous commit, run all tests with the new code while locks are still v1 2. update the locks n this commit to V2 so we can then re-run all tests in the final state.
jjmaestro
force-pushed
the
refactor-lockfile
branch
from
da44c65
to
90e74e3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Stacked on top of #94
refactor: lockfile, new pkg.bzl, and lock v1-to-v2 auto-migration
move all of the "package logic" to the new
pkg.bzlThe v2 lockfile format:
doesn't need the fast_package_lookup dict because it's already using a dict to store the packages.
has the dependencies sorted so the lockfile now has stable serialization and the diffs of the lock are actually usable and useful to compare with the changes to the manifest.
removes the package and dependency key from the lockfile, now it's done via an external function (
make_deb_import_keyindeb_import.bzl)Remove
add_package_dependencyfrom the lockfile API. Now, the package dependencies are passed as an argument to add_package. This way, the lockfile functionality is fully contained inlockfile.bzland e.g. we can remove the "consistency checks" that were only needed because users could forget to add the dependency as a package to the lockfile.Ensure backwards-compatibility by internally converting lock v1 to v2. Also, when a lock is set and it's in v1 format, there's a reminder that encourages the users to run @repo//:lock to update the lockfile format.
chore: migrate repo locks to v2
By separating the migration from the previous commit we make sure that the change is backwards compatible:
in the previous commit, run all tests with the new code while locks
are still v1
update the locks n this commit to V2 so we can then re-run all tests
in the final state.