Fix running python_sources with pex --executable (Cherry-pick of #21047) #21083
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eploy` goal (#20488) Changed the following things to achieve this - Add a `--dry-run` flag to the `experimental-deploy` goal to handle dry runs such as `terraform plan` - Run `plan` rather than `apply` when setting the `--dry-run` flag while deploying a `terraform_deployment` - Changed Helm deployments to use the new `--dry-run` flag instead of a passthrough arg for dry-running Tested - the terraform change via a unit test and also tried it out manually in my own project. - the helm change via a unit test. Closes #18490
Related issues and discussions: * #20548 * https://pantsbuild.slack.com/archives/C0D7TNJHL/p1708092220006959
…20554) When using go 1.22, trying to compile a go package using pants fails with: ``` 13:07:42.25 [INFO] Initialization options changed: reinitializing scheduler... 13:07:46.62 [INFO] Scheduler initialized. 13:07:51.89 [ERROR] Completed: Compile with Go - runtime/internal/syscall - runtime/internal/syscall failed (exit code 1). flag provided but not defined: -compiling-runtime usage: asm [options] file.s ... Flags: -D value predefined symbol with optional simple value -D=identifier=value; can be set multiple times -I value include directory; can be set multiple times -S print assembly and machine code -V print version and exit -d value enable debugging settings; try -d help -debug dump instructions as they are parsed -dynlink support references to Go symbols defined in other shared libraries -e no limit on number of errors reported -gensymabis write symbol ABI information to output file, don't assemble -linkshared generate code that will be linked against Go shared libraries -o string output file; default foo.o for /a/b/c/foo.s as first argument -p string set expected package import to path (default "<unlinkable>") -shared generate code that can be linked into a shared library -spectre list enable spectre mitigations in list (all, ret) -trimpath string remove prefix from recorded source file paths -v print debug output ``` This flag has been removed from version 1.22 - asm can now make the determination itself regarding whether or not we're compiling a `runtime` package, so there's no need to pass that flag anymore. See: https://cs.opensource.google/go/go/+/72946ae8674a295e7485982fe57c65c7142b2c14
Previously the top-level menu was called "Java and Scala" and "Kotlin" was a subpage. Now the top-level is called "JVM" and there are two subpages - "Java and Scala", and "Kotlin".
Previously the options sources list could have only one source per type (flag/env/config/default). Since we support multiple config files, we had the concept of a "merged config" that encompassed multiple "config sources". This led to duplication of the logic that traverses and merges sources. This change gets rid of the concept of merged config. Instead, the list of options sources we consult can now include multiple config files. This is achieved by enhancing the Source::Config enum to include an ordinal (to ensure that configs are applied in the right order) and the path to the config file (purely informational). The natural `#[derive(Ord)]` will order these augmented configs appropriately within all the sources. To make this easier to achieve we change the order of the enum to be from lowest to highest priority, so that we can add config files naturally in the order in which we encounter them. This also allows us to simplify some types. In particular, we no longer need to traffic in `Vec<DictEdit>` - a single source can only return a single `DictEdit`. More importantly, this supports derivation history that includes the specific config file each value comes from.
Upgrade to Rust v1.76.0. [Blog post](https://blog.rust-lang.org/2024/02/08/Rust-1.76.0.html)
Closes #20576. Although this is a relatively small change, I think it's good to understand the change in context so I've included the new and old logic below for comparison. New logic: ```py formatter_to_request_class: dict[Formatter, type[RewrittenBuildFileRequest]] = { Formatter.BLACK: FormatWithBlackRequest, Formatter.YAPF: FormatWithYapfRequest, Formatter.RUFF: FormatWithRuffRequest, } chosen_formatter_request_class = formatter_to_request_class.get( update_build_files_subsystem.formatter ) if not chosen_formatter_request_class: raise ValueError(f"Unrecognized formatter: {update_build_files_subsystem.formatter}") for request in union_membership[RewrittenBuildFileRequest]: if update_build_files_subsystem.fmt and request == chosen_formatter_request_class: rewrite_request_classes.append(request) if update_build_files_subsystem.fix_safe_deprecations and issubclass( request, DeprecationFixerRequest ): rewrite_request_classes.append(request) # If there are other types of requests that aren't the standard formatter # backends or deprecation fixers, add them here. if request not in formatter_to_request_class.values() and not issubclass( request, DeprecationFixerRequest ): rewrite_request_classes.append(request) ``` Old logic: ```py for request in union_membership[RewrittenBuildFileRequest]: if issubclass(request, (FormatWithBlackRequest, FormatWithYapfRequest)): is_chosen_formatter = issubclass(request, FormatWithBlackRequest) ^ ( update_build_files_subsystem.formatter == Formatter.YAPF ) if update_build_files_subsystem.fmt and is_chosen_formatter: rewrite_request_classes.append(request) else: continue if update_build_files_subsystem.fix_safe_deprecations or not issubclass( request, DeprecationFixerRequest ): rewrite_request_classes.append(request) ``` The `else: continue` in the old logic was load-bearing, because it would skip over the second top-level `if` statement and move to the next loop iteration in cases where the request class was one of the regular formatters (`FormatWithBlackRequest` or `FormatWithYapfRequest`). The `or not issubclass(request, DeprecationFixerRequest)` was intended to catch formatters that live outside of the list of formatter backends and deprecation fixers - the last `if` statement in the new logic is intended to cover this case as well.
Add docs to support the newly added `--format` option for the `dependencies` and `dependents` goals which are in 2.20.
This PR fixes #20427 by updating our use of `doc_url` to reflect the new website structure. It does this semi-automatically, spread across separate commits: 1. Update the `doc_url` function to reflect the new website structure, including support for unversioned URLs like https://www.pantsbuild.org/community/getting-help 2. Apply automated rewrites (see below for details) 3. Do one or two manual fix-ups The automatic rewrites were done by: 1. finding the args passed to `doc_url`: `rg --only-matching "doc_url\([^)]*\)" . | cut -f2 -d\( | cut -f1 -d\) | tr \' \" | sort -u` 2. put them into CSV file, then use https://github.com/pantsbuild/pantsbuild.org/blob/92d5ce8d3442890c3fbeb0fade620b762ae2aa7e/old_site_redirects.js to find the new URL for each one 3. transform the CSV into a long series of `s@...@...@` substitutions for `sed` 4. apply them with `git ls-files '*.py' '*.rs' | xargs sed -i '' "..."` 5. do a basic check that the appropriate files that call `doc_url` were updated: identify non-updated files via `comm -13 <(git diff --name-only be42bc0^...be42bc0) <( rg --files-with-matches doc_url | sort)`. The files without updates are expected, e.g. - `src/python/pants/util/docutil.py` that defines `doc_url` itself (and the test file etc.) - `src/python/pants/backend/codegen/avro/target_types.py` that didn't have an immediately obvious replacement, with more examples in #20584 This is marked for cherry-picking to 2.20.x. There _are_ redirects for the URLs for 2.20, but it seems a bit annoying to rely on them, if we don't have to. <details> ```csv old,new "getting-help","community/getting-help" "advanced-target-selection","docs/using-pants/advanced-target-selection" "anonymous-telemetry","docs/using-pants/anonymous-telemetry" "awslambda-python","docs/python/integrations/aws-lambda" "enabling-backends","docs/using-pants/key-concepts/backends" "google-cloud-function-python","docs/python/integrations/google-cloud-functions" "installation","docs/getting-started/installing-pants" "macros","docs/writing-plugins/macros" "options","docs/using-pants/key-concepts/options" "options#addremove-semantics","docs/using-pants/key-concepts/options#addremove-semantics" "options#config-file-entries","docs/using-pants/key-concepts/options#config-file-entries" "options#config-file-interpolation","docs/using-pants/key-concepts/options#config-file-interpolation" "options#pantsrc-file","docs/using-pants/key-concepts/options#pantsrc-file" "options#reading-individual-option-values-from-files","docs/using-pants/key-concepts/options#reading-individual-option-values-from-files" "pex","docs/python/overview/pex" "plugin-upgrade-guide","docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide" "plugins-overview","docs/writing-plugins/overview" "plugins-setup-py","docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs" "protobuf-go","docs/go/integrations/protobuf" "protobuf-python","docs/python/integrations/protobuf-and-grpc" "python-check-goal","docs/python/goals/check" "python-distributions","docs/python/overview/building-distributions" "python-interpreter-compatibility","docs/python/overview/interpreter-compatibility" "python-lockfiles#lockfiles-for-tools","docs/python/overview/lockfiles#lockfiles-for-tools" "python-test-goal","docs/python/goals/test" "python-test-goal#pytest-version-and-plugins","docs/python/goals/test#pytest-version-and-plugins" "python-third-party-dependencies","docs/python/overview/third-party-dependencies" "python-third-party-dependencies#local-requirements","docs/python/overview/third-party-dependencies#local-requirements" "python-third-party-dependencies#multiple-lockfiles","docs/python/overview/lockfiles#multiple-lockfiles" "python-third-party-dependencies#user-lockfiles","docs/python/overview/third-party-dependencies#user-lockfiles" "reference-deploy_jar#coderesolvecode","reference/targets/deploy_jar#resolve" "reference-global#remote_oauth_bearer_token","reference/global-options#remote_oauth_bearer_token" "remote-caching-execution","docs/using-pants/remote-caching-and-execution" "source-roots","docs/using-pants/key-concepts/source-roots" "tagging-docker-images","docs/docker/tagging-docker-images" "target-api-concepts","docs/writing-plugins/the-target-api/concepts" "targets","docs/using-pants/key-concepts/targets-and-build-files" "targets#field-default-values","docs/using-pants/key-concepts/targets-and-build-files#field-default-values" "targets#dependencies-and-dependency-inference","docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference" "thrift-python","docs/python/integrations/thrift" "troubleshooting","docs/using-pants/troubleshooting-common-issues" "troubleshooting#import-errors-and-missing-dependencies","docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies" "upgrade-tips","docs/releases/upgrade-tips" ``` ```python import pandas as pd print("\n".join( f"""s@\\(doc_url.*[\\"']\\){r.old}\\([\\"']\\)@\\1{r.new}\\2@g;""" for _, r in pd.read_csv("replacements.csv").iterrows() )) ``` ```shell git ls-files '*.py' '*.rs' | xargs sed -i '' " s@\(doc_url.*[\"']\)getting-help\([\"']\)@\1community/getting-help\2@g; s@\(doc_url.*[\"']\)advanced-target-selection\([\"']\)@\1docs/using-pants/advanced-target-selection\2@g; s@\(doc_url.*[\"']\)anonymous-telemetry\([\"']\)@\1docs/using-pants/anonymous-telemetry\2@g; s@\(doc_url.*[\"']\)awslambda-python\([\"']\)@\1docs/python/integrations/aws-lambda\2@g; s@\(doc_url.*[\"']\)enabling-backends\([\"']\)@\1docs/using-pants/key-concepts/backends\2@g; s@\(doc_url.*[\"']\)google-cloud-function-python\([\"']\)@\1docs/python/integrations/google-cloud-functions\2@g; s@\(doc_url.*[\"']\)installation\([\"']\)@\1docs/getting-started/installing-pants\2@g; s@\(doc_url.*[\"']\)macros\([\"']\)@\1docs/writing-plugins/macros\2@g; s@\(doc_url.*[\"']\)options\([\"']\)@\1docs/using-pants/key-concepts/options\2@g; s@\(doc_url.*[\"']\)options#addremove-semantics\([\"']\)@\1docs/using-pants/key-concepts/options#addremove-semantics\2@g; s@\(doc_url.*[\"']\)options#config-file-entries\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-entries\2@g; s@\(doc_url.*[\"']\)options#config-file-interpolation\([\"']\)@\1docs/using-pants/key-concepts/options#config-file-interpolation\2@g; s@\(doc_url.*[\"']\)options#pantsrc-file\([\"']\)@\1docs/using-pants/key-concepts/options#pantsrc-file\2@g; s@\(doc_url.*[\"']\)options#reading-individual-option-values-from-files\([\"']\)@\1docs/using-pants/key-concepts/options#reading-individual-option-values-from-files\2@g; s@\(doc_url.*[\"']\)pex\([\"']\)@\1docs/python/overview/pex\2@g; s@\(doc_url.*[\"']\)plugin-upgrade-guide\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/plugin-upgrade-guide\2@g; s@\(doc_url.*[\"']\)plugins-overview\([\"']\)@\1docs/writing-plugins/overview\2@g; s@\(doc_url.*[\"']\)plugins-setup-py\([\"']\)@\1docs/writing-plugins/common-plugin-tasks/custom-python-artifact-kwargs\2@g; s@\(doc_url.*[\"']\)protobuf-go\([\"']\)@\1docs/go/integrations/protobuf\2@g; s@\(doc_url.*[\"']\)protobuf-python\([\"']\)@\1docs/python/integrations/protobuf-and-grpc\2@g; s@\(doc_url.*[\"']\)python-check-goal\([\"']\)@\1docs/python/goals/check\2@g; s@\(doc_url.*[\"']\)python-distributions\([\"']\)@\1docs/python/overview/building-distributions\2@g; s@\(doc_url.*[\"']\)python-interpreter-compatibility\([\"']\)@\1docs/python/overview/interpreter-compatibility\2@g; s@\(doc_url.*[\"']\)python-lockfiles#lockfiles-for-tools\([\"']\)@\1docs/python/overview/lockfiles#lockfiles-for-tools\2@g; s@\(doc_url.*[\"']\)python-test-goal\([\"']\)@\1docs/python/goals/test\2@g; s@\(doc_url.*[\"']\)python-test-goal#pytest-version-and-plugins\([\"']\)@\1docs/python/goals/test#pytest-version-and-plugins\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies\([\"']\)@\1docs/python/overview/third-party-dependencies\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies#local-requirements\([\"']\)@\1docs/python/overview/third-party-dependencies#local-requirements\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies#multiple-lockfiles\([\"']\)@\1docs/python/overview/lockfiles#multiple-lockfiles\2@g; s@\(doc_url.*[\"']\)python-third-party-dependencies#user-lockfiles\([\"']\)@\1docs/python/overview/third-party-dependencies#user-lockfiles\2@g; s@\(doc_url.*[\"']\)reference-deploy_jar#coderesolvecode\([\"']\)@\1reference/targets/deploy_jar#resolve\2@g; s@\(doc_url.*[\"']\)reference-global#remote_oauth_bearer_token\([\"']\)@\1reference/global-options#remote_oauth_bearer_token\2@g; s@\(doc_url.*[\"']\)remote-caching-execution\([\"']\)@\1docs/using-pants/remote-caching-and-execution\2@g; s@\(doc_url.*[\"']\)source-roots\([\"']\)@\1docs/using-pants/key-concepts/source-roots\2@g; s@\(doc_url.*[\"']\)tagging-docker-images\([\"']\)@\1docs/docker/tagging-docker-images\2@g; s@\(doc_url.*[\"']\)target-api-concepts\([\"']\)@\1docs/writing-plugins/the-target-api/concepts\2@g; s@\(doc_url.*[\"']\)targets\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files\2@g; s@\(doc_url.*[\"']\)targets#field-default-values\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#field-default-values\2@g; s@\(doc_url.*[\"']\)targets#dependencies-and-dependency-inference\([\"']\)@\1docs/using-pants/key-concepts/targets-and-build-files#dependencies-and-dependency-inference\2@g; s@\(doc_url.*[\"']\)thrift-python\([\"']\)@\1docs/python/integrations/thrift\2@g; s@\(doc_url.*[\"']\)troubleshooting\([\"']\)@\1docs/using-pants/troubleshooting-common-issues\2@g; s@\(doc_url.*[\"']\)troubleshooting#import-errors-and-missing-dependencies\([\"']\)@\1docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies\2@g; s@\(doc_url.*[\"']\)upgrade-tips\([\"']\)@\1docs/releases/upgrade-tips\2@g; " ``` </details>
Add support for `pants package` using yarn@v2. Currently when trying to package a yarn@v2 project, the following error occurs: ``` Engine traceback: in `package` goal ProcessExecutionFailure: Process 'Installing [email protected].' failed with exit code 1. stdout: ➤ YN0050: The --frozen-lockfile option is deprecated; use --immutable and/or --immutable-cache instead stderr: ``` Thats because the option `--frozen-lockfile` was replaced with `--immutable` on yarn >= 2. This behavior can be reproducible at https://github.com/kevinobruno/pants-yarn running `pants package //yarnv2`
Changelogs: * https://github.com/pex-tool/pex/releases/tag/v2.1.164 * https://github.com/pex-tool/pex/releases/tag/v2.2.0 * https://github.com/pex-tool/pex/releases/tag/v2.2.1 (New home, new docs, but no direct Pants related changes.) ``` Lockfile diff: 3rdparty/python/user_reqs.lock [python-default] == Upgraded dependencies == cryptography 42.0.2 --> 42.0.3 pex 2.1.163 --> 2.2.1 urllib3 2.2.0 --> 2.2.1 ```
…20513) This was motivated by trying to add some additional module and stub-related mappings. While doing that, I realized the patterns only applied to implementation modules only, and realized that a refactor would be the easiest way forward, while also making the logic easier to step through. I haven't added any of those module mappings in this PR since I wanted this to only be the logic change. ## Changes * Make the replacement functions more generic (allow injecting any separator) * Better type hinting - more complete types for generics, also made return values more specific (return values and variables should be hinted as specifically as possible; functions can choose to accept more generic types as arguments) * Made impl and type package mappings behave identically * The previous logic seemed to have potential for bugs - it does a fallback on each call, has some hackiness around checking for a stubs package, nested conditions, etc. Simplified the module lookup logic. Now it's just: * check if it's in the standard module mapping * else check if it's in the type stub module mapping * else check if the patterns for type stub modules match - this needs to be done first because a stub module might match an implementation module pattern (example, `python-pkg-types`) * else check if the patterns for impl modules match * else use the fallback * Update tests to ensure symmetry between standard modules and type stub modules * Added new test for `test_generate_type_stub_mappings_from_pattern_matches_para`, which is the main new logic here * All other changes created by `pants fmt` ## Testing * Added and modified doctest examples, and `python src/python/pants/backend/python/dependency_inference/default_module_mapping.py` passes * `pants test src/python/pants/backend/python/dependency_inference/::` passes * Ran this build on our monorepo, which uses quite a tangle of every feature here * CI
This is a short-term workaround for #20577 and #20586, where Pants' internal/default use of Pex triggers user-visible warnings, and those warnings are now visible due to #20480... so, instead of showing them by default, let's hide them for now. Users with desire for insight into the warnings can still enable this. Doing this means we're not having to rush in fixes for the root causes of these warnings for 2.20.0 stable release, and thus reduce feature-creep/risk. We can/should reenable warnings by default in a future release. Per @cburroughs's idea in #20586 (comment), this explicitly switches it on for the Pants repo itself, so we're eating our own dogfood and catching real and/or spurious errors earlier.
…ation. (#20574) As discussed in #20572, this change adds a built-in goal `migrate-call-by-name` which emits all `Get` calls which should be migrated to call-by-name syntax. A followup change should use the resulting information (`@rule` function pointers and `Get` signatures) to execute rewrites of the relevant files. Emits lines like: ``` <function infer_python_conftest_dependencies at 0x113633af0> Get(<class 'pants.engine.target.Targets'>, [<class 'pants.engine.addresses.Addresses'>]) -> <function resolve_targets at 0x112335d30> Get(<class 'pants.engine.internals.graph.Owners'>, [<class 'pants.engine.internals.graph.OwnersRequest'>]) -> <function find_owners at 0x112349160> Get(<class 'pants.backend.python.util_rules.ancestor_files.AncestorFiles'>, [<class 'pants.backend.python.util_rules.ancestor_files.AncestorFilesRequest'>]) -> <function find_ancestor_files at 0x1131e6790> ```
Implemented for `fix` and `lint` goals.
… to target location (#20581) This PR fixes #20575 by adjusting the shared infrastructure for `adhoc_tool` and `code_quality_tool` to resolve the `execution_dependencies` field relative to the `adhoc_tool`/`code_quality_tool` target location, rather than relative to the `runnable=...` arg. I think this was a minor typo in #20255, and we didn't have tests covering it. I imagine it's common that the runnable and `..._tool` targets are often in the same BUILD file (in which case the behaviour is the same either way), but it's not impossible to have them be different (e.g. my work repo has a a few shared runnable that are used by more than one `adhoc_tool`). I think being relative to the target is easier to reason about, and this was the old behaviour of `adhoc_tool`. This PR also adds tests to confirm the behaviour of `execution_dependencies` (including its relative-path resolution behaviour), as well as the behaviour of `runnable_dependencies` while I'm there.
The `python_awslambda` target was replaced by `python_aws_lambda_function` in #19216, first released in 2.18.0.dev1 (the deprecation was extended in #20101). We're now up to 2.21 (#20609), and thus the deprecation has expired and this can be removed. This has been deprecated for 3 versions, and is easy for users to adapt to, so doesn't seem worth extending.
Title says it all. Just adding a new default module mapping. Signed-off-by: Jason Barnett <[email protected]>
…n=partial (#20616) There were three deprecations originally scheduled for 2.21 (#20609), two of will have only been deprecated for one release, and aren't a significant burden to support, so we can ease users life for a little longer: | description | deprecation started in | what could be removed | |------------------------------------------------------|------------------------|-------------------------------------------------------------------------------------------------| | the `[GLOBAL].remote_oauth_bearer_token_path` option | 2.20.0.dev1 (#20116) | configuration for the option, plus some a few small `if` statements, all in `global_options.py` | | passing `crossversion="partial"` for scala artifacts | 2.20.0 (#20264) ^ | an extra enum variant, and a test | ^ NB. due to an easy-to-make misuse of `start_version` I believe people will only be getting the deprecation warnings in 2.20.0 stable, not any of the release candidates. The third (`python_awslambda`) is removed in #20619.
Previously to pass along only/no-binary options Pants would create a
"requirements" file and use Pip's ability to put cli args in said file
[1]. This worked fine to generate the artifacts in a lockfile from
scratch, but meant that the lockfile itself did not contain sufficient
information to carry its own configuration forward. That is if one did
`pex3 lock update` on a Pants created lockfile that originally required
everything to use wheels, that specification could not be preserved in
the updated lockfile. Since v2.1.161 [2] Pex has provided
`--only-{wheel,build}` options to support this directly.
[1]
https://pip.pypa.io/en/stable/reference/requirements-file-format/#global-options
[2] https://github.com/pex-tool/pex/releases/tag/v2.1.161
ref #15704
`During a release, the contents of this directory is synced with our website repo (https://github.com/pantsbuild/pantsbuild.org) to be deployed.` Does this mean it's okay to edit the docs file in either the pants repo or the pantsbuild.rog repo?
Closes #20620. Added tests for both new fields; I couldn't figure out how to get the deprecated alias `skip_ruff` to still work in tests. I had another version of this change which kept the `skip_ruff` behavior as it is now to skip **both** linting and formatting; if that's a preferred solution, I can tack on those changes to this branch.
…ip_version = 24.0, for Py 3.12 support (#20365) This does three things to fix #20354 and give Pants Python 3.12 support by default: - update the default `[python].pip_version` value to `24.0` (from `23.1.2`), which is the current latest and is after 23.2, which is the first that supports Python 3.12 - run `pants run build-support/bin/generate_builtin_lockfiles.py -- --all-python` to update all tool lockfiles... this involves updating many tools, as a Big Bang, which is potentially unhelpful. - to make this work, `docformatter` has to be restricted to the current version 1.4: 1.5.1 (the newest version that satisfies the old constraints) crashes on Pants itself (PyCQA/docformatter#151), while 1.7.5 (the newest version) also doesn't work on Pants (#20498) Here's a list of (as best I can tell), the changes in versions for the "main" requirement for each subsystem: | subsystem options scope | main requirement | old version | new version | |-------------------------|------------------|-------------|-------------| | `helm-k8s-parser` | `hikaru` | 0.11.0b0 | 0.16.0b0 | | `helm-post-renderer` | `yamlpath` | 3.7.0 | 3.8.1 | | `autoflake` | `autoflake` | 2.0.1 | 2.1.1 | | `bandit` | `bandit` | 1.7.4 | 1.7.5 | | `black` | `black` | 23.1.0 | 23.3.0 | | `pyupgrade` | `pyupgrade` | 3.3.1 | 3.3.2 | | `ruff` | `ruff` | 0.2.1 | 0.2.2 | | `yapf` | `yapf` | 0.32.0 | 0.40.2 | | `coverage-py` | `coverage` | 7.2.1 | 7.2.7 | | `debugpy` | `debugpy` | 1.6.6 | 1.6.7.post1 | | `mypy` | `mypy` | 1.1.1 | 1.4.1 | | `terraform-hcl2-parser` | `python-hcl2` | 4.3.0 | 4.3.2 | | `yamllint` | `yamllint` | 1.29.0 | 1.32.0 | There's numerous other transitive dependencies updated too, including in subsystems for which the main requirement hasn't changed and aren't listed above. I haven't checked changelogs for any of these. I've confirmed that running the reproducer from #20354 with this code now works. The commits are 'sensible'.
We don't run tests on this platform in CI (and we only run a subset of tests on any platform other than Linux x86_64). Therefore some tests have rotted on that platform. This PR fixes them. See #20993 for more.
Previously we only provided a version for Linux x86_64.
`PATH` was silently overwritten by rule code.
Instead of picking the "shortest" provider available, pick the one closest to the thing being provided for a more accurate result. When a particular thing is available "out of backend" as a dependency, it will be listed as "activated by" the shortest plugin module path, for the docs site, all backends are enabled so everything will be activated by their primary backend. Meaning the online docs will be accurate, as well as the local CLI help will be accurate in reflecting which backend actually activated a particular feature. Replaces #21023. --------- Co-authored-by: Daniel Goldman <[email protected]>
…artial, remote_auth_bearer_token_path (#21041) We've branched for 2.22, so 2.23.0.dev0 is the next release on `main` (#21039), and thus this prepares for that by removes the deprecations that expire in that release: - `--changed-dependees` option being renamed to `--changed-dependents` - `dependees` goal being renamed to `dependents` - the old ruff backend path, from before splitting it into `.check` and `.format` backends - `scala_artifact`'s `crossversion="partial"` field value - `[GLOBAL].remote_auth_bearer_token_path` option being replaced by `remote_auth_bearer_token`
…ets (#21042) This PR captures two missing cases from migration. - Generator-based `MultiGet`s (very important) - `Get` calls within compound statements (helps with edge cases)
Update default pex version to 2.3.3 to pick up a fix: https://github.com/pex-tool/pex/releases/tag/v2.3.3, which can now generate proper URLs for artifacts on private PyPI. Lockfile diff ```shell Lockfile diff: 3rdparty/python/user_reqs.lock [python-default] == Upgraded dependencies == certifi 2024.2.2 --> 2024.6.2 cryptography 42.0.5 --> 42.0.8 exceptiongroup 1.2.0 --> 1.2.1 pex 2.3.1 --> 2.3.3 pluggy 1.4.0 --> 1.5.0 pygments 2.17.2 --> 2.18.0 requests 2.31.0 --> 2.32.3 ujson 5.9.0 --> 5.10.0 ```
This adds an pants.backend.tools.trufflehog backend that runs Trufflehog (https://github.com/trufflesecurity/trufflehog) secret scanning as a linter. ## Background TruffleHog is an open source secrets scanning tool that detects over 800 different types of secrets in a variety of sources, such as git repositories, local files, AWS S3, Docker images and more. It utilizes detector modules built for a large range of secret formats, and extracts matching data from plaintext files as well as rich text documents like PDFs. Then, it verifies the secret by checking the credential against the actual SaaS provider’s APIs, if available. ## What's implemented The backend implemented here is minimal, but enough to be functional: `pants lint --lint-only="trufflehog"`... in our repo ~identical to running it externally (except for more caching). Here's some features: 1. Adding a `trufflehog-config.yaml` file to the root of one's repo will result in that file being passed to `trufflehog --config trufflehog-config.yaml`. This file is used to configure additional detectors in trufflehog. 2. Certain file paths / folders can be excluded from trufflehog scanning via this format in `pants.toml` ``` [trufflehog] exclude = ["**/tests/*", "path/to/file.txt"] ``` 3. A generic `args` field can be supplied to append args to the trufflehog command execution, i.e. ``` [trufflehog] args = ["--exclude-detector DETECTOR"] ``` --------- Co-authored-by: Benjy Weinberger <[email protected]>
# The problem Imagine you have a toml file `tables.toml` with sql tables defined: ```toml [tables.users] id = "int" name = "string" [tables.companies] id = "int" name = "string" ``` Now you want to use these tables in 2 different jobs that process the data from the tables. You can write your own plugin that defines a new target `sql_table`: ```python sql_table(name='users', source='tables.toml', table='users') sql_table(name='companies', source='tables.toml', table='companies') ``` and then add dependencies to your code with `InferDependenciesRequest`. So far so good. Now imagine I add the company `address = "string"` in the toml: ```toml [tables.users] id = "int" name = "string" [tables.companies] id = "int" name = "string" address = "address" ``` Now if I do `pants --changed-since` it will show me that both `sql_table(name='users')` and `sql_table(name='companies')` has changed, but I only want `sql_table(name='companies')` to change. # Solution This pr adds logic to get information about changed line numbers from git and pass it to plugins. This pr does NOT implement the logic to figure out the targets that own these changed lines, this is delegated to plugins. Fine grained diff might be expensive to process, so it's hidden behind a flag. If you want to use it, you have to explicitly specify all the files that need it with `--changed-files-with-line-numbers`: ```bash pants --changed-since=origin/main --changed-files-with-line-numbers=tables.toml dependents ``` Then you need to inherit `BlockOwnersRequest` and add a rule for it: ```python @DataClass(frozen=True) class TomlBlockOwnersRequest(BlockOwnersRequest): pass @rule async def get_toml_block_owners(request: TomlBlockOwnersRequest) -> Owners: ... def rules(): return [ *collect_rules(), UnionRule(BlockOwnersRequest, TomlBlockOwnersRequest), ] ``` Here is [POC repo](https://github.com/grihabor/pants-playground/tree/fd633e87b84d0c048256778d1ac05be48c799585/changes-per-line) that shows how to use this.
This MR will resolve addresses passed to COPY through an ARG.
This MR also add inference of bare files.
The existing Docker integration allows copying `package` targets into
the docker container, but requires specifying the anticipated output
path. This is confusing. This MR also add inference of bare files.
For example, the following now works:
```Dockerfile
FROM python:3.9
ARG PEX_BIN="testprojects/src/python/hello/main:main"
ARG PEX_BIN_DOTTED_PATH="testprojects.src.python.native/main.pex"
COPY ${PEX_BIN} /app/pex_bin
COPY $PEX_BIN_DOTTED_PATH /app/pex_var
COPY testprojects.src.python.native/main.pex /app/pex_bin_dotted_path
```
This MR also add inference of bare files. I thought that the docs were
broken but I just missed that you needed a manual link but I already
implemented it so why not push.
```Dockerfile
FROM python:3.9
COPY testprojects/src/docker/file.txt /app/
```
closes #20718
The `tailor` goal now has independent options for tailoring `shell_sources` and `shunit2_tests` targets. The option was split from "tailor" into "tailor_sources" and "tailor_shunit2_tests". The "tailor" option is now deprecated. For the duration, setting it to False will disable tailor for both source types. If set to True (the default), the other 2 options will be consulted. fixes #20882 --------- Co-authored-by: Huon Wilson <[email protected]>
First pass of the `migrate-call-by-name` goal, followed by a manual removal of `implicitly` calls.
Closes #21045 One open question on my end: how are negative paths e.g. `output_directories=(".", "!but_not_me")` handled in this?
…1060) The IPython/repl docs (https://www.pantsbuild.org/2.21/docs/python/goals/repl) refer to options that no longer exist, like `version`. This PR rewrites them for the new `install_from_resolve` equivalent. Process things: - Doesn't seem worth calling out in release notes - Just a doc improvement from many versions ago, so can cherry-pick to have it released sooner Rendered: 
…ds (#21051) Add support for "workspace invalidation" sources for the `adhoc_tool` and `shell_command` target types. This supports allows those targets to depend on the content of files in the repository without materializing those sources in the execution sandbox. This support is intended to be used in conjunction with the workspace environment where execution does not take place in a sandbox. The new field `workspace_invalidation_sources` on both target types is a list of globs into the repository. The digest of the referenced files will be inserted as an environment variable in the process executed (which makes it part of the process's cache key).
Scala parser now understands `_root_` as a marker for an absolute package import/qualified name.
Changes from 0.4.4 to 0.4.9 include: * A new Ruff language server that can be used for local development * Format and lint rule updates * Some rule changes to start supporting Python 3.13 features
The start_release.py script modifies workspace files, and so will race with the file watcher to complete before being killed and restarted. And since the process is not idempotent (e.g., it creates a local git branch) that restarted process can fail.
This fixes the feature added in #20497 as it broke using `pants run` on a python_source if the file name has `-` or other invalid characters. Bug reported here: https://pantsbuild.slack.com/archives/C046T6T9U/p1717624913138789?thread_ts=1717624913.138789&cid=C046T6T9U The other integration test for the pex --executable feature only tested running pex_binary, not python_source. So, I missed applying some of the path logic to both cases.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes the feature added in #20497 as it broke using
pants runon a python_source if the file name has-or other invalid characters. Bug reported here:https://pantsbuild.slack.com/archives/C046T6T9U/p1717624913138789?thread_ts=1717624913.138789&cid=C046T6T9U
The other integration test for the pex --executable feature only tested running pex_binary, not python_source. So, I missed applying some of the path logic to both cases.