Uploads throttled while using token-based upload from public repo

[mattwthompson]

Here's the relevant snippet from a log:

evenName: schedule
evenName: schedule
evenName: schedule
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Signature made Fri Aug 16 23:23:14 [20](https://github.com/openforcefield/openff-toolkit/actions/runs/10840617964/job/30116692249#step:19:21)24 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C  62FF 806B B28A ED77 9869
==> Uploader SHASUM verified (eb902009cb800fdefbce291c1357953f80e29382f43c7a98cce3b57d2b4d8cec  codecov)
==> Running version latest
==> Running version v0.7.4
==> Running git config --global --add safe.directory /home/runner/work/openff-toolkit/openff-toolkit
/usr/bin/git config --global --add safe.directory /home/runner/work/openff-toolkit/openff-toolkit
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit --git-service github -Z
info - 2024-09-13 15:36:47,530 -- ci service found: github-actions
info - 2024-09-13 15:36:47,7[22](https://github.com/openforcefield/openff-toolkit/actions/runs/10840617964/job/30116692249#step:19:23) -- Process Commit creating complete
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report --git-service github -Z
info - 20[24](https://github.com/openforcefield/openff-toolkit/actions/runs/10840617964/job/30116692249#step:19:25)-09-13 15:36:48,477 -- ci service found: github-actions
info - 2024-09-13 15:36:48,652 -- Process Report creating complete
info - 2024-09-13 15:36:48,652 -- Finished creating report successfully --- {"response": "{\"code\":null}"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload -Z -f ./coverage.xml --git-service github
info - 2024-09-13 15:36:49,400 -- ci service found: github-actions
warning - 2024-09-13 15:36:49,411 -- xcrun is not installed or can't be found.
warning - 2024-09-13 15:36:49,465 -- No gcov data found.
warning - 2024-09-13 15:36:49,465 -- coverage.py is not installed or can't be found.
info - 2024-09-13 15:36:49,673 -- Found 2 coverage files to report
info - 2024-09-13 15:36:49,673 -- > /home/runner/work/openff-toolkit/openff-toolkit/coverage.xml
info - 2024-09-13 15:36:49,673 -- > /home/runner/work/openff-toolkit/openff-toolkit/examples/deprecated/check_dataset_parameter_coverage/check_parameter_coverage.ipynb
info - 2024-09-13 15:36:49,8[33](https://github.com/openforcefield/openff-toolkit/actions/runs/10840617964/job/30116692249#step:19:34) -- Process Upload complete
error - 2024-09-13 15:36:49,834 -- Upload failed: {"detail":"Request was throttled."}
Error: Codecov:
                        Failed to properly upload report: The process '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1

Things I've thought about:

• There haven't been commits to this repo in the better part of a month and we have cron jobs that run the latest main branch nightly. This results in many uploads from the same commit, which has been throttled in the past. However, this was previously made clear in the error and all I'm seeing here is a non-descript failure and exit code 1.
• We're only specifying @v4 in the action, which is the latest major version, but I'm pretty sure that's suposed to bring down the latest 4.x.x anyway.
• Maybe it's getting choked up on a different file that happens to have coverage in its name? It's a notebook and obviously not storing code coverage data, but that file has been there for years without issue. I could exclude: it, but I'm already specifying file: coverage.xml so it's surprising that it's even looking elsewhere ... ?
• This is a public repo and it is not a tokenless upload.
• Maybe something got funky with the token, so I re-generated it, stored the new one in the repo settings, and re-started CI. Same result.
• coverage.py is indeed not installed - but a package named coverage is, and pytest ran with a bunch of coverage-related options. I'd expect this to be a problem if the coverage report wasn't generated, but I think it is being generated.

[mmoayyed]

I am seeing the same problem, with a public repository and a valid token defined:

Run codecov/codecov-action@v4
evenName: workflow_dispatch
evenName: workflow_dispatch
evenName: workflow_dispatch

gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Signature made Fri Aug 16 23:23:14 2024 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C  62FF 806B B28A ED77 9869
==> Uploader SHASUM verified (eb902009cb800fdefbce291c1357953f80e29382f[43](https://github.com/mmoayyed/cas/actions/runs/10908163834/job/30273568539#step:9:44)c7a98cce3b57d2b4d8cec  codecov)
==> Running version latest
==> Running version v0.7.4
==> Running git config --global --add safe.directory /home/runner/work/cas/cas
/usr/bin/git config --global --add safe.directory /home/runner/work/cas/cas
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v create-commit --git-service github
info - 2024-09-17 17:50:14,290 -- ci service found: github-actions
debug - 2024-09-17 17:50:14,293 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-09-17 17:50:14,296 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-09-17 17:50:14,299 -- Loading config from /home/runner/work/cas/cas/.github/codecov.yml
debug - 2024-09-17 17:50:14,301 -- Starting create commit process --- {"commit_sha": "df33fa7e14cb174e71d733179573259ef5d5b5af", "parent_sha": null, "pr": null, "branch": "master", "slug": "***/cas", "token": "2******************", "service": "github", "enterprise_url": null}
info - 2024-09-17 17:50:14,422 -- Process Commit creating complete
debug - 2024-09-17 17:50:14,423 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=201, text='{\"message\":\"make sure amqp messages are persistent; queues are durable\",\"timestamp\":\"2024-09-17T06:30:07Z\",\"ci_passed\":true,\"state\":\"complete\",\"repository\":{\"name\":\"cas\",\"is_private\":false,\"active\":true,\"language\":\"java\",\"yaml\":{\"fixes\":[\"/home/runner/work/cas/cas/::\"],\"codecov\":{\"max_report_age\":false,\"strict_yaml_branch\":\"master\"},\"coverage\":{\"range\":[80.0,90.0],\"round\":\"up\",\"status\":{\"project\":{\"default\":{\"base\":\"auto\",\"flags\":[\"unit\"],\"paths\":[\"^src.*\"],\"target\":90.0,\"branches\":[\"^master$\"],\"threshold\":0.0,\"only_pulls\":false,\"if_ci_failed\":\"error\",\"if_not_found\":\"success\",\"informational\":false}}},\"precision\":5}}},\"author\":{\"avatar_url\":\"https://avatars0.githubusercontent.com/u/1205228?v=3&s=55\",\"service\":\"github\",\"username\":\"***\",\"name\":\"Misagh Moayyed\",\"ownerid\":329824},\"commitid\":\"df33fa7e14cb174e71d733179573259ef5d5b5af\",\"parent_commit_id\":\"e8011e7b73b76d69a394acc53387860361465c46\",\"pullid\":null,\"branch\":\"master\"}')"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v create-report --git-service github
info - 2024-09-17 17:50:15,176 -- ci service found: github-actions
debug - 2024-09-17 17:50:15,180 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-09-17 17:50:15,182 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-09-17 17:50:15,185 -- Loading config from /home/runner/work/cas/cas/.github/codecov.yml
debug - 2024-09-17 17:50:15,187 -- Starting create report process --- {"commit_sha": "df33fa7e14cb174e71d733179573259ef5d5b5af", "code": "default", "slug": "***/cas", "service": "github", "enterprise_url": null, "token": "2******************"}
info - 2024-09-17 17:50:15,293 -- Process Report creating complete
debug - 2024-09-17 17:50:15,293 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=201, text='{\"code\":null}')"}
info - 2024-09-17 17:50:15,294 -- Finished creating report successfully --- {"response": "{\"code\":null}"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v do-upload -f ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml -F amqp --git-service github -n amqp
info - 2024-09-17 17:50:16,039 -- ci service found: github-actions
debug - 2024-09-17 17:50:16,042 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-09-17 17:50:16,0[44](https://github.com/mmoayyed/cas/actions/runs/10908163834/job/30273568539#step:9:45) -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-09-17 17:50:16,047 -- Loading config from /home/runner/work/cas/cas/.github/codecov.yml
debug - 2024-09-17 17:50:16,050 -- Starting upload processing --- {"branch": "master", "build_code": "10908163834", "build_url": "https://github.com/***/cas/actions/runs/10908163834", "commit_sha": "df33fa7e14cb174e71d733179573259ef5d5b5af", "disable_file_fixes": false, "disable_search": false, "enterprise_url": null, "env_vars": {}, "files_search_exclude_folders": [], "files_search_explicitly_listed_files": ["build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml"], "files_search_root_folder": "/home/runner/work/cas/cas", "flags": ["amqp"], "git_service": "github", "handle_no_reports_found": false, "job_code": "Unit & Integration Tests", "name": "amqp", "network_filter": null, "network_prefix": null, "network_root_folder": "/home/runner/work/cas/cas", "plugin_names": ["xcode", "gcov", "pycoverage"], "pull_request_number": null, "report_code": "default", "slug": "***/cas", "token": "2******************", "upload_file_type": "coverage"}
debug - 2024-09-17 17:50:16,051 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"]}
debug - 2024-09-17 17:50:16,051 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-09-17 17:50:16,051 -- Running xcode plugin...
warning - 2024-09-17 17:50:16,051 -- xcrun is not installed or can't be found.
debug - 2024-09-17 17:50:16,051 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-09-17 17:50:16,051 -- Running gcov plugin...
warning - 2024-09-17 17:50:17,136 -- No gcov data found.
debug - 2024-09-17 17:50:17,136 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-09-17 17:50:17,136 -- coverage.py is not installed or can't be found.
debug - 2024-09-17 17:50:17,136 -- Collecting relevant files
info - 2024-09-17 17:50:19,952 -- Found 1 coverage files to report
info - 2024-09-17 17:50:19,952 -- > /home/runner/work/cas/cas/build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml
debug - 2024-09-17 17:50:20,061 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-09-17 17:50:20,2[46](https://github.com/mmoayyed/cas/actions/runs/10908163834/job/30273568539#step:9:47) -- Sending upload request to Codecov
info - 2024-09-17 17:50:20,350 -- Process Upload complete
debug - 2024-09-17 17:50:20,350 -- Upload result --- {"result": "RequestResult(error=RequestError(code='HTTP Error 429', params={}, description='{\"detail\":\"Request was throttled.\"}'), warnings=[], status_code=429, text='{\"detail\":\"Request was throttled.\"}')"}
error - 2024-09-17 17:[50](https://github.com/mmoayyed/cas/actions/runs/10908163834/job/30273568539#step:9:51):20,350 -- Upload failed: {"detail":"Request was throttled."}

Setup:

      - name: "Upload to Codecov"
        uses: "codecov/codecov-action@v4"
        if: env.CODECOV_TOKEN != null
        with:
          token: ${{ env.CODECOV_TOKEN }}
          files: ./build/reports/jacoco/jacocoRootReport/jacocoRootReport.xml
          flags: ${{ matrix.category }}
          name: ${{ matrix.category }}
          verbose: true

[pete-resim]

Same error:

Run codecov/codecov-action@v4
  with:
    files: coverage.out
    verbose: true
evenName: pull_request
baseRef: resim-ai:main | headRef: resim-ai:pete/more-coverage-fixing
evenName: pull_request
baseRef: resim-ai:main | headRef: resim-ai:pete/more-coverage-fixing
evenName: pull_request
baseRef: resim-ai:main | headRef: resim-ai:pete/more-coverage-fixing
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
Received SHA256SUM REDACTED  codecov
Received SHA256SUM signature -----BEGIN PGP SIGNATURE-----
REDACTED
-----END PGP SIGNATURE-----

gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Signature made Fri Aug 16 23:23:14 2024 UTC
gpg:                using RSA key REDACTED
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: REDACTED
==> Uploader SHASUM verified (eb902009cb800fdefbce291c1357953f80e29382f43c7a98cce3b57d2b4d8cec  codecov)
==> Running version latest
==> Running version v0.7.4
==> Running git config --global --add safe.directory /home/runner/work/rerun/rerun
/usr/bin/git config --global --add safe.directory /home/runner/work/rerun/rerun
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov -v create-commit --git-service github -C e8ead5241664509664d94717f5e126809aef5591
info - 2024-09-19 18:20:34,933 -- ci service found: github-actions
debug - 2024-09-19 18:20:34,936 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
debug - 2024-09-19 18:20:34,938 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.NoVersioningSystem'>
warning - 2024-09-19 18:20:34,939 -- No config file could be found. Ignoring config.
debug - 2024-09-19 18:20:34,939 -- No codecov_yaml found
debug - 2024-09-19 18:20:34,940 -- Starting create commit process --- {"commit_sha": "e8ead5241664509664d94717f5e126809aef5591", "parent_sha": null, "pr": "1212", "branch": "pete/more-coverage-fixing", "slug": "resim-ai/rerun", "token": null, "service": "github", "enterprise_url": null}
Error: Codecov token not found. Please provide Codecov token with -t flag.
Warning: Codecov: Failed to properly create commit: The process '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov' failed with exit code 1

Minimal GHA workflow:

---
name: Coverage
on:
  workflow_call: {}

jobs:
  coverage:
    name: Upload Coverage Report
    runs-on: ubuntu-20.04
    steps:
      - name: Generate coverage
        run: |
          echo "mode: set" > coverage.out
          echo "github.com/username/repo/file1.go:10.32,15.2 3 1" >> coverage.out
          echo "github.com/username/repo/file1.go:17.36,20.2 2 0" >> coverage.out
          echo "github.com/username/repo/file2.go:5.45,8.2 2 1" >> coverage.out
          echo "github.com/username/repo/file2.go:10.40,12.2 1 1" >> coverage.out
          echo "github.com/username/repo/file2.go:14.55,16.2 1 0" >> coverage.out

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: coverage.out
          verbose: true

[adcroft]

We have this same problem (CI for a push will upload to codecov but ci for a pull requests will not). I noticed that in the "Setup" part of the log (very top) the push job has

Secret source: Actions

but the pull request job has

Secret source: None

I think this means the token is indeed not made available. I'm not sure how to work around this, nor why this is not an issue for everyone.