🌱 Vulnerabilities check: New probe: Do releases have direct dependencies with known vulnerabilities?

[AdamKorcz]

What kind of change does this PR introduce?

New check

• PR title follows the guidelines defined in our pull request documentation

• Tests for the changes have been added (for bug fixes/features)

Special notes for your reviewer

This PR adds a new probe for the Vulnerabilities check which analyzes a projects release history and looks for vulnerabilities in the direct dependencies of a project that were known at the time of release. A project can score 10 if none of its max 10 most recent releases had known vulnerabilities at the time of each release. This probe will encourage projects to remove all known vulnerabilities from its direct dependencies which will lead to more secure products for downstream users. In addition, some regulated industries such as critical infrastructure require that releases don't contain known vulnerabilities.

Under the hood, the probe works like this:

1. First, the probe collects the commits of each of the projects last 10 releases.
2. Next, the probe collects the tar balls of each of the 10 commits.
3. The probe then downloads each tar ball.
4. With each tar ball, the probe scans the projects direct dependencies using osv-scalibr.
5. After collecting all the dependencies, the probe sends a batched query to osv.dev with the library names and versions.
6. The probe analyzes the responses from osv.dev to see if they contain vulnerabilities that were published before the release was issued.
7. The probe scores based on its findings.

Later, we can add a mechanism that checks if projects have SBOMs that describe whether a project is actually affected by known vulnerabilities as a way to give projects more control.

To test this with debugging: RELEASES_DEPS_DEBUG=1 SCALIBR_DEBUG=1 go run main.go --repo=https://github.com/owner/repo --checks=ReleasesDirectDepsVulnFree

How this differs from Vulnerabilities's existing probe

Currently the Vulnerabilities check has a single probe which checks whether a project has vulnerabilities at the time of analysis. The probe in this PR checks whether direct dependencies in releases contain vulnerabilities that were known at the time of release.

Does this PR introduce a user-facing change?

Yes

Add new check: Direct dependencies in releases are free of known vulnerabilities at the time of release.

[codecov]

Codecov Report

❌ Patch coverage is 59.23345% with 234 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.73%. Comparing base (353ed60) to head (20263fb).
⚠️ Report is 303 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4876      +/-   ##
==========================================
+ Coverage   66.80%   67.73%   +0.92%     
==========================================
  Files         230      255      +25     
  Lines       16602    16231     -371     
==========================================
- Hits        11091    10994      -97     
+ Misses       4808     4354     -454     
- Partials      703      883     +180     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[spencerschrock]

• With each tar ball, the check scans the projects direct dependencies using osv-scalibr.
• After collecting all the dependencies, the check sends a batched query to osv.dev with the library names and versions.
• The check analyzes the responses from osv.dev to see if they contain vulnerabilities that were published before the release was issued.

Any reason why you dont use osv-scanner for steps 4-5? Which is essentially a wrapper over osv-scalibr and osv.dev already? I'm also not sure of the limitation to direct dependencies here.

I think there are obviously similarities to the Vulnerabilities check, but instead of "are there any vulns now", the question is "Does this project have some process to avoid vulns when releases are made". Is that enough of a difference to warrant a separate check?

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity

[AdamKorcz]

Any reason why you dont use osv-scanner for steps 4-5? Which is essentially a wrapper over osv-scalibr and osv.dev already? I'm also not sure of the limitation to direct dependencies here.

I will look into the feasibility of that.

I think there are obviously similarities to the Vulnerabilities check, but instead of "are there any vulns now", the question is "Does this project have some process to avoid vulns when releases are made". Is that enough of a difference to warrant a separate check?

IMO this should fit into the Vulnerabilities check. I think the main hurdle for that is to decide how the scoring should be changed.

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity

[github-actions]

This pull request has been marked stale because it has been open for 10 days with no activity