Draft
Conversation
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
dosubot
bot
added
the
size:XXL
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4952 +/- ##
==========================================
+ Coverage 66.80% 69.67% +2.87%
==========================================
Files 230 251 +21
Lines 16602 15654 -948
==========================================
- Hits 11091 10907 -184
+ Misses 4808 3873 -935
- Partials 703 874 +171 🚀 New features to boost your workflow:
|
justaugustus
commented
Feb 27, 2026
Member
Author
justaugustus
left a comment
justaugustus
left a comment
There was a problem hiding this comment.
@ossf/scorecard-maintainers @ossf/scorecard-fe-maintainers @eddie-knight @puerco @evankanderson @mlieberman85 — based on conversations from this week with various WG ORBIT-adjacent maintainers, I'm tossing this early draft up for review.
Feel free to comment away while I work through this!
- Add AGENTS.md with project overview, build/test commands, architecture
guide, contribution conventions, and AI agent collaboration guidelines
(co-authorship trailer, OpenSpec workflow, git hygiene rules)
- Bootstrap openspec/ directory structure with initial specs:
- openspec/specs/platform-clients/spec.md: platform client abstraction
- openspec/changes/pvtr-integration/specs/pvtr-baseline/spec.md:
OSPS Baseline integration requirements and scenarios
- Incorporate guidance from the OSPO Engineering Playbook into AGENTS.md
Signed-off-by: Stephen Augustus <foo@auggie.dev>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Stephen Augustus <foo@auggie.dev>
The scope of this work is OSPS Baseline conformance within the ORBIT ecosystem — Privateer/PVTR interoperability is one aspect, not the whole story. Signed-off-by: Stephen Augustus <foo@auggie.dev> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Complete rewrite of the proposal and spec to cover the full scope of the 2026 roadmap, not just Privateer/PVTR interoperability: - Conformance engine producing PASS/FAIL/UNKNOWN/NOT_APPLICABLE/ATTESTED - OSPS output format (--format=osps) - Versioned control-to-probe mapping files - Applicability engine for precondition detection - Security Insights ingestion for ORBIT ecosystem interop - Attestation mechanism for non-automatable controls - Gemara Layer 4 compatibility - CI gating support - Phased delivery aligned with quarterly milestones - ORBIT ecosystem positioning (complement PVTR, don't duplicate) Highlights Spencer's review notes as numbered open questions (OQ-1 through OQ-4): - OQ-1: Attestation identity model (OIDC? tokens? workflows?) - OQ-2: Enforcement detection vs. being an enforcement tool - OQ-3: scan_scope field usefulness in output schema - OQ-4: Evidence should be probe-based only, not check-based Renames spec subdirectory from pvtr-baseline to osps-conformance. Signed-off-by: Stephen Augustus <foo@auggie.dev> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
- Add openspec/specs/core-checks/spec.md and openspec/specs/probes/spec.md documenting existing Scorecard architecture for spec-driven development - Update .gitignore to exclude roadmap drafting notes Signed-off-by: Stephen Augustus <foo@auggie.dev> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Stephen's responses to clarifying questions (CQ-1 through CQ-8) and feedback on the proposal draft: - Both scoring and conformance modes coexist; no deprecation needed now - Target OSPS Baseline v2026.02.19 (latest), align with maintenance cadence - Provide degraded-but-useful evaluation without Security Insights - Invest in Gemara SDK integration for multi-tool consumption - Prioritize Level 1 conformance; consume external signals where possible - Approval requires Stephen + Spencer + 1 non-Steering maintainer - Q2 outcome should be OSPS Baseline Level 1 conformance - Land capabilities across all surfaces (CLI, Action, API) Key changes requested: - Correct PVTR references (it's the Privateer plugin, not a separate tool) - Add Darnit and AMPEL comparison - Replace quarterly timelines with phase-based outcomes - Plan to extract Scorecard's control catalog for other tools - Use Mermaid for diagrams - Create separate OSPS Baseline coverage analysis in docs/ - Create docs/ROADMAP.md for public consumption Signed-off-by: Stephen Augustus <foo@auggie.dev> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Changes based on Stephen's review: - Replace all "PVTR" references with "Privateer plugin for GitHub repositories" — it's the Privateer plugin, not a separate tool - Add ecosystem tooling comparison section covering Darnit (compliance audit + remediation), AMPEL (attestation-based policy enforcement), Privateer plugin (Baseline evaluation), and Scorecard (measurement) - Replace quarterly timeline (Q1-Q4) with phase-based delivery (Phase 1-3) focused on outcomes, not calendar dates - Update OSPS Baseline version from v2025-10-10 to v2026.02.19 - Convert ASCII ecosystem diagram to Mermaid - Add Scorecard control catalog extraction to scope - Add Gemara SDK integration to scope - Update coverage snapshot to reference docs/osps-baseline-coverage.md (to be created with fresh analysis) - Add approval process section based on governance answers - Update Security Insights requirement to degraded-but-useful mode - Add integration pipeline diagram (Scorecard -> Darnit -> AMPEL) Signed-off-by: Stephen Augustus <foo@auggie.dev> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Create docs/osps-baseline-coverage.md with a control-by-control analysis of Scorecard's current probe coverage against OSPS Baseline v2026.02.19. Coverage summary: 8 COVERED, 17 PARTIAL, 31 GAP, 3 NOT_OBSERVABLE across 59 controls. Create docs/ROADMAP.md with a publicly-consumable 2026 roadmap organized into three phases: conformance foundation + Level 1, release integrity + Level 2, and enforcement detection + Level 3. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
…h CQ-12 Remove reference to docs/roadmap-ideas.md from the coverage analysis document since it is not committed to the repo. Add four new clarifying questions to the proposal: NOT_OBSERVABLE controls in Phase 1 (CQ-9), mapping file ownership (CQ-10), OSPS output schema stability guarantees (CQ-11), and Phase 1 probe gap prioritization (CQ-12). Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Replace \n with <br/> in Mermaid node labels so line breaks render correctly in GitHub's Mermaid renderer. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Replace remaining "Darn" references with "Darnit" throughout the proposal. Add Minder to the ecosystem comparison table, integration diagram, and "What Scorecard must not do" section. Minder is an OpenSSF Sandbox project in the ORBIT WG that consumes Scorecard findings for policy enforcement and auto-remediation. Add CQ-13 (Minder integration surface) and CQ-14 (Darnit vs. Minder delineation) as new clarifying questions. Update docs/ROADMAP.md ecosystem alignment to include Minder. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Add a new section to docs/osps-baseline-coverage.md listing existing Scorecard issues and PRs that are directly relevant to closing OSPS Baseline coverage gaps, including: - ossf#2305 / ossf#2479 (Security Insights) - #30 (secrets scanning) - ossf#1476 / ossf#2605 (SBOM) - ossf#4824 (changelog) - ossf#2465 (private vulnerability reporting) - ossf#4080 / ossf#4823 / ossf#2684 / ossf#1417 (signed releases) - ossf#2142 (threat model) - ossf#4723 (Minder/Rego integration, closed) Add CQ-15 asking whether existing issues should be adopted as Phase 1 work items or whether new issues should reference them. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Remove openspec system specs (core-checks, platform-clients, probes) that were scaffolding for documenting existing Scorecard architecture. These are not part of the OSPS conformance proposal and can be recreated if needed. Remove docs/roadmap-ideas.md from .gitignore. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
justaugustus
changed the title
justaugustus
force-pushed
the
roadmap-baseline
branch
from
f0b229d to
afe4c8d
Compare
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
Add Allstar (Scorecard sub-project) to the ecosystem comparison table, integration flow diagram, and ORBIT ecosystem diagram. Allstar continuously monitors GitHub orgs and enforces Scorecard checks as policies with auto-remediation, and already enforces controls aligned with OSPS Baseline (branch protection, security policy, binary artifacts, dangerous workflows). Add Allstar to "Existing Scorecard surfaces that matter" section and to docs/ROADMAP.md ecosystem alignment. Add CQ-16 asking whether Allstar should be an explicit Phase 1 consumer of OSPS conformance output, and whether it is considered part of the enforcement boundary Scorecard does not cross. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
Update the commit guidelines to make the -s flag requirement unambiguous. Add a complete commit message format example showing how to combine the HEREDOC pattern with -s for DCO sign-off and the Co-Authored-By trailer. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
justaugustus
force-pushed
the
roadmap-baseline
branch
from
4f5c6ce to
bd76d94
Compare
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
Contributor
|
Hey @justaugustus, thanks for leading this collaboration! Looking forward to hammering this out. Some things to clarify:
An alternative plan would be to for us to spend a week consolidating checks/probes into the pvtr plugin (with relevant CODEOWNERS), then update Scorecard to selectively execute the plugin under the covers. This would allow us to:
|
Add "ORBIT WG feedback" section documenting Eddie Knight's feedback from PR ossf#4952. Eddie is the ORBIT WG TSC Chair and maintainer of Gemara, Privateer, and OSPS Baseline. Five feedback items documented as EK-1 through EK-5: - EK-1: Mapping file could live in Baseline repo with CODEOWNERS - EK-2: No "OSPS output format" exists; use Gemara SDK formats - EK-3: Current proposal duplicates Privateer despite stating otherwise - EK-4: Catalog extraction needs concrete implementation plan - EK-5: Alternative architecture — shared plugin model Add five new clarifying questions (CQ-17 through CQ-21) for Steering Committee decisions: - CQ-17: Mapping file location (Scorecard repo vs shared) - CQ-18: Output format (--format=osps vs Gemara SDK) - CQ-19: Build vs integrate (own engine vs shared plugin) - CQ-20: Catalog extraction scope - CQ-21: Privateer code duplication acceptability Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Stephen Augustus <foo@auggie.dev>
justaugustus
temporarily deployed
to
integration-test
GitHub Actions
Inactive
Member
Author
|
@eddie-knight — Thanks for the thoughtful response and yes! Looking forward to working on this with you all! I've integrated your feedback as open questions in 2ee759f. Can you take a quick look and see if that commit accurately captures your questions and concerns before I continue? |
These are clarifying edits to ensure we've captured the recommendations and open questions correctly. Co-authored-by: Eddie Knight <knight@linux.com> Signed-off-by: Stephen Augustus <justaugustus@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
(Is it a bug fix, feature, docs update, something else?)
OSPS Baseline Conformance
Scorecard's primary initiative for 2026 is adding OSPS Baseline conformance evaluation, enabling Scorecard to answer the question: does this project meet the security requirements defined by the OSPS Baseline at a given maturity level?
This is a new product surface alongside Scorecard's existing 0-10 scoring model. Existing checks, probes, and scores are unchanged. The conformance layer consumes existing Scorecard signals and adds a per-control PASS/FAIL/UNKNOWN/NOT_APPLICABLE/ATTESTED output aligned with the ORBIT WG ecosystem.
The goal of this PR is to create a collaboration/decision-making nexus for Scorecard and WG ORBIT tooling maintainers to ensure that we build interfaces that easily interact with other tools and minimize duplication of work across our maintainers and others in the OpenSSF ecosystem.
What is the current behavior?
What is the new behavior (if this is a feature change)?**
Which issue(s) this PR fixes
Special notes for your reviewer
Does this PR introduce a user-facing change?
For user-facing changes, please add a concise, human-readable release note to
the
release-note(In particular, describe what changes users might need to make in their
application as a result of this pull request.)