[syntax-error-quality] Daily Syntax Error Quality Check - Setup Required

[github-actions]

🔍 Workflow Analysis Status

Analysis Date: 2026-02-12
Status: ⚠️ Setup Required
Workflow: Daily Syntax Error Quality Check

---

Executive Summary

The Daily Syntax Error Quality Check workflow is designed to test compiler error message quality by introducing syntax errors into workflows and evaluating the error output. However, the current workflow environment lacks the necessary gh-aw compiler binary to perform the analysis.

Key Issues:

• gh-aw compiler binary not available in workflow PATH
• Cannot build binary due to permission restrictions
• Unable to compile test workflows and capture error messages
• Quality analysis cannot be completed without compiler access

---

Required Setup

To enable this workflow to function correctly, the following setup is required:

1. Add Workflow Setup Steps

The workflow needs to include steps that build or install the gh-aw binary before the agent runs:

steps:
  - name: Setup Go
    uses: actions/setup-go@v5
    with:
      go-version: '1.23'
      
  - name: Build gh-aw
    run: |
      cd /home/runner/work/gh-aw/gh-aw
      make build
      
  - name: Add to PATH
    run: |
      echo "$GITHUB_WORKSPACE" >> $GITHUB_PATH

2. Grant Build Permissions

Ensure the workflow has permissions to:

• Read repository contents
• Build Go binaries
• Write to temporary directories

3. Verify Binary Access

Add a verification step to ensure the binary is accessible:

steps:
  - name: Verify gh-aw installation
    run: |
      which gh-aw || echo "gh-aw not in PATH"
      ./gh-aw --version || echo "gh-aw binary not executable"

---

Analysis Design

Once the setup is complete, the workflow should execute this analysis process:

Phase 1: Select Test Workflows

Select 3 diverse workflows for testing:

1. Simple workflow (< 100 lines, minimal config)
2. Medium workflow (100-300 lines, moderate config)
3. Complex workflow (> 300 lines, many tools/features)

Criteria:

• Avoid daily-* and test workflows
• Choose different complexity levels
• Prefer different engines, tools, safe-outputs configurations

Phase 2: Introduce Syntax Errors

Create 3 test cases with different error categories:

Category A: Frontmatter Syntax Errors

engine copilot  # Missing colon after key

Category B: Configuration Errors

engine: copiilot  # Typo in engine name

Category C: Semantic Errors

tools:
  github:
    lockdown: true
    toolsets: [default]  # Conflicting: lockdown with toolsets

Phase 3: Compile and Capture Output

For each test case:

./gh-aw compile test-workflow.md 2>&1 | tee test-output.txt

Phase 4: Evaluate Error Quality

Score error messages across 5 dimensions:

Dimension	Weight	Criteria
Clarity	25 pts	Clear, non-technical language; obvious what went wrong
Actionability	25 pts	Specific suggestions, steps to fix, documentation links
Context	20 pts	Shows problematic code, highlights error location
Examples	15 pts	Provides correct usage examples
Consistency	15 pts	IDE-parseable format (file:line:column), consistent terminology

Quality Threshold: Average score ≥ 70 across all test cases

Phase 5: Create Issue with Findings

Only create an issue if:

• Average score < 70 across all test cases, OR
• Any individual test case scores < 55, OR
• Critical pattern issues are identified

---

Example Test Workflows

View Test Case Examples

Test 1: Frontmatter Syntax Error (Simple Workflow)

File: test-1-frontmatter-syntax.md

---
description: Test workflow
on:
  issues:
    types: [opened]
engine copilot  # ERROR: Missing colon
tools:
  github:
    toolsets: [default]
---

# Test Workflow
``````

**Expected Error Output**:
``````
test-1-frontmatter-syntax.md:8:7: error: invalid YAML syntax
  8 | engine copilot
    |       ^ expected ':' after key

Valid engines: copilot, claude, codex, custom

Correct usage:
  engine: copilot

Test 2: Invalid Engine Name (Medium Workflow)

File: test-2-invalid-engine.md

---
description: Test workflow
on:
  workflow_run:
    workflows: [CI]
engine: copiilot  # ERROR: Typo in engine name
tools:
  github:
    toolsets: [default]
---

# Test Workflow
``````

**Expected Error Output**:
``````
test-2-invalid-engine.md:6:9: error: invalid engine 'copiilot'

Valid engines: copilot, claude, codex, custom

Did you mean: copilot?

Correct usage:
  engine: copilot

Test 3: Conflicting Configuration (Complex Workflow)

File: test-3-conflicting-config.md

---
description: Test workflow
on:
  slash_command:
    name: test
engine: copilot
tools:
  github:
    lockdown: true
    toolsets: [default]  # ERROR: Conflicting with lockdown
---

# Test Workflow
``````

**Expected Error Output**:
``````
test-3-conflicting-config.md:9:5: error: conflicting configuration

github.lockdown and github.toolsets cannot be used together

When lockdown is enabled:
  - No GitHub tools are available
  - Remove 'toolsets' field
  
When using toolsets:
  - Remove 'lockdown: true'
  - Specify allowed toolsets: [repos, issues, pull_requests]

Choose one approach:
  # Lockdown mode (no GitHub access)
  github:
    lockdown: true
    
  # Selective access
  github:
    toolsets: [repos, issues]

---

Success Criteria

A successful implementation will:

• ✅ Build or install gh-aw binary before agent execution
• ✅ Test 3 different workflows with diverse complexity
• ✅ Introduce 3 different error types (one per category)
• ✅ Capture complete compiler output for each test
• ✅ Provide detailed quality scores across all dimensions
• ✅ Generate specific, actionable improvement suggestions
• ✅ Create issue only when quality is below threshold (avg < 70 or any < 55)
• ✅ Clean up temporary test files after analysis

---

Priority Recommendations

🔴 High Priority - Enable Workflow Execution

1. Add build steps to workflow frontmatter

   • Include steps: section with Go setup and build commands
   • Ensure binary is in PATH before agent runs
   • Problem: Cannot analyze error quality without compiler
   • Impact: Workflow completely non-functional

2. Grant necessary permissions

   • Add contents: read permission
   • Ensure write access to /tmp directories
   • Problem: Build failures due to permission restrictions
   • Impact: Cannot build gh-aw binary

🟡 Medium Priority - Improve Analysis Quality

3. Create test workflow repository

   • Maintain curated set of test workflows
   • Include common error patterns
   • Store expected error outputs for comparison
   • Impact: More consistent, reproducible testing

4. Add regression testing

   • Track error message quality over time
   • Compare current errors to previous baselines
   • Detect when error messages improve or degrade
   • Impact: Catch error message regressions early

🟢 Low Priority - Enhanced Reporting

5. Add visual error comparison

   • Show side-by-side before/after for error improvements
   • Include screenshots of IDE error display
   • Impact: Better communication of error quality improvements

6. Track developer feedback

   • Survey developers about error helpfulness
   • Correlate error types with resolution time
   • Impact: Data-driven error message improvements

---

Implementation Guide

Step 1: Update Workflow Frontmatter

Update .github/workflows/daily-syntax-error-quality.md:

---
description: Daily syntax error quality check
on:
  schedule: daily
permissions:
  contents: read
  issues: write
engine: copilot
tools:
  bash:
    - "*"
  github:
    toolsets: [default]
steps:
  - name: Setup Go
    uses: actions/setup-go@v5
    with:
      go-version: '1.23'
      
  - name: Build gh-aw
    run: |
      make build
      
  - name: Verify Installation
    run: |
      ./gh-aw --version
      
safe-outputs:
  create-issue:
    max: 1
timeout-minutes: 15
---

Step 2: Test Workflow Locally

Before deploying, test the setup locally:

# Build binary
make build

# Create test workflows
mkdir -p /tmp/syntax-error-tests

# Run analysis
./gh-aw compile /tmp/syntax-error-tests/test-1.md 2>&1

# Verify error output quality

Step 3: Deploy and Monitor

1. Commit updated workflow
2. Compile workflow: gh aw compile daily-syntax-error-quality
3. Monitor first run for setup issues
4. Verify compiler output is captured correctly

---

Related Documentation

• Workflow Setup: .github/aw/github-agentic-workflows.md
• Compiler Documentation: docs/cli-reference/compile.md
• Error Message Guidelines: scratchpad/error-messages.md
• Testing Patterns: scratchpad/testing.md

---

References:

• §21931528477 - Current workflow run

AI generated by Daily Syntax Error Quality Check

• expires on Feb 15, 2026, 2:48 AM UTC