Skip to content

Refactor/profiling #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Refactor/profiling #67

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b121cca
Changed: Refactors benchmark workflow using Python utils (internal)
acgetchell Sep 5, 2025
8a5f176
Added: Comprehensive profiling benchmarks and memory stress tests
acgetchell Sep 6, 2025
547a45d
feat: Add memory profiling system and performance optimizations for v…
acgetchell Sep 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .codecov.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ coverage:

ignore:
- "src/lib.rs"
- "benches/**"
160 changes: 22 additions & 138 deletions .github/workflows/benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,144 +231,60 @@ jobs:

- name: Prepare baseline for comparison
if: steps.find_baseline.outputs.found == 'true'
shell: bash
env:
RELEASE_TAG: ${{ steps.find_baseline.outputs.release_tag }}
SOURCE_TYPE: ${{ steps.find_baseline.outputs.source_type }}
run: |
set -euo pipefail
if [[ -f "baseline-artifact/baseline_results.txt" ]]; then
echo "📦 Prepared baseline from artifact (origin: ${SOURCE_TYPE:-unknown}, tag: ${RELEASE_TAG:-n/a})"
echo "BASELINE_EXISTS=true" >> "$GITHUB_ENV"
echo "BASELINE_SOURCE=artifact" >> "$GITHUB_ENV"
echo "BASELINE_ORIGIN=${SOURCE_TYPE:-unknown}" >> "$GITHUB_ENV"
if [[ -n "${RELEASE_TAG:-}" ]]; then
echo "BASELINE_TAG=${RELEASE_TAG}" >> "$GITHUB_ENV"
fi

# Show baseline metadata
echo "=== Baseline Information (from artifact) ==="
head -n 3 baseline-artifact/baseline_results.txt
else
echo "❌ Downloaded artifact but no baseline_results.txt found"
echo "BASELINE_EXISTS=false" >> "$GITHUB_ENV"
echo "BASELINE_SOURCE=missing" >> "$GITHUB_ENV"
echo "BASELINE_ORIGIN=unknown" >> "$GITHUB_ENV"
# Set additional environment variables based on source
echo "BASELINE_ORIGIN=${SOURCE_TYPE:-unknown}" >> "$GITHUB_ENV"
if [[ -n "${RELEASE_TAG:-}" ]]; then
echo "BASELINE_TAG=${RELEASE_TAG}" >> "$GITHUB_ENV"
fi

# Prepare baseline using Python utility
uv run benchmark-utils prepare-baseline

- name: Set baseline status if none found
if: steps.find_baseline.outputs.found != 'true'
shell: bash
run: |
set -euo pipefail
echo "📈 No baseline artifact found for performance comparison"
echo "BASELINE_EXISTS=false" >> "$GITHUB_ENV"
echo "BASELINE_SOURCE=none" >> "$GITHUB_ENV"
echo "BASELINE_ORIGIN=none" >> "$GITHUB_ENV"
run: uv run benchmark-utils set-no-baseline

- name: Extract baseline commit SHA
if: env.BASELINE_EXISTS == 'true'
shell: bash
run: |
set -euo pipefail
bc_sha="$(grep "^Git commit:" baseline-artifact/baseline_results.txt | awk '{print $3}' || true)"
if [[ -z "$bc_sha" || ! "$bc_sha" =~ ^[0-9A-Fa-f]{7,40}$ ]]; then
if [[ -f "baseline-artifact/metadata.json" ]]; then
bc_sha="$(jq -r '.commit // empty' baseline-artifact/metadata.json 2>/dev/null || true)"
fi
fi
if [[ -n "$bc_sha" && "$bc_sha" =~ ^[0-9A-Fa-f]{7,40}$ ]]; then
echo "BASELINE_COMMIT=$bc_sha" >> "$GITHUB_ENV"
else
echo "BASELINE_COMMIT=unknown" >> "$GITHUB_ENV"
fi
run: uv run benchmark-utils extract-baseline-commit

- name: Determine if benchmarks should run
if: env.BASELINE_EXISTS == 'true'
shell: bash
env:
SAFE_COMMIT_SHA: ${{ github.sha }}
run: |
set -euo pipefail
if [[ "$BASELINE_COMMIT" == "unknown" ]]; then
echo "SKIP_BENCHMARKS=false" >> "$GITHUB_ENV"
echo "SKIP_REASON=unknown_baseline" >> "$GITHUB_ENV"
elif [[ "$BASELINE_COMMIT" == "$SAFE_COMMIT_SHA" ]]; then
echo "SKIP_BENCHMARKS=true" >> "$GITHUB_ENV"
echo "SKIP_REASON=same_commit" >> "$GITHUB_ENV"
else
if ! git cat-file -e "$BASELINE_COMMIT^{commit}" 2>/dev/null; then
echo "SKIP_BENCHMARKS=false" >> "$GITHUB_ENV"
echo "SKIP_REASON=baseline_commit_not_found" >> "$GITHUB_ENV"
elif git diff --name-only "$BASELINE_COMMIT"..HEAD | \
grep -qE '^(src/|benches/|Cargo\.toml|Cargo\.lock)'; then
echo "SKIP_BENCHMARKS=false" >> "$GITHUB_ENV"
echo "SKIP_REASON=changes_detected" >> "$GITHUB_ENV"
else
echo "SKIP_BENCHMARKS=true" >> "$GITHUB_ENV"
echo "SKIP_REASON=no_relevant_changes" >> "$GITHUB_ENV"
fi
fi
uv run benchmark-utils determine-skip \
--baseline-commit "$BASELINE_COMMIT" \
--current-commit "$SAFE_COMMIT_SHA"

- name: Skip benchmarks - no code changes
if: env.BASELINE_EXISTS == 'true' && env.SKIP_BENCHMARKS == 'true'
shell: bash
run: |
set -euo pipefail
case "${SKIP_REASON:-unknown}" in
same_commit) echo "🔍 Current commit matches baseline (${BASELINE_COMMIT}); skipping benchmarks.";;
no_relevant_changes) echo "🔍 No relevant code changes since ${BASELINE_COMMIT}; skipping benchmarks.";;
*) echo "🔍 Benchmarks skipped.";;
esac
uv run benchmark-utils display-skip-message \
--reason "$SKIP_REASON" \
--baseline-commit "$BASELINE_COMMIT"

- name: Skip benchmarks - no baseline available
if: env.BASELINE_EXISTS != 'true'
shell: bash
run: |
set -euo pipefail
echo "⚠️ No performance baseline available for comparison."
echo " - No baseline artifacts found in recent workflow runs"
echo " - Performance regression testing requires a baseline"
echo ""
echo "💡 To enable performance regression testing:"
echo " 1. Create a release tag (e.g., v0.4.3), or"
echo " 2. Manually trigger the 'Generate Performance Baseline' workflow"
echo " 3. Future PRs and pushes will use that baseline for comparison"
echo " 4. Baselines use full benchmark settings for accurate comparisons"
run: uv run benchmark-utils display-no-baseline

- name: Run performance regression test
if: env.BASELINE_EXISTS == 'true' && env.SKIP_BENCHMARKS == 'false'
shell: bash
run: |
set -euo pipefail
echo "🚀 Running performance regression test..."
echo " Baseline source: ${BASELINE_SOURCE:-unknown}"
echo " Baseline origin: ${BASELINE_ORIGIN:-unknown}"

# This will exit with code 1 if significant regressions are found
echo " Using CI performance suite against ${BASELINE_ORIGIN:-unknown} baseline"
BASELINE_PATH="baseline-artifact/baseline_results.txt"
if uv run benchmark-utils --help >/dev/null 2>&1; then
uv run benchmark-utils compare --baseline "$BASELINE_PATH"
elif uv run python -c "import importlib; importlib.import_module('scripts.benchmark_utils')" \
>/dev/null 2>&1; then
uv run python -m scripts.benchmark_utils compare --baseline "$BASELINE_PATH"
else
echo "❌ benchmark-utils entrypoint and module not found" >&2
exit 2
fi

# Run regression test using Python utility
uv run benchmark-utils run-regression-test \
--baseline "baseline-artifact/baseline_results.txt"

Comment on lines 278 to 284
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix env var name when echoing baseline metadata.

You set BASELINE_ORIGIN earlier; here you echo BASELINE_SOURCE (unset).

-          echo "   Baseline source: ${BASELINE_SOURCE:-unknown}"
+          echo "   Baseline source: ${BASELINE_ORIGIN:-unknown}"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo " Baseline source: ${BASELINE_SOURCE:-unknown}"
echo " Baseline origin: ${BASELINE_ORIGIN:-unknown}"
# This will exit with code 1 if significant regressions are found
echo " Using CI performance suite against ${BASELINE_ORIGIN:-unknown} baseline"
BASELINE_PATH="baseline-artifact/baseline_results.txt"
if uv run benchmark-utils --help >/dev/null 2>&1; then
uv run benchmark-utils compare --baseline "$BASELINE_PATH"
elif uv run python -c "import importlib; importlib.import_module('scripts.benchmark_utils')" \
>/dev/null 2>&1; then
uv run python -m scripts.benchmark_utils compare --baseline "$BASELINE_PATH"
else
echo "❌ benchmark-utils entrypoint and module not found" >&2
exit 2
fi
# Run regression test using Python utility
uv run benchmark-utils run-regression-test \
--baseline "baseline-artifact/baseline_results.txt"
echo " Baseline source: ${BASELINE_ORIGIN:-unknown}"
echo " Baseline origin: ${BASELINE_ORIGIN:-unknown}"
# Run regression test using Python utility
uv run benchmark-utils run-regression-test \
--baseline "baseline-artifact/baseline_results.txt"
🤖 Prompt for AI Agents 
In .github/workflows/benchmarks.yml around lines 278 to 284, the echo statement
prints BASELINE_SOURCE which is never set (the variable earlier is
BASELINE_ORIGIN); change the echo to reference BASELINE_ORIGIN (or set
BASELINE_SOURCE where intended) so the workflow emits the correct baseline
metadata value; update the line to echo "   Baseline origin:
${BASELINE_ORIGIN:-unknown}" (or ensure the variable name matches your earlier
definition).

- name: Display regression test results
if: env.BASELINE_EXISTS == 'true' && env.SKIP_BENCHMARKS == 'false' && always()
shell: bash
run: |
set -euo pipefail
if [[ -f "benches/compare_results.txt" ]]; then
echo "=== Performance Regression Test Results ==="
cat benches/compare_results.txt
else
echo "⚠️ No comparison results file found"
fi
run: uv run benchmark-utils display-results

- name: Upload regression test results
if: env.BASELINE_EXISTS == 'true' && env.SKIP_BENCHMARKS == 'false' && always()
Expand All @@ -381,36 +297,4 @@ jobs:

- name: Summary
if: always()
shell: bash
run: |
set -euo pipefail
echo "📊 Performance Regression Testing Summary"
echo "==========================================="
echo "Baseline source: ${BASELINE_SOURCE:-none}"
echo "Baseline origin: ${BASELINE_ORIGIN:-unknown}"
echo "Baseline tag: ${BASELINE_TAG:-n/a}"
echo "Baseline exists: ${BASELINE_EXISTS:-false}"
echo "Skip benchmarks: ${SKIP_BENCHMARKS:-unknown}"
echo "Skip reason: ${SKIP_REASON:-n/a}"

if [[ "${BASELINE_EXISTS:-false}" == "true" && "${SKIP_BENCHMARKS:-true}" == "false" ]]; then
if [[ -f "benches/compare_results.txt" ]]; then
if grep -q "REGRESSION" benches/compare_results.txt; then
echo "Result: ⚠️ Performance regressions detected"
else
echo "Result: ✅ No significant performance regressions"
fi
else
echo "Result: ❓ Benchmark comparison completed but no results file found"
fi
elif [[ "${SKIP_BENCHMARKS:-true}" == "true" ]]; then
case "${SKIP_REASON:-unknown}" in
same_commit) echo "Result: ⏭️ Benchmarks skipped (same commit as baseline)";;
no_relevant_changes) echo "Result: ⏭️ Benchmarks skipped (no relevant code changes)";;
baseline_commit_not_found) \
echo "Result: ⚠️ Baseline commit not found in history (force-push/shallow clone?)";;
*) echo "Result: ⏭️ Benchmarks skipped";;
esac
else
echo "Result: ⏭️ Benchmarks skipped (no baseline available)"
fi
run: uv run benchmark-utils generate-summary
80 changes: 15 additions & 65 deletions .github/workflows/generate-baseline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,28 +40,9 @@ jobs:
cache: true
# Toolchain from rust-toolchain.toml

- name: Install jq (required for baseline generation)
run: |
# macOS runner should have jq, but ensure it's available
if ! command -v jq >/dev/null 2>&1; then
echo "Installing jq..."
brew install jq
else
echo "jq is already available: $(jq --version)"
fi

- name: Determine tag information
id: tag_info
run: |
if [[ "${GITHUB_REF:-}" == refs/tags/* ]]; then
TAG_NAME="${GITHUB_REF#refs/tags/}"
echo "Using push tag: $TAG_NAME"
else
TAG_NAME="manual-$(date -u +%Y%m%d-%H%M%S)"
echo "Using generated tag name: $TAG_NAME"
fi
echo "tag_name=$TAG_NAME" >> "$GITHUB_OUTPUT"
echo "Final tag name: $TAG_NAME"
run: uv run benchmark-utils determine-tag
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Fix step order: uv used before it’s installed.

Move “Install uv” (and “Verify uv”) above “Determine tag information” to avoid a hard failure.

-      - name: Determine tag information
-        id: tag_info
-        run: uv run benchmark-utils determine-tag
-
-      - name: Install uv (Python package manager)
+      - name: Install uv (Python package manager)
         uses: astral-sh/setup-uv@557e51de59eb14aaaba2ed9621916900a91d50c6  # v6.6.1
         with:
           version: "latest"
 
       - name: Verify uv installation
         run: uv --version
+
+      - name: Determine tag information
+        id: tag_info
+        run: uv run benchmark-utils determine-tag

Also applies to: 47-54

🤖 Prompt for AI Agents 
.github/workflows/generate-baseline.yml around line 45 (also applies to lines
47-54): the workflow invokes "uv run benchmark-utils determine-tag" before the
"Install uv" and "Verify uv" steps, causing a hard failure; reorder the steps so
the "Install uv" and its "Verify uv" step appear before the "Determine tag
information" run step (and update any dependent step names/ids if necessary) so
uv is installed and verified prior to running the determine-tag command.


- name: Install uv (Python package manager)
uses: astral-sh/setup-uv@557e51de59eb14aaaba2ed9621916900a91d50c6 # v6.6.1
Expand All @@ -76,60 +57,35 @@ jobs:
TAG_NAME: ${{ steps.tag_info.outputs.tag_name }}
run: |
echo "🚀 Generating performance baseline for tag $TAG_NAME"

# Run baseline generation using Python CLI tool with CI performance suite
# Generate baseline using Python CLI tool
uv run benchmark-utils generate-baseline \
|| uv run python -m scripts.benchmark_utils generate-baseline

# Verify baseline was created
if [ ! -f "baseline-artifact/baseline_results.txt" ]; then
echo "❌ Baseline generation failed - no baseline file created"
exit 1
fi

--output "baseline-artifact/baseline-$TAG_NAME.txt" \
--tag "$TAG_NAME"

echo "✅ Baseline generated successfully"
Comment on lines 59 to 66
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Ensure output directory exists before writing baseline.

Create baseline-artifact/ explicitly to avoid failures if the CLI doesn’t create parent dirs.

           echo "🚀 Generating performance baseline for tag $TAG_NAME"
-          
+          mkdir -p baseline-artifact
           # Generate baseline using Python CLI tool
           uv run benchmark-utils generate-baseline \
             --output "baseline-artifact/baseline-$TAG_NAME.txt" \
             --tag "$TAG_NAME"
-          
+
           echo "✅ Baseline generated successfully"
-          
+
           # Display baseline summary
           uv run benchmark-utils display-summary \
             --baseline "baseline-artifact/baseline-$TAG_NAME.txt"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo "🚀 Generating performance baseline for tag $TAG_NAME"
# Run baseline generation using Python CLI tool with CI performance suite
# Generate baseline using Python CLI tool
uv run benchmark-utils generate-baseline \
|| uv run python -m scripts.benchmark_utils generate-baseline
# Verify baseline was created
if [ ! -f "baseline-artifact/baseline_results.txt" ]; then
echo "❌ Baseline generation failed - no baseline file created"
exit 1
fi
--output "baseline-artifact/baseline-$TAG_NAME.txt" \
--tag "$TAG_NAME"
echo "✅ Baseline generated successfully"
echo "🚀 Generating performance baseline for tag $TAG_NAME"
mkdir -p baseline-artifact
# Generate baseline using Python CLI tool
uv run benchmark-utils generate-baseline \
--output "baseline-artifact/baseline-$TAG_NAME.txt" \
--tag "$TAG_NAME"
echo "✅ Baseline generated successfully"
# Display baseline summary
uv run benchmark-utils display-summary \
--baseline "baseline-artifact/baseline-$TAG_NAME.txt"
🤖 Prompt for AI Agents 
.github/workflows/generate-baseline.yml around lines 59 to 66: the workflow
writes to baseline-artifact/baseline-$TAG_NAME.txt but doesn't ensure the
baseline-artifact directory exists; before running the Python CLI add a step to
create the directory (e.g., run mkdir -p baseline-artifact) so the output path
is guaranteed to exist and the job won't fail if the CLI doesn't create parent
dirs.


# Show baseline summary
echo "📊 Baseline summary:"
head -n 10 baseline-artifact/baseline_results.txt
echo "..."
benchmark_count=$(grep -c "^===" baseline-artifact/baseline_results.txt || echo "0")
echo "Total benchmarks: $benchmark_count"

# Display baseline summary
uv run benchmark-utils display-summary \
--baseline "baseline-artifact/baseline-$TAG_NAME.txt"

- name: Prepare baseline artifact
env:
TAG_NAME: ${{ steps.tag_info.outputs.tag_name }}
SAFE_COMMIT_SHA: ${{ github.sha }}
SAFE_RUN_ID: ${{ github.run_id }}
run: |
# The baseline file is already in baseline-artifact/ from the script
# Just copy it with tag-specific name
cp baseline-artifact/baseline_results.txt "baseline-artifact/baseline-$TAG_NAME.txt"

# Create metadata file
cat > "baseline-artifact/metadata.json" <<EOF
{
"tag": "$TAG_NAME",
"commit": "$SAFE_COMMIT_SHA",
"workflow_run_id": "$SAFE_RUN_ID",
"generated_at": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")",
"runner_os": "$RUNNER_OS",
"runner_arch": "$RUNNER_ARCH"
}
EOF

# Create metadata file using Python utility
uv run benchmark-utils create-metadata --tag "$TAG_NAME"

echo "📦 Artifact contents:"
ls -la baseline-artifact/

- name: Create safe artifact name
id: safe_name
env:
SAFE_TAG_NAME: ${{ steps.tag_info.outputs.tag_name }}
run: |
# Sanitize tag name for artifact upload (remove special characters)
CLEAN_TAG_NAME=$(echo "$SAFE_TAG_NAME" | sed 's/[^a-zA-Z0-9._-]/_/g')
echo "artifact_name=performance-baseline-$CLEAN_TAG_NAME" >> "$GITHUB_OUTPUT"
echo "Using sanitized artifact name: performance-baseline-$CLEAN_TAG_NAME"
TAG_NAME: ${{ steps.tag_info.outputs.tag_name }}
run: uv run benchmark-utils sanitize-artifact-name --tag "$TAG_NAME"

- name: Upload baseline artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
Expand All @@ -139,12 +95,6 @@ jobs:
retention-days: 90 # Keep baselines ~90 days (align with repo settings; adjust if needed)
compression-level: 6 # Good balance of speed/compression

- name: Cleanup temporary baseline file
if: ${{ success() }}
run: |
# Remove temporary baseline file from working directory
rm -f baseline-artifact/baseline_results.txt
echo "✅ Cleaned up temporary baseline file"

- name: Display next steps
env:
Expand Down
Loading
Loading