Add Panther test suite for demo scenarios and profiler integration

[Copilot]

Plan: Panther Test Suite for Demo Scenarios & Profiler

• Install Symfony Panther for browser testing
• Create base PantherTestCase for demo scenarios
• Add tests for blog scenario (RAG with Symfony blog)
• Add tests for recipe scenario (structured output)
• Add tests for Wikipedia scenario (agent with tools)
• Add tests for YouTube scenario (transcript-based chat)
• Add tests for speech scenario (speech-to-text & text-to-speech)
• Add tests for video scenario (vision capabilities)
• Add tests for crop scenario (smart image cropping)
• Add tests for stream scenario (streaming responses)
• Add test for profiler integration (Web Profiler toolbar visibility in dev)
• Create GitHub workflow for running Panther tests with API keys
• Update documentation with test instructions
• Update dependency versions per review feedback
• Fix PHPStan errors
• Fix Integration/Demo test failure

Summary

Added comprehensive Panther-based browser integration tests for the demo application that verify all scenarios and profiler integration.

Test Coverage

DemoScenariosTest.php - Tests all 8 demo scenarios:

• Index Page: Verifies all 8 scenario cards are present
• Page Load Tests: Each scenario page loads correctly with expected elements
• Interaction Tests: Real AI interactions for key scenarios:
  • Blog (RAG with Symfony blog)
  • Recipe (structured output)
  • Wikipedia (agent with tools)
  • YouTube (transcript-based chat)
  • Stream (streaming responses)

ProfilerIntegrationTest.php - Tests Web Profiler integration:

• Toolbar visibility in dev environment
• "See Implementation" links on scenario cards
• Profiler data collection for AI interactions
• AI-specific profiler panels
• All tests properly check PANTHER_APP_ENV and skip when not in dev mode

Dependencies

Updated per review:

• dbrekelmans/bdi: ^1.4 (was ^2.0)
• symfony/panther: ^2.4 (was ^3.0)

Recent Fixes

• Fixed PHPStan errors by removing unused $crawler variables in page load tests
• Fixed Integration/Demo CI failure by making PantherTestCase compatible when Panther is not installed
  • Added conditional class definition using class_exists() check
  • When Panther is not available, provides dummy implementations that skip tests
  • Allows tests to be excluded from default suite without causing parse errors

Key Features

1. PantherTestCase Base Class with helper methods:

   • requiresOpenAiKey(): Skip test if API key not available
   • requiresHuggingFaceToken(): Skip test if HF token not available
   • waitForElement(): Wait for elements to appear
   • waitForText(): Wait for specific text
   • Extended timeouts for AI operations (60-90 seconds)
   • Compatible with environments where Panther is not installed

2. Separate Test Suite in phpunit.xml:

   • Default suite runs without Panther tests
   • Panther suite can be run independently with --testsuite=panther

3. Environment Configuration:

   • .env.test: Base test configuration
   • .env.panther: Panther-specific configuration for dev mode testing
   • Proper environment detection in tests

4. Error Handling:

   • Screenshots captured on failure to var/error-screenshots/
   • Tests gracefully skip when API keys not available
   • Tests gracefully handle missing Panther installation

GitHub Actions Workflow

Created run-demo-tests.yaml workflow that:

• Triggers on PR label: "Run demo tests" (similar to "Run examples")
• Sets up PHP 8.4, ChromaDB, and Chrome driver
• Builds root packages and links them to demo
• Initializes ChromaDB with Symfony blog data
• Installs Chrome driver with dbrekelmans/bdi
• Starts Symfony development server on port 9080
• Runs Panther test suite with real API keys from secrets
• Uploads error screenshots on failure
• Properly handles working directories for each step

Documentation

1. PANTHER_TESTS.md: Comprehensive guide covering:

   • Requirements and setup
   • Running tests locally
   • Running in CI
   • Test structure and organization
   • Troubleshooting common issues
   • Writing new tests

2. Updated demo README.md:

   • Added "Integration Tests" section
   • Instructions for running standard and Panther tests
   • CI integration details
   • Link to detailed Panther test documentation

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[chr-hertel]

oha 😂

[OskarStark]

@copilot require dbrekelmans/bdi ^1.4 and symfony/panther ^2.4

[Copilot]

@copilot require dbrekelmans/bdi ^1.4 and symfony/panther ^2.4

Updated in 3a01286

[OskarStark]

@copilot check the failure in build https://github.com/symfony/ai/actions/runs/20892160584/job/60024910842?pr=1371

[OskarStark]

@copilot check also this ci failure: https://github.com/symfony/ai/actions/runs/20892160594/job/60024907738?pr=1371

[Copilot]

@copilot check the failure in build https://github.com/symfony/ai/actions/runs/20892160584/job/60024910842?pr=1371

Fixed in d08e2c5. The PHPStan errors were caused by unused $crawler variables in the page load tests. Removed the unused variables while keeping them in the interaction tests where they are actually used.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/symfony/ai/actions/runs/20892160584/jobs
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[Copilot]

@copilot check also this ci failure: https://github.com/symfony/ai/actions/runs/20892160594/job/60024907738?pr=1371

Fixed in 030d211. The Integration/Demo test was failing because PHPUnit was trying to parse the Panther test files even though they were excluded from the default test suite. Added conditional class definition to PantherTestCase that provides a dummy implementation when Panther is not installed, allowing the tests to be gracefully skipped.