Fix: nightly integration tests

[blove]

CI: Fix “Nightly Integration Tests” and Add Cross-Adapter E2E Coverage (OpenAI, Azure, Gemini, Ollama, Writer)

Summary

Our Nightly Integration Tests workflow is failing on main. We need to make the nightly job reliable and ensure each supported adapter has its own end-to-end (e2e) coverage that runs (and is skipped when secrets aren’t present). The current workflow uses Node 22.x and executes nx run-many -t e2e, but at least one adapter is exiting with a non-zero code.

Current Behavior

• Workflow: .github/workflows/nightly.yml
  • Runs nightly via cron.
  • Uses Node 22.x.
  • Executes npm ci then npx nx run-many -t e2e --parallel=3.
  • Supplies secrets for OPENAI_API_KEY, GOOGLE_API_KEY, AZURE_API_KEY, AZURE_ENDPOINT, AZURE_API_VERSION, and WRITER_API_KEY.
  • Latest run shows failure (exit code 1).

Desired Behavior

• Nightly workflow should:
  • Pass when all provider secrets are valid.
  • Gracefully skip adapter suites when secrets are missing.
  • Run isolated jobs for each adapter for clearer CI visibility.
• Each supported adapter should have minimal e2e tests covering:
  1. Basic text completion
  2. Structured/JSON output
  3. Streaming response handling
  4. Tool/function calling (if supported)

Supported Adapters

• OpenAI
• Azure OpenAI
• Google Gemini
• Writer
• Ollama

Working Theories

• Missing/invalid secrets may be causing the global run-many job to fail.
• Node 22.x might introduce incompatibilities; Node 20.x should also be tested.
• Ollama requires a running model server, which CI currently lacks.

---

Proposed Approach

1. Split Nightly Workflow into an Adapter Matrix

Run one job per adapter with independent pass/fail states:

name: Nightly Integration Tests
on:
  schedule:
    - cron: '0 0 * * *'
  workflow_dispatch:

jobs:
  e2e:
    name: ${{ matrix.adapter }} (Node ${{ matrix.node }})
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        node: [20.x, 22.x]
        adapter: [openai, azure, gemini, writer, ollama]
    services:
      ollama:
        image: ollama/ollama:latest
        ports: ['11434:11434']
        options: >-
          --health-cmd="curl -s http://localhost:11434/api/tags || exit 1"
          --health-interval=5s --health-retries=20 --health-timeout=2s
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: 'npm'

      - name: Install deps
        run: npm ci

      - name: Set env for adapter
        run: node ./.github/scripts/export-adapter-env.cjs ${{ matrix.adapter }}
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
          AZURE_API_KEY: ${{ secrets.AZURE_API_KEY }}
          AZURE_ENDPOINT: ${{ secrets.AZURE_ENDPOINT }}
          AZURE_API_VERSION: ${{ secrets.AZURE_API_VERSION }}
          WRITER_API_KEY: ${{ secrets.WRITER_API_KEY }}
          OLLAMA_HOST: http://localhost:11434

      - name: Run adapter e2e
        run: npx nx run e2e-${{ matrix.adapter }}:e2e
        env:
          DEBUG: hashbrown:*