Capture LLM Metadata in Cloud Submissions

[gltanaka]

Problem

When code is generated and submitted to PDD Cloud, we do not capture:

• Which LLM was used
• Temperature setting
• Thinking token budget
• Reasoning type (effort/budget)

From Dec 13 Benchmarking Meeting:

"Right now I do not put the LM that was used. I do not put down a temperature that was used... I do not put in thinking tokens"

Current Firestore Schema (inferred)

prompt, generated_code, generated_example, generated_test, embedding

Proposed Addition

llm_model, temperature, thinking_tokens, reasoning_type, generation_cost, timestamp

Benefits

1. Understand which models produce best results for which prompts
2. Enable model routing based on historical success
3. Track cost trends over time
4. Support future cloud model routing - automatically pick best model for problem type

Context from Meeting

"Different models excel at different tasks (jagged intelligence). Users should not have to manually figure out which model works best."

This metadata capture is a prerequisite for intelligent model routing in PDD Cloud.