Add Proxy API Configuration Options with Timeout, Retry, and Resilience Features

[Ironboxplus]

Description:
We propose improvements to LobeChat's proxy API configuration to better handle scenarios requiring scalable and stable performance under demanding conditions. Currently, users may experience:

1. Unresponsive workflows during prolonged API interactions.
2. Data loss when processing extended outputs.
3. Bottlenecks in scenarios requiring simultaneous multi-process handling.

Suggested Enhancements:

1. Adaptive Request Throttling
   • Dynamic adjustment of request frequency based on API health signals.
2. Connection Pool Optimization
   • Configurable limits for parallel request channels with intelligent queuing.
3. Priority-Based Task Scheduling
   • Critical operations (e.g., stream continuation) prioritized over new requests.
4. Stateful Session Management
   • Maintain request context integrity across retry attempts.
5. Resource Allocation Profiles
   • Presets for different usage patterns (e.g., "High Throughput" vs "Low Latency").

Performance-Centric Additions:

• Automatic load detection and failover switching
• Batch processing capabilities for grouped operations
• Memory-aware response caching strategies

Technical Implementation Hints:

• Implement circuit-breaker patterns for degraded API performance
• Add metrics for request/response lifecycle monitoring
• Introduce sliding window rate limiting

This would empower users to maintain smooth operations even when managing complex workflows with intensive processing requirements, while preserving system responsiveness.

[arvinxx]

it seems not the target of lobechat, we are more of a frontend, not an api proxy gateway

[Ironboxplus]

it seems not the target of lobechat, we are more of a frontend, not an api proxy gateway

yeah, i mean the default built-in proxy function in lobechat
it will be used if "client mode" can't apply