1
Redis Queueing & Concurrency Dispatch
Clients push inference requests to the Redis queue drai:ai_queue via LPUSH.
Dr.AI continuously checks the queue with a dedicated non-blocking worker loop.
The system processes tasks concurrently up to the number of configured Gemini API keys (1 task per key concurrently).
2
Dynamic Model Discovery & Complexity Tiering
Tasks specify complexity tiers: fast, simple, or complex.
Dr.AI automatically discovers and scores models available per API key every 24 hours:
- Flash models (e.g. Gemini 2.5 Flash, 2.0 Flash) handle Fast tasks with tight timeouts.
- Pro models (e.g. Gemini 2.5 Pro) handle Complex deep reasoning.
- Gemma models handle lightweight processing.
3
Instant 429 Failover & Local OCI Fallback
When an API key encounters an HTTP 429 Rate Limit, the system marks the key/model as inactive in Redis and instantly rotates to the next available API key with zero backoff delay.
If all cloud keys are exhausted, inference seamlessly falls back to the private OCI AI server running Ollama (llama3.2:1b for fast/simple, qwen2.5:3b for complex).
4
Real-time WebSocket Streaming & Structured Storage
Task lifecycle events (processing, completed, error) and queue depth updates are broadcast to connected dashboard clients in real time via Socket.io.
Results are logged to Redis transaction history (drai:transactions).