HTTP 504 Gateway Timeout

Fix checklist

• →Switch to streaming for long generations.
• →Raise gateway/proxy read timeouts above max generation time.
• →Reduce max_tokens for latency-sensitive paths.
• →Retry idempotent requests with backoff.

async function fetchWithRetry(url: string, init: RequestInit, maxRetries = 5) {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, init);
    // 504 is retryable — back off and try again.
    if (res.status !== 504 || attempt >= maxRetries) return res;
    const retryAfter = Number(res.headers.get("retry-after"));
    const delay = Number.isFinite(retryAfter) && retryAfter > 0
      ? retryAfter * 1000
      : Math.min(60_000, 1000 * 2 ** attempt) * (0.5 + Math.random()); // expo backoff + jitter
    await new Promise((r) => setTimeout(r, delay));
  }
}