Letter2025 / dsh-model-failover

dsh-model-failover

English | 中文

Two-level model circuit breaker with failover for DeepSeek Harness. When a model (or a whole provider) starts failing repeatedly, the plugin opens a circuit and routes the next model request to a configured fallback — no core changes, installable with dsh plugin.

What it does

• Model circuit — a provider/model route opens after modelCircuitThreshold failures inside burstWindowMs.
• Platform circuit — a provider opens when platformCircuitThreshold distinct models under it are open at once (a platform outage usually takes down every model on it).
• Failover — the next request goes to the first healthy fallback in fallbacks; the switch is recorded by the loop itself (request/header change) and optionally announced as a user-visible message.
• Recovery probes — an open model circuit is probed after modelCooldownMs with a tiny real call; a successful probe closes the circuit, a failed one extends the cooldown.
• Composes with llm-retry — per-request backoff retries stay owned by the bundled llm-retry policy; the breaker observes the failures that escape it, so transient blips that recover after a retry never trip a circuit.

Install

dsh plugin --profile web add dsh-model-failover

Then configure fallbacks (the only field you must set) and, if needed, the thresholds in your profile cordis.patch.yml — see the plugin row in cordis.patch.yml for the full default config.

A companion skill, configure-model-failover, walks the agent through setting the fallback models (AI probes the current model config, writes the fallback override, then asks you to confirm). It is installed automatically with the plugin: the package ships skills/configure-model-failover/SKILL.md and the plugin registers it as a bundled skill whenever the skills service is present — no copy step needed. For a standalone install (profile without the plugin), copy it into ~/.dsh/skills/configure-model-failover/ (user-level skills are picked up live).

How it works

The plugin decorates two agent-loop waterfalls (both official extension points, no core changes):

Waterfall	Role
agent/request-error	Records failures whose code is in tripCodes into the circuit breaker, then delegates through next() so llm-retry still owns retries.
agent/request	await next() for the resolved config, then returns the healthy primary route, or the first healthy fallback when the primary's circuit is open.

request ──> agent/request ──> primary (mock/m1) ──> fail ×2 ──> circuit open
                                                                │
next request ──> agent/request ──> primary open ──> fallback (mock2/m2) ✔
                                                                │
                                     probe after cooldown ──> success ──> circuit closed

Configuration

Field	Default	Meaning
enabled	true	Master switch.
fallbacks	[]	Ordered fallback routes {provider, model}; must point at providers with a registered adapter.
tripCodes	RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, QUOTA, EMPTY_RESPONSE	Failure codes that count toward a circuit; e.g. AUTH/INVALID_CREDENTIAL stay terminal.
modelCircuitThreshold	2	Failures inside the burst window that open a model circuit.
modelCooldownMs	60000	Cooldown before an open model circuit is probed.
platformCircuitThreshold	2	Distinct open models that open the whole provider.
platformCooldownMs	120000	Provider-wide cooldown.
burstWindowMs	300000	Failures older than this start a fresh burst.
enableProbe	true	Probe open models after cooldown to recover circuits.
probeMaxTokens	8	Output cap for probe calls.
stripReasoningEffort	true	Drop the primary's reasoning effort when failing over (fallbacks may not support it).
notifyUser	true	Append a user-visible message when a route switches.

Events

Plugin-defined (emit) events, typed via the @deepseek-ai/cordis augmentation in src/types.ts:

• model-failover/circuit-opened — {provider, model, level: 'model' | 'platform'}
• model-failover/circuit-closed — {provider, model, level: 'model'}
• model-failover/failover — {from, to, agentId}
• model-failover/probe — {provider, model, ok, message?}

Known Limitations and Deferred Work

• Process-local state — circuit state lives in memory and resets on plugin reload (like every harness registry). Cross-instance sharing is deferred.
• Agent-loop calls only — agent/request covers the main conversation loop. Auxiliary calls (session-title, compaction, hand-built ctx.llm.stream) are not routed.
• retryPolicy.mode: 'always' — the bundled llm-retry never delegates a failure to this breaker in that mode, so failover stays idle by design (the operator chose unbounded retries).
• No context-window adaptation — a fallback with a smaller context window may hit CONTEXT_WINDOW_EXCEEDED; set stripReasoningEffort and pick compatible fallbacks.
• No platform probe — the platform circuit recovers by cooldown expiry; only model circuits are probed.

License

MIT