CanGeng / llm-fallback

llm-fallback

English | 中文

Multi-level provider fallback for same-named models in the DeepSeek Harness.

A pool maps one model id to an ordered provider list. The first provider is the primary; each later one is a fallback. The plugin keeps every step on the primary (stable request prefix → KV cache reuse) and only fails over when the current level is exhausted.

Behavior

• agent/request picks the pool level for each step. A new step starts at the primary (level 0) unless the explicitly requested provider is itself a pool member — then it starts there and fails over forward. A recovered primary is therefore reused immediately on the next step. It also probes each candidate with resolveCallConfig and skips (recording) any that would fail before streaming — a missing route (NO_ADAPTER), an unknown model, or an unsupported reasoning effort.
• agent/request-error records each level's failure, then advances the level by returning { kind: 'retry' }. When every level is exhausted it throws one aggregated error listing each level's failure (provider: code — message).

Same-provider retries happen first (via dsh-llm-retry, which sits outside this plugin), so transient errors like RATE_LIMIT never trigger a failover and never break the prefix. Only terminal errors (QUOTA, AUTH, …) or an exhausted retry budget fail over.

retryPolicy.mode: always is incompatible

always retries one provider forever and cannot be stopped from a listener, so it conflicts with a terminating fallback chain:

• alwaysPolicy: reject (default) — refuse the provider, naming it and the model. Providers registered at load time are rejected when the plugin loads; routes activated later through settings (dormant routes) are rejected the first time a pool request would use them.
• alwaysPolicy: degrade — warn and cap that provider at alwaysMaxRetries same-provider retries before failing over.

In both modes, an always provider may never be the last level of a pool (that can never terminate) — it is always rejected.

Configuration

See cordis.patch.yml. The pools list maps a model id to its ordered provider chain. The config is validated when the plugin loads: an unknown alwaysPolicy, a pool without a non-empty model, or a pool whose providers is not an array throws immediately instead of failing silently later.

config:
  alwaysPolicy: reject
  alwaysMaxRetries: 2
  pools:
    - model: deepseek-v4-pro
      providers: [deepseek-official, ark-coding, ark-agent]

Development

node --test   # or: npm test

Install

dsh plugin --profile web add link:/path/to/llm-fallback
dsh web   # restart to load the new bundle layer