cyanseek / dsh-tool-chaos

dsh-tool-chaos

English · 简体中文

Deterministic fault injection and evidence-backed resilience tests for the DeepSeek Harness tool pipeline.

dsh-tool-chaos runs an isolated DSH experiment through baseline → dry-run → fault injection → PASS / FAIL / INCONCLUSIVE. It helps plugin and Agent authors verify retry, fallback, cancellation, policy-denial, and untrusted-result behavior before those paths fail unexpectedly.

Status: 0.1.0-next.0 prerelease. DSH is still in developer preview, so pin versions and review the tested compatibility matrix.

Highlights

• Five DSH-native actions: deny, error, delay, abort, and block.
• Reproducible matching and sampling; no runtime randomness.
• Disabled and dry-run by default, with bounded injections and explicit global-wildcard opt-in.
• JSON and Markdown reports with structured decision events and a three-state verdict.
• Native Cordis plugin, zero-runtime-dependency CLI, TypeScript engine API, and Codex/Agent Skill surfaces.

Quick start

The npm prerelease has not been published yet. The currently available path is a source checkout and a locally built tarball:

git clone https://github.com/cyanseek/dsh-tool-chaos.git
cd dsh-tool-chaos
npm ci
npm run build
npm pack --silent

dsh plugin --profile chaos-demo add ./dsh-tool-chaos-0.1.0-next.0.tgz
dsh --profile chaos-demo --dump-config

The effective config should contain tool-chaos and tool-chaos-reporter. Both are inert after installation: the main plugin is disabled and the reporter is disabled.

Generate a complete dry-run overlay:

node bin/dsh-tool-chaos.mjs init .dsh-chaos/retry.patch.yml --preset retry

Review the generated tool matcher, task, and assertion, then run it against the disposable profile:

node bin/dsh-tool-chaos.mjs run \
  --mode dry-run \
  --profile chaos-demo \
  --patch .dsh-chaos/retry.patch.yml \
  --task "Use web_fetch and recover from one transient failure" \
  --expect "expected stable phrase" \
  --no-install \
  --json

A dry-run must show the intended dryRun: true decision while leaving tool execution unchanged. Run a real experiment only after that match is exact.

Install from a pinned GitHub commit

DSH can install this repository directly:

dsh plugin --profile chaos-demo add \
  github:cyanseek/dsh-tool-chaos#<commit-sha>

pnpm 10+ can intentionally block a Git dependency's prepare script. If it does, review the pinned source and copy the exact package key printed by pnpm into that profile's pnpm-workspace.yaml:

allowBuilds:
  dsh-tool-chaos@https://codeload.github.com/cyanseek/dsh-tool-chaos/tar.gz/<commit-sha>: true

Rerun the install and verify the effective config. This approval allows package code to execute during installation; a reviewed tarball remains the simpler option.

To uninstall and verify removal:

dsh plugin --profile chaos-demo remove dsh-tool-chaos
dsh --profile chaos-demo --dump-config

Configuration

Installed defaults cannot inject a fault:

- id: tool-chaos
  config:
    enabled: false
    dryRun: true
    seed: dsh-tool-chaos
    allowGlobalWildcard: false
    rules: []

A bounded preview rule looks like this:

- id: tool-chaos
  config:
    enabled: true
    dryRun: true
    seed: retry-suite-v1
    allowGlobalWildcard: false
    rules:
      - id: first-github-error
        tool: github_*
        action: error
        scope: root
        argumentsPattern: ''
        probability: 1
        afterMatches: 0
        every: 1
        maxInjections: 1
        delayMs: 1000
        message: deterministic GitHub transport failure

tool is an anchored, case-sensitive glob. argumentsPattern runs against stable key-sorted JSON. DSH patch rows replace a row's complete config, so every overlay must restate all keys it relies on.

Fault actions

Action	DSH pipeline phase	Behavior
deny	tools/pre-execute	Reject before dispatch
error	tools/execute	Return a structured injected error without running the tool body
delay	tools/execute	Add abort-aware latency
abort	tools/execute	Abort a derived signal and restore the upstream signal
block	tools/post-execute	Run the tool, then withhold its result

Each decision emits tool-chaos/decision. The optional reporter converts it to a stable stderr JSONL boundary prefixed with DSH_TOOL_CHAOS_EVENT .

Evidence and verdicts

The CLI writes a JSON source-of-truth report and a Markdown summary. Reports include configuration hashes, bounded process metadata, parsed chaos decisions, assertions, verdict, and a reproduction command. Raw command output is excluded unless explicitly requested.

Verdict	Meaning
PASS	The intended fault occurred and the declared recovery behavior is supported by evidence
FAIL	The intended fault occurred and at least one recovery assertion failed
INCONCLUSIVE	Baseline, installation, matching, safety, or observation evidence is incomplete

Exit code alone is never sufficient evidence of resilience.

CLI

dsh-tool-chaos doctor [--profile name] [--json]
dsh-tool-chaos setup [--profile name] [--source npm-or-path] [--codex] [--json]
dsh-tool-chaos init [path] [--preset retry|timeout|nested|result-block|deny] [--force]
dsh-tool-chaos run --patch file --task prompt [--mode baseline|dry-run|experiment]
                   [--expect regex] [--profile name] [--json]
dsh-tool-chaos install-plugin [--profile name] [--source npm-or-path] [--json]
dsh-tool-chaos install-skill [--global | --project path]
dsh-tool-chaos install-codex-plugin [--dir path] [--json]
dsh-tool-chaos verify [--json]
dsh-tool-chaos prompt [--lang en|zh] <requirement...>
dsh-tool-chaos codex [--cwd path] [--json] <requirement...>

Automation should consume --json instead of scraping human-readable output.

Codex and Agent Skill

The repository contains a Codex marketplace plugin and an open Agent Skill. Install the Codex plugin from a checkout:

codex plugin marketplace add /absolute/path/to/dsh-tool-chaos
codex plugin add dsh-tool-chaos@dsh-tool-chaos

Start a new thread and invoke:

$dsh-chaos-test Prove the retry path survives one deterministic GitHub tool error.

See Codex integration for the source-checkout workflow. npx dsh-tool-chaos ... commands become available only after the npm prerelease is published.

Programmatic engine

The dsh-tool-chaos/engine export has no Cordis or DSH runtime import:

import { ChaosEngine, resolveConfig } from 'dsh-tool-chaos/engine'

const engine = new ChaosEngine(resolveConfig({
  enabled: true,
  dryRun: false,
  seed: 'ci-retry-v1',
  rules: [{
    id: 'first-web-error',
    tool: 'web_fetch',
    action: 'error',
    maxInjections: 1,
  }],
}))

const decision = engine.decide('execute', {
  callId: 'call-1',
  rootCallId: 'call-1',
  name: 'web_fetch',
  arguments: { url: 'https://example.test' },
})

See the engine example.

Safety boundaries

• Use disposable profiles and synthetic or read-only test inputs.
• Keep dryRun: true until the exact intended match is visible.
• Exact global * requires allowGlobalWildcard: true.
• Every rule injects at most once by default.
• The runtime plugin does not rewrite arguments, read files, or access the network.
• abort is cooperative; a same-process tool that ignores AbortSignal cannot be hard-killed.
• Never represent skipped or unobservable checks as success.

Production/shared profiles, external mutations, credentials, broad targeting, or privilege changes require separate safeguards and explicit authorization.

Compatibility

Surface	Verified state
Tests	Strict TypeScript compile and 32 unit/CLI/real-runtime tests pass
Node	24.11.1 locally; GitHub CI passes on 22.19.0 and 24.x
DSH	@deepseek-ai/dsh-tools@0.1.0-rc.6 runtime integration passes
Tarball consumer	Install, repeated add, config dump, and read-only headless dry-run preview pass
GitHub source	Pinned commit install, repeated install, config dump, uninstall, and reinstall pass with pnpm 11.7.0
npm	Not published; no npm availability claim is made

Evidence was collected on August 14, 2026 with isolated profiles and synthetic read-only input. See Compatibility for the precise matrix and limitations.

Documentation

• CLI and machine API
• Report schema
• Architecture
• Codex integration
• Compatibility
• Contributing
• Security
• Support
• Changelog

Development

npm ci
npm run sync:skill
npm run check

Compatibility claims must be backed by the exact environment and command that passed.

License

MIT