Culeot / dsh-batch-pipeline

dsh-batch-pipeline

A batch-task pipeline plugin for DeepSeek Harness: hand the agent a list of work (50 essays, 60 years of PDFs, 100 listings), and it tracks progress, resumes after restarts, stops when the budget runs out, and checks each product file.

What problem it solves

You often ask the agent to do "a big list of similar work". The common trait: one session cannot finish it — 50 essays, interrupted at #30, session restarted, progress gone, start over.

dsh-batch-pipeline gives that work a track:

• Hand it the list; it remembers which item you're on
• Session died and restarted? It tells you where to resume
• Budget exceeded? It stops you before you burn through
• Each item finished, it inspects the product (file exists? non-empty?), flags failures instead of silently passing them

Why not just count files (read this first)

You may wonder: can't the plugin scan the directory and count files itself? Why must the agent report?

Because "written" ≠ "file exists". The agent may write halfway, save to the wrong path, or keep it in memory without persisting — a directory scan shows "exists or not", not "done or not". So the design is: the agent reports "I finished item N, product is here" after each item, and the plugin records it. If the index matches, it advances; if not, it rejects and tells the agent which item it should be doing.

Features

• Batch list: create once with all items (≤200). Each item is one line the agent understands.
• Progress tracking: each item reported on completion, persisted across restarts.
• Resume after restart: new session says "continue", picks up from the last progress.
• Budget circuit-breaker: per-item cost estimated, auto-pause when cumulative spend exceeds the cap.
• Product assertions: each report checks the product exists + non-empty; failures are flagged, not silently passed.
• Automatic stage flow: after the last item, auto-enters the finishing stage — do the assembly (packaging/merging/indexing), then archive.

Tools

Tool	Purpose
batch_create	Create a task: name + list + optional budget + optional assert rules
batch_status	Progress: current item, spend, estimated total, recent failures
batch_report	Report one item done: index + ok/fail + product path
batch_pause	Pause
batch_resume	Continue (takeover=true to claim after a restart)
batch_finish	Archive: returns success/failure stats

Install

⚠️ Not yet published to npm. Install from a local path with file:.

# 1. add to your profile
cd ~/.dsh/profiles/<name>
pnpm add dsh-batch-pipeline@file:/path/to/dsh-batch-pipeline

# 2. add a row in the agent preset (~/.dsh/.agent-presets/<preset>/agent.cordis.yml)
- id: batch
  name: 'dsh-batch-pipeline'

# 3. restart DSH

No preset? Mount on the profile host plane (~/.dsh/profiles/<name>/cordis.patch.yml):

- insert:
    - id: batch
      name: 'dsh-batch-pipeline'

Prerequisite: the storage trio (dsh-storage, dsh-storage-json, dsh-storage-domain — web profile ships with them).

Usage examples

Write 50 essays:

batch_create name=putonghua-essays items=[topic1, topic2, ..., topic50] minBytes=500
# after each essay:
batch_report itemIndex=current ok=true product=essays/topicN.md
# all done → auto finishing → archive:
batch_finish

Download 60 years of contest papers:

batch_create name=IMO-papers items=[1959, 1960, ..., 2026]
# after each year:
batch_report itemIndex=current ok=true product=IMO/2024/IMO_2024_chs.pdf
batch_finish

Configuration

Option	Default	Description
budgetUsd	5	Spending cap in USD (estimated)
injectEnabled	true	Inject batch progress (current item/budget) each step

How the budget estimate works (the numbers)

• Estimate per item: (item chars + injection chars) ÷ 4 × input price + product bytes ÷ 4 × output price
• 4 chars ≈ 1 token (CJK/English approximation); prices at the DSH flash tier (input $0.14/M, output $0.28/M)
• ±50% estimate error — it's a magnitude reference, not a precise ledger; crossing the cap hard-pauses
• Your current batch jobs (50 essays) cost roughly $0.03–0.06; the budget guard is against "accidental runaway", not day-to-day spend tracking

Boundaries (v0.1)

• One task at a time: finish/pause the current task before creating a new batch
• Lists ≤200 items: larger lists (v0.2) via JSON file reference
• Single-session progress: two sessions reporting concurrently would desync; a new session resuming must pass takeover=true
• No UI: progress via tools + per-step injection; a panel is a v0.2 idea

Development

npm install && npm run build && npm test   # build + 32 unit tests
npm run smoke                              # headless end-to-end (incl. cross-session resume)

License

MIT