zhuiyueya / dsh-visionary

---

✨ Why you need it

DeepSeek's API is text-only. Paste an image into the chat and the harness rejects it outright (UNSUPPORTED_CONTENT) — no screenshots, no documents, no photos.

dsh-visionary sits at the exact boundary where that rejection happens and rewrites the request: every image block becomes a structured text block (exact OCR transcription + a detailed vision-model description) before the adapter ever sees it. The user-facing chat keeps the original image; only the model-facing request is converted.

⚡ Features

• 🔌 Transparent & framework-level — hooks the single chokepoint every LLM call passes through (streamWithRegistration); works for web, headless, and sub-agents alike
• 🖥️ Configure in the Models settings page — six vision providers are pre-registered (GLM-4V, Qwen-VL, SiliconFlow, OpenRouter, Gemini, Ollama); add one like any chat model, no config files
• 🔁 Multi-backend fallback chain — try free → paid → local Ollama in order; the first success wins, failures are aggregated and never block the conversation
• 🔤 OCR + VLM hybrid — screenshots/documents go through exact local OCR; natural images get a structured VLM description (with OCR text as cross-check context)
• 🧠 Capability-aware — if the target model declares image input modality, the plugin passes through untouched
• 🗃️ Double-layer caching — session-level (vision/describe events) + global KV (~/.dsh/storages): the same image is converted exactly once across turns and sessions
• 🛡️ Prompt-inflation guards — per-description maxChars truncation, per-request image budget, optional downscaling of huge images before VLM dispatch
• 💪 Failure-safe — a failed conversion becomes an explicit placeholder the model can report to the user; strict mode available

🔧 How it works

flowchart LR
    U[User pastes image] --> S[Attachment store<br/>bytes + metadata]
    S --> M[User message<br/>with image block]
    M --> B{dsh-visionary<br/>at LLM boundary}
    B -->|model supports image| P[Pass through<br/>unchanged]
    B -->|text-only model| T[Convert each image block]
    T --> C{Cache hit?}
    C -->|yes| F[Reuse converted text]
    C -->|no| O[OCR - local tesseract / HTTP service]
    O --> V[VLM fallback chain<br/>GLM-4V → Qwen-VL → Ollama...]
    V --> F
    F --> R[Text-only model answers<br/>as if it saw the image]

• Interception point: LlmRuntime.streamWithRegistration — the shared exit of both llm.stream() and prepareCall().stream()
• Model-side vs user-side separation: the session log keeps the original image (UI unchanged); only the model request is rewritten
• Image → text format:

[用户上传的图片 <attachmentId>, 1024x768, image/png]
── OCR 识别文本 ──
<exact text from the image>
── 视觉模型描述（glm-4v-flash）──
<structured description>

🚀 Quick start

# 1. Install the plugin into a profile
dsh plugin --profile web add dsh-visionary

# 2. Activate it in ~/.dsh/profiles/web/cordis.patch.yml

- insert:
    - id: visionary
      name: dsh-visionary

Restart dsh web. The plugin is now live — screenshots work with zero configuration (built-in tesseract OCR). Add a vision model for full effect:

🖥️ Configure vision models in the Settings page

Open Settings → Models, exactly like configuring DeepSeek itself:

1. Click Add model and pick from the catalog:

   Catalog entry	Default endpoint	Notes
   GLM-4V-Flash (Zhipu, free)	https://open.bigmodel.cn/api/paas/v4	free, strong Chinese, recommended first
   Qwen-VL (Alibaba DashScope)	https://dashscope.aliyuncs.com/compatible-mode/v1	top-tier Chinese
   Qwen2.5-VL (SiliconFlow)	https://api.siliconflow.cn/v1	new-user credits
   OpenRouter vision (incl. :free)	https://openrouter.ai/api/v1	one key, many models
   Gemini vision (Google)	https://generativelanguage.googleapis.com/v1beta/openai	large free tier
   Ollama local vision	http://localhost:11434/v1	offline, private, no key

2. Paste the API key (stored in the DSH credential vault — the Fetch models button can auto-discover the model list).

3. Confirm the base URL and model id, save. Done — no YAML touched.

Multiple configured vision models automatically form an ordered fallback chain (catalog order = priority). Configured models stay editable/removable on the page. Any model declared with image input modality on any provider route is picked up automatically.

OCR tuning, modes, and caching are advanced options under the vision: settings namespace — the defaults work out of the box.

🧭 Modes & fallback chain

Mode	Behavior	Best for
auto (default)	OCR first; if OCR yields ≥ ocr.minChars chars the image is text-dominant → OCR alone. Otherwise the VLM describes it, with OCR text as cross-check context	everything
ocr	OCR only	screenshots, documents, code, tables
vlm	Vision model only	photos, natural images, diagrams
both	OCR and VLM (VLM corrects/extends the OCR)	documents needing semantic reading

Fallback order for each image: configured Models-page providers → vlm.backends — each backend is tried until one succeeds; timeouts and API errors roll to the next; total failure becomes a visible placeholder (or a hard error in strict mode).

⚙️ Advanced configuration

All options live in settings.yaml under vision: (hot-reloaded, editable in the UI):

vision:
  mode: auto            # auto | ocr | vlm | both
  ocr:
    engine: auto        # auto(http→rapid-json→tesseract) | tesseract | http | rapid-json | none
    languages: [chi_sim, eng]   # tesseract language packs
    minChars: 20        # auto mode: OCR ≥ this many chars = text-dominant, skip VLM
    # http: { url: "http://127.0.0.1:8000/ocr" }   # self-hosted PaddleOCR/RapidOCR
    # binaryPath: "/path/to/RapidOCR-json"         # Windows/Linux high-precision OCR
  vlm:
    enabled: true
    maxChars: 2000       # hard cap per description (prompt constraint + truncation)
    maxPixels: 1500000   # downscale above this before VLM dispatch (needs sharp)
    prompt: |            # default detailed Chinese prompt; override freely
      请以中文详细描述这张图片……
    backends:            # legacy ordered fallback chain (Models page takes priority)
      - name: local-ollama
        baseUrl: http://localhost:11434/v1
        model: qwen2.5vl
  skipVisionModels: true         # vision-capable target models pass through
  cache: { session: true, global: true, maxAgeMs: 2592000000 }
  concurrency: 4                 # parallel image conversions per request
  maxImagesPerRequest: 10        # per-request image budget (skip marker beyond)
  placeholderOnError: true       # false = fail the LLM request loudly

OCR engines

Engine	Description
tesseract	built-in, pure JS + WASM, zero native deps; language packs auto-download (set langPath for offline)
http	any self-hosted OCR service (PaddleOCR/RapidOCR FastAPI, Dify OCR node) — best Chinese accuracy; POSTs {"image":"data:…"} expecting {"text":"…"}
rapid-json	RapidOCR-json binary (Windows/Linux, high-precision Chinese)
none	disable OCR (pure VLM mode)

API key resolution

apiKey (literal) → apiKeyEnv environment variable → DSH credentials service. No key is legal for local servers (Ollama); a keyless cloud call fails with 401 and rolls to the next backend.

🗃️ Caching

• Session layer: vision/describe events in the session log — the same image in the same session costs zero repeated OCR/VLM calls across every turn
• Global layer: KV unit in ~/.dsh/storages — the same image in a new session is not re-described
• Cache keys fingerprint the exact conversion configuration (mode, OCR engine, backend endpoints/models, prompt, caps) so changing a model invalidates stale entries automatically

❓ FAQ

Does the user still see their image? Yes — the chat log keeps the original; only the model-facing request carries text.

Does it cost money? OCR is local and free. The default VLM (GLM-4V-Flash) is officially free; you control the chain (free → paid → local Ollama).

Is my image sent to third parties? Only when you configure a cloud VLM. With mode: ocr (or local Ollama) everything stays on your machine.

Why not a tool-based approach? Tool-based plugins depend on the model remembering to call a tool (and usually require hard-coded rules). This is push-based: conversion happens before the adapter, unconditionally — the model never has the option to fail.

Multiple images in one message? Converted in parallel (bounded by concurrency), cached, budget-capped.

🧪 Development

git clone git@github.com:zhuiyueya/dsh-visionary.git
cd dsh-visionary && npm install

# end-to-end tests (real tesseract OCR + mock VLM server + transformation asserts)
node test/e2e.mjs          # 20/20 PASS

# real DeepSeek API integration (run from a profile dir, see script header)

📌 Topics

Recommended GitHub tags for discoverability:

deepseek-harness  dsh-plugin  vision-bridge  image-understanding
multimodal  ocr  vlm  text-only-llm  llm-plugin  glm-4v  qwen-vl  ollama

📜 License

MIT

🙏 Credits

Architecture informed by studying OpenClaw's media-understanding (push-based pre-processing), opencode's image normalization, block/goose's local-inference modality gating, and vision-bridge-mcp's fallback chain — synthesized into a native DeepSeek Harness plugin.