HiSeax / dsh-image-to-text

dsh-image-to-text

English | 简体中文

Let text-only (non-multimodal) models accept images in DSH.

By default DSH refuses to send images to models whose inputModalities do not include image — you get Model does not support image input / MODEL_DOES_NOT_SUPPORT_IMAGES, and text-only adapters reject raw image blocks with UNSUPPORTED_CONTENT. This plugin solves that problem:

• Bypasses the image admission gate for non-multimodal models, so image uploads are accepted into the conversation.
• Rewrites image blocks into local file paths right before the request reaches the text-only adapter, so the model request stays text-only and never trips UNSUPPORTED_CONTENT.
• Registers the image_to_text model tool, which OCRs/describes those images through configurable vision API providers (multi-image, priority-based failover), then returns the recognized text to the model.

No DSH source code is modified — everything is a self-contained plugin.

Highlights

• 🔓 Images become sendable in DSH sessions even when the active model is not multimodal (pi-ai / DeepSeek and other text-only routes).
• 🖼️ image_to_text tool: OCR + description, multiple images per call, automatic provider failover.
• 🔧 Every provider supports custom apiKey and endpoint via plugin config or environment variables.
• 🔐 No API keys are bundled; a prepack safety check refuses to publish files that look like they contain real keys or local machine paths.
• 🌐 Works with any OpenAI-compatible or Gemini-style vision endpoint.

How it works

For models whose resolved inputModalities do not include image, the plugin:

1. wraps llm.resolveModelInfo so the image prompt passes the api-proxy admission check (config: bypassImageGate, default true);
2. wraps llm.streamWithRegistration (the shared entry for llm.stream and preparedCall.stream) — before the text-only adapter serializes the request, each ImageBlock is materialized to %TEMP%/dsh-image-to-text/<attachmentId>.<ext> and replaced with [Attached image: <path>] text (config: rewriteImageBlocks, default true).

Vision-capable models are left untouched.

Requirements

• DSH (DeepSeek Harness), web profile or any profile with Python access
• Python 3
• At least one vision API key (or a custom endpoint)

Install

dsh plugin --profile web add dsh-image-to-text

Then register the row in $DSH_HOME/profiles/web/cordis.patch.yml:

- insert:
    - id: image-to-text
      name: 'dsh-image-to-text'

DSH hot-reloads the patch. Installation is confirmed when image_to_text appears in the session tool list.

Configuration

Plugin row config fields:

Field	Default	Meaning
pythonPath	Windows probes C:\Program Files\Python313\python.exe, else python	Python executable path or PATH command
providers	built-in 7 provider templates	Override by name (optionally model), or add new providers
bypassImageGate	true	Wrap llm.resolveModelInfo to admit image prompts
rewriteImageBlocks	true	Rewrite image blocks to local paths before text-only adapters

Custom API URL / key (plugin config)

- insert:
    - id: image-to-text
      name: 'dsh-image-to-text'
      config:
        pythonPath: 'python'
        providers:
          # Override a built-in provider: only name (and optionally model) is needed
          - name: zhipu
            apiKey: 'your-zhipu-key'
            endpoint: 'https://open.bigmodel.cn/api/paas/v4/chat/completions'
          # Add a fully custom OpenAI-compatible provider
          - name: my-vision
            model: 'vision-model'
            apiKey: 'sk-xxx'
            endpoint: 'https://api.example.com/v1/chat/completions'
            format: openai
            priority: 1

Built-in provider names: zhipu, gemini (shared by several Gemini models), grok. New providers need the full name/model/apiKey/endpoint/format/priority.

Custom API URL / key (environment variables)

Named by provider name (non-alphanumeric → _, uppercase):

IMAGE_TO_TEXT_API_KEY_ZHIPU=your-key
IMAGE_TO_TEXT_ENDPOINT_ZHIPU=https://open.bigmodel.cn/api/paas/v4/chat/completions

Environment variables are read on every call and never written to disk.

Runtime files

On startup the plugin merges the template with your provider overrides and writes:

$DSH_HOME/image-to-text/config.json   runtime config (env-provided keys are not persisted)
$DSH_HOME/image-to-text/state.json    Gemini daily quota accounting

Uploaded images are materialized before each model request to:

%TEMP%/dsh-image-to-text/<attachmentId>.<ext>

Usage

Upload an image in a session, or the model sees an image in context and calls image_to_text:

{
  "images": ["C:\\path\\to\\image.png"]
}

Optional arguments: model (force a specific model), prompt (custom prompt). Returns:

{
  "text": "OCR and description",
  "provider": "gemini",
  "model": "gemini-3.6-flash"
}

Development & publishing

npm pack --dry-run   # preflight: prepack refuses packages that may contain real keys
npm publish          # publish to npm

The published package contains no real API keys. All api_key values in assets/image-to-text/config.example.json are empty strings, and scripts/check-no-keys.mjs runs on prepack to block common real-key patterns.

When releasing updates, bump version in package.json; users who want to force a hot reload can also increment the row's config.revision in cordis.patch.yml.

Notes

• bypassImageGate and rewriteImageBlocks wrap internal methods of the current DSH implementation (llm.resolveModelInfo, llm.streamWithRegistration), not public Service APIs; they may need adaptation after DSH upgrades.
• If every provider fails, the tool reports an error — check network, quota, and configuration.
• Gemini free tier has per-model daily quotas; state.json accounts by Pacific date; deleting it resets today's quota.