cyijun / dsh-surfing-plugin

dsh-surfing-plugin

English | 中文

dsh-surfing-plugin adds self-hosted web access to DeepSeek Harness: web_search uses SearXNG and web_fetch uses Crawl4AI. It registers providers with DSH's ctx.web service, so the native tool names, arguments, rendering, timeouts, cancellation, and result limits remain unchanged.

Architecture

flowchart LR
  A[Native DSH web_search] --> B[surfing-searxng provider]
  B --> C[SearXNG /search]
  D[Native DSH web_fetch] --> E[surfing-crawl4ai provider]
  E --> F[Crawl4AI /crawl]

The bundled cordis.patch.yml mounts this plugin, selects surfing-searxng and surfing-crawl4ai, and adds a fetch-only native tool consumer. The separate consumer works in both DSH assemblies: headless keeps its host-level web_search, while the Web UI keeps the web_search mounted by each Agent Preset. The built-in DeepSeek search provider may remain mounted but is not selected.

Requirements

• Node.js ^22.19.0 or >=24.0.0
• DeepSeek Harness >=0.1.0-rc.6 <0.2.0
• Reachable SearXNG and Crawl4AI services
• JSON enabled in SearXNG's search.formats

Quick start

The endpoint values may be service roots or complete /search and /crawl URLs:

export SEARXNG_URL=http://127.0.0.1:8080
export CRAWL4AI_URL=http://127.0.0.1:11235

# Current Crawl4AI releases enable Bearer authentication by default.
export CRAWL4AI_API_TOKEN=replace-with-your-token

Install a local checkout into the web profile:

dsh plugin --profile web add .
dsh --profile web --dump-config
dsh --profile web

After npm publication:

dsh plugin --profile web add dsh-surfing-plugin

Remove it with dsh plugin --profile web remove dsh-surfing-plugin.

Configuration

Explicit configuration wins over environment values. Override this plugin's row in $DSH_HOME/profiles/web/cordis.patch.yml:

- id: surfing-plugin
  config:
    searxng:
      url: https://search.example.com
      apiKeyEnv: MY_SEARXNG_KEY
      authHeader: X-API-Key
      authScheme: ''
      language: en
      categories: general,news
      safeSearch: 1
      timeRange: month
    crawl4ai:
      url: https://crawl.example.com
      apiKeyEnv: CRAWL4AI_API_TOKEN
      authHeader: Authorization
      authScheme: Bearer
      markdownMode: raw
      maxContentChars: 100000

Field	Environment or default	Meaning
searxng.url	SEARXNG_URL	Service root or /search endpoint
searxng.apiKey	none	Optional literal key
searxng.apiKeyEnv	SEARXNG_API_KEY	Environment variable containing the optional key
searxng.authHeader	Authorization	Authentication header
searxng.authScheme	Bearer	Authentication prefix; empty sends the key directly
searxng.language	server default	SearXNG language parameter
searxng.categories	server default	Comma-separated categories parameter
searxng.safeSearch	server default	0, 1, or 2
searxng.timeRange	none	day, month, or year
crawl4ai.url	CRAWL4AI_URL	Service root or /crawl endpoint
crawl4ai.apiKey	none	Optional literal key
crawl4ai.apiKeyEnv	CRAWL4AI_API_TOKEN	Environment variable containing the optional key
crawl4ai.authHeader	Authorization	Authentication header
crawl4ai.authScheme	Bearer	Authentication prefix; empty sends the key directly
crawl4ai.markdownMode	raw	Prefer raw, fit, or citations markdown
crawl4ai.maxContentChars	100000	Provider content cap before returning to DSH

A literal apiKey takes precedence over the environment variable named by apiKeyEnv. No authentication header is sent when no key is available.

Provider behavior

The SearXNG provider sends form-encoded POST /search requests with format=json. It keeps absolute HTTP(S) result URLs, deduplicates by URL, maps title, content, and publishedDate into DSH sources, and exposes non-empty SearXNG answers as result content.

The Crawl4AI provider sends the minimal { "urls": [url] } body to POST /crawl; model input cannot supply browser or crawler configuration. Only HTTP(S) targets are accepted. Target non-2xx status codes remain successful DSH fetch results, while Crawl4AI API failures become structured WebError values.

fit and citations modes fall back to raw markdown when their preferred field is empty. HTML is returned only when no markdown representation exists.

Security

• Prefer apiKeyEnv; never commit credentials.
• Use HTTPS for non-loopback services.
• Crawl4AI owns browser isolation, target-network access, and SSRF policy. Restrict its network and authentication before exposing it.
• Backend redirects are rejected so credentials are not forwarded to another endpoint.

GitHub installation

Git installation builds from source through prepare. Pin a commit:

dsh plugin --profile web add github:cyijun/surfing-plugin#COMMIT_SHA

pnpm 10 and newer require build-script approval for Git dependencies. If the first install is blocked, copy its exact package key into the profile's pnpm-workspace.yaml and retry:

allowBuilds:
  dsh-surfing-plugin: true

npm packages and pnpm pack tarballs already contain lib/ and do not need this approval.

Development and publishing

corepack pnpm install
corepack pnpm run check
corepack pnpm pack

CI runs on main and pull requests. Tags matching v* trigger the npm Trusted Publishing workflow. Before the first release, configure this repository and publish.yml as an npm Trusted Publisher and confirm that the dsh-surfing-plugin package name remains available.

License

MIT