Nico0713520 / dsh-research

dsh-research

Local, free, keyless research layer for dsh coding agents.

Other tools give the agent a chunk of page text. dsh-research gives it a filesystem. GitHub repos shallow-clone into real source trees; web pages and PDFs become clean markdown files in your workspace. The agent navigates them with the grep/read/ls tools it already has — no API spend, no proprietary content jail, clones cached across sessions.

$ agent: "fetch_content https://github.com/liliMozi/openhanako"
→ Materialized → .cache/dsh-research/repos/liliMozi/openhanako
  (real source tree — grep it, read any file, zero context cost)

Why

Coding agents researching external material face two bad options: fetch a page and dump 100k characters into context, or paste HTML crumbs. Firecrawl's MCP solves this in the cloud, per-request, with a key. pi-web-access solves it inside a closed artifact store the agent can only page through.

Materialization solves it locally: external content becomes plain files at stable paths, the context window only receives paths plus a preview, and the agent pays tokens for exactly what it greps. Fetch the same URL tomorrow — cache hit, zero cost.

	Firecrawl MCP	pi-web-access	dsh-research
Content destination	their cloud	closed artifact store	your filesystem
Cost	per-request	API keys	free, local
Access granularity	what their API returns	page-by-page	any grep/read
GitHub repos	indexed copy	shallow clone	shallow clone, cross-session cache

Tools

Tool	What it does
github_search	GitHub site search (repositories keyless, code with token). Returns full name, stars, URL.
fetch_content	Materialize a URL: GitHub → shallow clone; PDF → markdown via dsh-doc-to-markdown; web → markdown via dsh's ctx.web seam. Returns paths + preview.
read_response	Sequential paging fallback for the rare linear read. Grep is the primary path.

Web search is deliberately not here — dsh's packages/web family already owns web_search.

Storage layout

<workspace>/.cache/dsh-research/
├── repos/<owner>/<repo>/    # shallow clones, reused across sessions
├── pages/<url-hash>/
│   ├── content.md           # the materialized markdown
│   └── index.json           # provenance (url, kind, time)
└── tmp/                     # in-flight PDF downloads

Config

- id: dsh-research
  name: '@ajin/dsh-research'
  githubToken: <optional PAT, env fallback GITHUB_TOKEN>
  previewChars: 1500

Requirements

• Node.js >= 18, git on PATH
• PDF conversion: Python 3.9+ with pymupdf4llm (shared with dsh-doc-to-markdown)

Development

npm install
npm test        # vitest

Status

0.2.0 — materialization pipeline complete: three sources, hash-addressed cache, cache-hit short-circuit, path-scoped read_response.