NotExpectedYet
040c69b57d
The configuration page registered into seats that 0.1.7 removed, so it could
not appear at all — and the schema carried none of the metadata the new
settings document requires.
Host half. `ctx.settings.installSection` no longer exists: a plugin exports a
`Config` schema, the Harness keys one form per composed entry by that row's
`id`, and `ctx.settings.describe` skips an entry with no `volatile()` field
outright. Every field is now volatile, which is also what makes a saved edit
apply to the running instance instead of remounting it — so `apply` reads the
section through `settingsSnapshot`, which unwraps the live references a
volatile field arrives as. Writes go through `ctx.settings.update` fenced by
the revision `describe` reports. The `.min`/`.max` bounds the schema can state
are stated there, so the form refuses an impossible draft in front of the
person typing it; `validateSettings` keeps the cross-field rules no single
field can express, and now runs on the section a call is about to act on
rather than at write time, because 0.1.7 has no write-time validator hook.
Browser half. `settingsScope` and the `settings.plugin.item` slot are gone. The
page renders through `plugins.row.config`, keyed `<bundle package>#<row id>`
(`dsh-recall#dsh-recall`), which is what puts a Configure control on the
plugin's row in the Plugins panel. It uses the framework's own staged form —
`SettingsFormModel` over `ctx.configForms`, drawn by `SettingsForm` and
`SettingsValueField`/`SettingsSecretField` — instead of reproducing card chrome
by hand, so the unsaved, overridden, invalid, and save states come from the
same code as every other plugin's page. All 33 settings are editable now; the
old card exposed 23. The right-sidebar tab also needed the stage-one tab TYPE
(`ctx.sidebarRightTabs.register`) that its keyed body slot is dispatched under,
so a tab can finally be opened. `--dsh-border`, defined nowhere, becomes the
product's `--dsw-alias-border-l2`.
Both halves are now pinned by tests: every field is volatile, the field list
and the schema cannot drift, the page claims the key the patch declares, and a
blank API key is never written — the secret control reports configured-and-
absent instead.
Verified against the harness rather than only in unit tests: `--dump-config-
schema` emits the volatile and secret metadata, `apply` registers `recall` on
the real tool registry, `settings.describe({redactSecrets:true})` serves
`dsh-recall` with 33 fields and the key reported set-but-redacted, and a
scratch profile boots the web app with the plugin composed and its bundle
served byte-for-byte as this checkout has it.
391 lines
18 KiB
Markdown
391 lines
18 KiB
Markdown
# dsh-recall
|
||
|
||
Local-embedding code recall for **DeepSeek Harness**. One tool, `recall`, that
|
||
answers "where is this handled?" with ranked **excerpts** — file path, line range,
|
||
and a bounded slice of the code — instead of you reading whole files into the
|
||
conversation.
|
||
|
||
## Why this exists
|
||
|
||
A long agent session does not get expensive because the model reasons. It gets
|
||
expensive because **every step re-sends the accumulated transcript**, and in a
|
||
coding session most of that transcript is file contents the agent has already
|
||
read.
|
||
|
||
Measured on one real DSH install over 32 days, from the `dsh-cost-meter` ledger:
|
||
|
||
| | |
|
||
|---|---|
|
||
| total spend | **$160.27** (32 days — $5.01/day, ~$150/month) |
|
||
| model calls | 39,804 |
|
||
| prompt cache hit rate | **99.18%** — caching was already saturated |
|
||
| cost from **re-reading cached context** | **65.1%** |
|
||
| cost from new input | 16.9% |
|
||
| cost from output (60% of it reasoning) | 18.0% |
|
||
| **share of context that is message history** | **96.0%** (system 0.8%, tools 3.2%) |
|
||
| average context re-read **per call** | **332,000 tokens** |
|
||
|
||
That reframes the problem. Caching had nothing left to give, and trimming the
|
||
system prompt or tool definitions would have chased 4% of the context. The money
|
||
was in **how much history each step carried**. A tool that adds a 40-line excerpt
|
||
instead of a 900-line file does not just save those tokens once — it saves them
|
||
on *every subsequent step of the session*, because the context it never added is
|
||
never re-sent.
|
||
|
||
## What it does, measured
|
||
|
||
Run against `ha-runtime` (684 indexable files) with the default 8-result limit:
|
||
|
||
```
|
||
query : how does the occupancy state machine decide a room is occupied
|
||
returned : 13,075 chars ~3,269 tokens
|
||
source : 78,669 chars ~19,667 tokens across the 4 files the results came from
|
||
ratio : 6.0x smaller
|
||
```
|
||
|
||
All eight results landed inside `src/controllers/room/occupancy/state_machine/`
|
||
— the correct module — best cosine similarity **0.61**. Scoring a warm index takes
|
||
about a millisecond; the only per-query network cost is one embedding call.
|
||
|
||
## Install
|
||
|
||
```sh
|
||
dsh plugin --profile web add /path/to/dsh-recall
|
||
```
|
||
|
||
Then point it at your embedding endpoint. Anything Ollama-compatible works, and
|
||
the endpoint must be stated because `127.0.0.1:11434` is a convention rather than
|
||
a fact:
|
||
|
||
```yaml
|
||
# your profile's cordis.patch.yml
|
||
- id: dsh-recall
|
||
name: dsh-recall
|
||
config:
|
||
embeddingApiBase: http://10.50.1.2:11434
|
||
```
|
||
|
||
Or edit every setting from the UI: **Plugins → dsh-recall → Configure**. The
|
||
plugin ships **two halves**, and the page appears only when both are present and
|
||
the deployment serves the namespace:
|
||
|
||
- the **host half** (`index.js`) exports a `Config` schema whose every field is
|
||
`.volatile()`. That schema *is* the settings page's description — the Harness
|
||
keys one form per profile entry by that row's `id` (`dsh-recall` in
|
||
`cordis.patch.yml`), and it skips an entry with no volatile field entirely, so
|
||
a schema without those markers has no settings surface at all rather than an
|
||
empty one;
|
||
- the **browser half** (`client/client.js`) renders that form into the Plugins
|
||
panel's per-row configuration seat, `plugins.row.config`, keyed
|
||
`<bundle package>#<row id>` — here `dsh-recall#dsh-recall`. Registering it is
|
||
what adds the **Configure** control to the plugin's row.
|
||
|
||
A served namespace with no page draws no control, and a page whose namespace is
|
||
not served is deliberately not registered — so both are required. A restart is
|
||
needed for a newly installed browser half to appear, because client bundle bytes
|
||
are snapshotted when the profile activates.
|
||
|
||
Editing in the UI writes to the profile's own patch layer through the settings
|
||
document, so the value survives a restart and stays visible in
|
||
`cordis.patch.yml`.
|
||
|
||
Every value is also checkable from the tool, without touching a file:
|
||
|
||
```
|
||
recall action=configure config={"embeddingApiBase":"http://10.50.1.2:11434"}
|
||
```
|
||
|
||
The API key is declared `role('secret')`, so it is stripped from every settings
|
||
response: the page can show that a key is set without ever receiving it, and a
|
||
blank draft writes nothing — leaving it alone means "unchanged", never "clear".
|
||
|
||
Pull the model once:
|
||
|
||
```sh
|
||
ollama pull unclemusclez/jina-embeddings-v2-base-code
|
||
```
|
||
|
||
## The tool
|
||
|
||
`recall` — three actions plus configuration.
|
||
|
||
| action | what it does |
|
||
|---|---|
|
||
| `search` *(default)* | answers `query`, warming a bounded slice of the index first |
|
||
| `index` | builds or refreshes the whole index; **background job by default** |
|
||
| `status` | index and endpoint state, touching nothing |
|
||
| `configure` | writes settings through the settings document |
|
||
|
||
Parameters: `query`, `action`, `limit`, `path` (restrict to a subdirectory),
|
||
`run_in_background`.
|
||
|
||
### What it shows you
|
||
|
||
Two surfaces, both reading a host projection rather than polling:
|
||
|
||
- a **chip in the session header** — `Recall 3 · 40× · 457 KB kept out`. No price
|
||
on it, deliberately: at the measured rate a whole conversation's avoided
|
||
context is a fraction of a penny, and a chip advertising that reads as the tool
|
||
being useless rather than as the arithmetic being honest. The tooltip carries
|
||
the full comparison.
|
||
- a **right-sidebar Recall tab** — the same figures with room for what they are
|
||
not: *"An estimate, and an upper bound: it assumes a whole file would have been
|
||
read each time. Token counts are bytes ÷ 4."*
|
||
|
||
Both render nothing when there is nothing to say — a conversation that never used
|
||
recall costs no layout at all, which matters for the header corner because the
|
||
framework lays it out only while its occupant renders something.
|
||
|
||
The data path is `presentationMeta` → the `tool/result` event's `meta` → a
|
||
projection unit (`recallUsage`) with a wire view the browser reads through
|
||
`useProjection`. No new event type, no side channel, and nothing leaves the
|
||
machine.
|
||
|
||
The tool description steers the model on when *not* to use it: prefer `grep` when
|
||
you already know the symbol, `read` when you need the whole file, and do not
|
||
follow up with a `read` of a range you were just handed.
|
||
|
||
## Focus: letting a conversation see less
|
||
|
||
A workspace-wide index has to rank across everything, and in a codebase built
|
||
from many near-identical modules that is where embedding similarity fails. On the
|
||
repository this was written against there are **18 controllers** — same idioms,
|
||
same naming, same shape — across 228k lines, and a query about one returns
|
||
plausible fragments of all of them.
|
||
|
||
Narrowing the index to one of them removes the confusability rather than ranking
|
||
through it:
|
||
|
||
| | |
|
||
|---|---|
|
||
| median controller scope | **114 chunks, 2.4% of the workspace** |
|
||
| candidate pool | 4,738 → 114 chunks, a **41.6× smaller haystack** |
|
||
| cold start (projected from measured throughput) | **3.1 min → 5s** |
|
||
|
||
Set it per call, or configure a default:
|
||
|
||
```
|
||
recall action=index focus=src/controllers/room/occupancy
|
||
```
|
||
|
||
```yaml
|
||
focus: src/controllers/room/occupancy # or leave empty to resolve by marker
|
||
focusMarkers: [CONTROLLER.md] # files that make a directory its own scope
|
||
```
|
||
|
||
Resolution is most-explicit-first: an explicit `focus`, then the nearest
|
||
enclosing directory carrying a marker file, then the workspace. The last fallback
|
||
is the pre-focus behaviour, so adopting this cannot regress a deployment that
|
||
never configures it. **`recall status` lists every scope it can find**, so
|
||
choosing one is a matter of reading the list.
|
||
|
||
Two things worth knowing, both measured:
|
||
|
||
- **Scoping hides shared code.** 41.6% of that workspace — 1,972 chunks of shared
|
||
types, services and infrastructure — sits outside every controller. A weak
|
||
result inside a scope says so and tells you to retry without it, because a
|
||
scope is a window, not a relevance filter.
|
||
- **Choosing a scope is explicit, never inferred.** A session sitting at the
|
||
repository root is not thereby *about* any one controller, and silently
|
||
confining it to whichever scope sorted first would hide code it needs. So
|
||
`status` lists them and a caller names one.
|
||
|
||
## Two opt-in local-model steps
|
||
|
||
Both are **off by default**, and both are built so that any failure is a
|
||
fallback rather than an error:
|
||
|
||
| setting | what it does | default |
|
||
|---|---|---|
|
||
| `expansionEnabled` | asks the local model for alternative phrasings, ranks each, and fuses the orders | `false` |
|
||
| `rerankEnabled` | has the local model score the top candidates against the query | `false` |
|
||
|
||
They are off because **their benefit has not been measured against a real
|
||
model**, and a feature that spends latency on an unquantified gain is not a
|
||
default worth having. Turn them on and run `scripts/baseline.mjs` if you want to
|
||
find out.
|
||
|
||
### Why the failures matter more than the features
|
||
|
||
Both run on the critical path of a search, against a model on another box that
|
||
can be slow, wedged, or absent. That is not hypothetical here — the endpoint
|
||
this was developed against spent the whole project in exactly that state. So the
|
||
contracts are narrow:
|
||
|
||
- **expansion always returns the original query**, first, whatever happens. The
|
||
worst case is a search with no expansion, never a failed search.
|
||
- **re-ranking returns `undefined` for "no opinion"**, which is deliberately not
|
||
the same as zero. Replacing a good embedding score with a zero because the
|
||
model timed out would turn a working search into an empty one.
|
||
- both use their **own deadline** inside the caller's signal, so a slow local
|
||
model gives up and lets the search continue: cancelling mean stop, being slow
|
||
does not.
|
||
- a non-finite timeout is **clamped, not propagated** — `AbortSignal.timeout(NaN)`
|
||
throws, and the catch below it would report that as "the model had nothing to
|
||
say". A misconfiguration must not wear the disguise of a slow model.
|
||
|
||
Tested, including the wedged case: a socket that opens and never answers must
|
||
give up on its own deadline rather than pass the wait on.
|
||
|
||
## Measuring it
|
||
|
||
`scripts/baseline.mjs` is the before/after instrument, and it reads two stores
|
||
because they answer different questions:
|
||
|
||
```sh
|
||
node scripts/baseline.mjs # human report
|
||
node scripts/baseline.mjs --json # machine readable
|
||
```
|
||
|
||
- the **cost ledger** knows requests, cache-read tokens, and therefore what each
|
||
request carried;
|
||
- the **session projection cache** knows the surface each session reached, and
|
||
whether it used recall at all.
|
||
|
||
```
|
||
BASELINE
|
||
sessions 109
|
||
requests 39,706
|
||
avg context/request (pooled) 339,182 every request counts once
|
||
avg context/request (typical) 155,175 every session counts once
|
||
cost $160.11
|
||
```
|
||
|
||
**Both averages are reported on purpose.** They differ by a factor of two here,
|
||
because the largest sessions dominate the pooled figure — quoting only one would
|
||
mislead in whichever direction it happened to favour. Which to use depends on
|
||
the question: pooled is what maps to money, typical is what a new conversation
|
||
should expect.
|
||
|
||
Sessions that used recall form the "after" group; everything else is the
|
||
baseline. While no session has used it, the report says so rather than printing
|
||
a comparison of nothing against nothing.
|
||
|
||
## How it works
|
||
|
||
**Chunking is structure-aware.** Splitting on a fixed line count produces units
|
||
that begin mid-function; their embedding sits between two topics and matches
|
||
neither. Chunks are cut at column-0 declarations instead, neighbouring small
|
||
declarations are merged, and a chunk that absorbs several of them is labelled with
|
||
all of their names — because for a small file that label is the module's only
|
||
description in the index. Minified and generated files are detected and skipped
|
||
rather than indexed as noise.
|
||
|
||
**Ranking is hybrid, fused by rank.** Pure vector search is weak on code: an
|
||
embedding of `getUserById` sits near every other lookup in the repo. Lexical
|
||
scoring is weak in the opposite direction. The two are combined by reciprocal
|
||
rank fusion rather than by adding normalised scores — min-max normalising a
|
||
cluster of unrelated similarities stretches noise into a confident-looking
|
||
ranking, and rank fusion cannot make that mistake.
|
||
|
||
**The index is incremental and lives outside your repository.** Files are keyed by
|
||
content hash; only changed files are re-embedded. The SQLite index sits in
|
||
`$DSH_HOME/dsh-recall/workspaces/<key>/` — never in the repo, because for a
|
||
300k-line project it is tens of megabytes. Vectors are stored unit-length, so
|
||
scoring is a dot product.
|
||
|
||
**Cold start is bounded, and honest.** A search reads and hashes everything
|
||
(fast), then embeds only what fits its time budget, and reports exactly what is
|
||
still pending. `apply` never blocks on embedding a repository. In the measurement
|
||
above, the first search spent its 90s budget on 309 of 684 files (2,257 chunks);
|
||
a full `index` finished the remainder in about 75s more.
|
||
|
||
**Changing the model or the vector width discards the index** rather than silently
|
||
mixing two incompatible vector spaces into one ranking.
|
||
|
||
## Honest limitations
|
||
|
||
- **The 6.0x figure is a comparison, not a guarantee.** It measures what the tool
|
||
returned against the whole files those results came from. An unusually careful
|
||
agent might have read only one of those files, so treat it as an upper bound on
|
||
the saving rather than a promise.
|
||
- **Token figures are `characters / 4`.** An approximation, not a measurement.
|
||
- **Excerpts are capped at 40 lines by default**, so most results in the
|
||
measurement above were truncated. The excerpt identifies the location; a
|
||
*targeted* follow-up `read` of that range is still sometimes the right move. The
|
||
saving is in not reading the other 90% of each file.
|
||
- **A locally built index costs minutes of CPU** on a large repository. It costs
|
||
no API money, and only once.
|
||
- **Retrieval quality is the model's quality.** A different embedding model will
|
||
rank differently. `scripts/smoke.mjs` is the check that the top hit is actually
|
||
correct.
|
||
- **A file edited since it was embedded is flagged `[stale]`**, not silently
|
||
trusted.
|
||
|
||
## Operational notes
|
||
|
||
- The **only outbound request** this plugin makes is to the embedding endpoint you
|
||
configure. No source code goes anywhere else, and there is no telemetry.
|
||
- The endpoint receives your source code. Point it at infrastructure you trust —
|
||
that is the whole reason it is explicit configuration rather than a default.
|
||
- The workspace walk **never follows a symlink**, so a link out of the repository
|
||
cannot cause the machine to be indexed.
|
||
- The index is a cache. Deleting `$DSH_HOME/dsh-recall/` costs one re-embed and
|
||
nothing else.
|
||
|
||
### Give the endpoint room
|
||
|
||
A cold index is a sustained CPU load — hundreds of batched requests. The defaults
|
||
(`embeddingBatchSize: 16`, `embeddingConcurrency: 2`) are deliberately
|
||
conservative, and they were chosen the hard way.
|
||
|
||
**What happened during development.** An index run at concurrency 4 against a
|
||
small Ollama host that was *also* holding a generation model resident
|
||
(`expires_at` set to the year 2319 — pinned forever) left the host unable to load
|
||
the embedding model. It did not error: it accepted connections and never
|
||
answered. `ollama ps` afterwards showed only the pinned model. That needs a
|
||
restart of the Ollama service on that host; nothing this plugin can do reaches it.
|
||
|
||
Three things now guard against it:
|
||
|
||
- **Lower concurrency and batch size by default.** If you raise them, raise them
|
||
on hardware you know.
|
||
- **`embeddingKeepAlive`** (default `5m`) keeps the embedding model resident for
|
||
the length of a run instead of letting it be evicted and reloaded between
|
||
requests.
|
||
- **A short timeout, and a probe.** A wedged endpoint does not refuse a
|
||
connection, so a long timeout turns a broken backend into a tool call that
|
||
appears to hang. `recall status` issues a single no-retry probe with a 10-second
|
||
ceiling and reports whether the endpoint **answered** — which is a different
|
||
claim from "is configured", and the one worth having.
|
||
|
||
If `status` reports `endpoint did NOT answer`, check `ollama ps` on that host
|
||
before assuming the plugin is at fault.
|
||
|
||
## Development
|
||
|
||
```sh
|
||
npm test # 91 tests, no network required
|
||
node scripts/smoke.mjs [workspace] [query] # live retrieval check
|
||
node scripts/measure.mjs [workspace] [query] [budgetMs]
|
||
```
|
||
|
||
The `lib/` tests run anywhere. `test/config.test.js` and `test/plugin.test.js`
|
||
import `index.js`, which imports `@deepseek-ai/dsh-tools` and
|
||
`@deepseek-ai/schemastery` — resolvable once the plugin is installed into a
|
||
profile (or via a local `node_modules` link to the profile's copies, which the
|
||
`.gitignore` excludes).
|
||
|
||
`test/plugin.test.js` validates every tool result against **the tool's own
|
||
declared output schema**, which is `additionalProperties: false`. That is not
|
||
pedantry: the host enforces the schema before the model sees a result, so an
|
||
internal field leaking into the returned object is a failed call. It caught
|
||
exactly that during development.
|
||
|
||
`test/config.test.js` covers the load-time contract. Cordis rejects a `Config`
|
||
before `apply` ever runs, so a broken schema cannot be reported from inside the
|
||
plugin — the profile simply refuses to boot. Every other test in the suite calls
|
||
`apply` directly and would not notice. It also pins the second half of that
|
||
contract: that every field is `volatile()`, because that marker is what makes
|
||
the Harness serve a settings page at all, and its absence is invisible from
|
||
inside the process.
|
||
|
||
`test/client.test.js` runs the browser half in Node by stubbing `window`,
|
||
`require('react')`, and the settings-form primitives. That is worth more than it
|
||
sounds: the failure modes most likely to reach you silently are structural — a
|
||
page registered into a slot nobody declares (which is what the 0.1.5 seat,
|
||
`settings.plugin.item`, became), a right-sidebar body with no tab type behind it,
|
||
and a field list that has drifted from the schema — and all three are invisible
|
||
until someone opens Plugins. It also pins that an untouched API key is never
|
||
written, and that the field list and `Config` name the same settings.
|