dsh-slopo/README.md

# dsh-slopo

[slopo](https://github.com/rafal-qa/slopo) — an embedding-based duplicate-code
detector — as a DeepSeek Harness bundle.

Slopo finds non-obvious duplicated logic: similar implementations of the same
problem that are not copy-paste, and that a human reviewer, a coding agent and a
clone detector all miss. This bundle gives you three user-triggered skills and
one configuration-aware tool, and a settings card. Run it against whichever workspace you
are in without remembering flags.

## What it adds

| Contribution | What it is |
| --- | --- |
| `/slopo-review` | Slopo reports similar code involving your Git changes, and the agent reviews them as a note-taker. |
| `/slopo-analyze-ignore` | Triages the whole codebase's clusters, writing the obvious false positives into the ignore file. |
| `/slopo-analyze-one` | Reports one cluster at a time — optionally by hash — and judges whether it is worth acting on. |
| the `slopo` tool | Resolves the CLI, materialises the config, runs the work, and turns slopo's terse failures into something actionable. |
| the `dsh-slopo` configuration | The embedding backend and per-workspace state. It is the Host plugin's own `Config`, so the deployment's schema and the form can never disagree. |
| the settings card | A form for all of it, on the **Plugins** page under the `dsh-slopo` bundle. |

All three skills are **user-invocable and deliberately not model-invocable**,
matching slopo's own agent configs. Typing `/slopo-review` anywhere in a message
injects that skill; the model will not reach for one on its own. Every one of
these commands can spend minutes embedding, so the human decides when.

## Install

```sh
dsh plugin --profile web add git+https://git.notexpectedyet.com/Not-Expected-Yet/dsh-slopo.git
```

Then restart the profile. A new bundle's modules are resolved at startup, and a
client half's bytes are snapshotted at activation — so the settings card only
appears after that restart.

Plain ESM, no build step, and therefore no `prepare` script and no `allowBuilds`
entry to authorise. That is deliberate: `dsh plugin add git+…` fetches the source
tree rather than build output, so a `prepare` script would make every user
allowlist a build before the first install can succeed.

## Settings

The plugin ships a configuration page on the **Plugins** page — the sidebar
entry that lists bundles, rows and official plugins — reached by opening the
`dsh-slopo` bundle and pressing **Configure**. Its key is the bundle's package
name, which is also the namespace: the Loader entry id the bundle's patch
declares.

That the namespace is the entry id is the point. Since dsh 0.1.7-rc.2 the
framework projects this plugin's exported `Config` schema onto the entry that
composed it, so there is one source of truth for what is configurable. Every
field is marked `volatile()`, which is what puts it on the page, and
`embeddingApiKey` additionally carries `role('secret')`, so the browser is told
whether a key is stored and never receives the key itself.

Every setting in the table below is editable there; overridden fields are
marked, a draft that is not a valid value blocks the save, and Reset returns a
field to the deployment default rather than to empty. A save is one
revision-fenced write, so a second editor cannot be silently overwritten.

The card is the bundle's second half — the browser half. It needs no build step
and ships as plain ESM in the module-loader wrapper, but the platform snapshots
client bundles at activation, so **a client half only appears after the profile
restarts.** Without it the namespace still works — it is editable in the
profile's patch layer and through the tool's `configure` action — there is
simply no form drawn for it, because the page only renders a bundle that
registers a configuration entry for itself.

## Configure it

Nothing is defaulted. Slopo itself requires a source directory, an embedding
model and its dimensions, and guessing an embedding backend would mean guessing
where money is spent — so the plugin starts unconfigured and says so.

Ask the tool what it wants:

```jsonc
{ "action": "status" }
```

Then set it up:

```jsonc
{
  "action": "configure",
  "config": {
    "embeddingModel": "ollama/unclemusclez/jina-embeddings-v2-base-code:latest",
    "embeddingDimensions": 768,
    "embeddingApiBase": "http://127.0.0.1:11434"
  }
}
```

Values land in the `config` of the `dsh-slopo` row in the profile's patch layer
(and, from the browser, in the same row through the settings domain — the card
and the tool write the same place).

| Setting | Meaning |
| --- | --- |
| `embeddingModel` | LiteLLM model string. **Include the provider prefix** — see below. |
| `embeddingDimensions` | Must match what the model actually returns. Slopo freezes it at first index. |
| `embeddingApiBase` | Provider base URL. Becomes slopo's `embedding_params.api_base`. |
| `embeddingApiKey` | Optional. Passed to slopo in its environment, **never written to disk**. |
| `embeddingParamsJson` | Extra LiteLLM embedding params as a JSON object string. |
| `sourceDir` | Code to index. Defaults to the session's working directory. |
| `sourceDirExclude` | gitignore-style paths to skip. |
| `autoInstall` | Install slopo into `$DSH_HOME` with `uv` on first use. Default `true`. |
| `binary` | An explicit slopo executable, overriding discovery. |
| `stateDir` | Override the per-workspace state directory. |
| `ignoreFile` | Override the ignore file path. |
| `embeddingRequestDelay` | Milliseconds between embedding requests, for rate-limited providers. |
| `timeoutMs` | Cooperative budget for a foreground run. |

### How to find `embeddingDimensions`

It is the width of the vector the model returns, and slopo **freezes it into the
database on the first index** — get it wrong and every later run refuses with
`Configuration mismatch` until you delete the index and re-embed. So ask the
endpoint rather than guessing from a model name.

**The tool will tell you.** `status` probes the endpoint whenever a model and a
base URL are set — even when dimensions is not — and reports the width it
observed:

```jsonc
{ "action": "status" }
// → Missing: embedding_dimensions.
//   The endpoint is reachable and returned 768 dimensions, so set
//   embeddingDimensions to 768.
```

**Or ask it directly**, which is the same request:

```sh
# The width of the vector that comes back.
curl -s http://127.0.0.1:11434/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{"model":"unclemusclez/jina-embeddings-v2-base-code:latest","input":["probe"]}' |
  python3 -c 'import json,sys; print(len(json.load(sys.stdin)["data"][0]["embedding"]))'

# On Ollama, the model's own recorded width, with no inference at all.
curl -s http://127.0.0.1:11434/api/show \
  -d '{"model":"unclemusclez/jina-embeddings-v2-base-code:latest"}' |
  python3 -c 'import json,sys; print(json.load(sys.stdin)["model_info"]["jina-bert-v2.embedding_length"])'
```

Both must agree. If they do not, the endpoint is truncating or padding — for
example a server honouring a `dimensions` parameter — and it is the **returned**
width that slopo needs, because that is what lands in the database.

### Choose the provider prefix carefully

The provider prefix is not cosmetic — it selects which LiteLLM validator runs.

```
openai/<model>   ✗  UnsupportedParamsError: Setting dimensions is not supported
                    for OpenAI `text-embedding-3` and later models
ollama/<model>   ✓  works
```

Slopo always passes `dimensions`, and the `openai/` prefix makes LiteLLM apply
OpenAI-specific parameter validation to *any* model behind it. A local
OpenAI-compatible server therefore needs its own prefix (`ollama/`,
`hosted_vllm/`), not `openai/`. This is the single most likely thing to trip you
up on first setup, and slopo reports it only as `Error: Embedding failed.` on
stdout — the real message is in the agent log, which the tool reads for you.

### A local backend costs nothing

A local Ollama server is a good default: free, private, no API key and no
egress. Measured throughput on a small CPU box is roughly **230 ms per unique
unit** (~4/second), with repeated inputs served from cache. Budget minutes, not
seconds, for a first index of a real repository.

## Use it

Type the skill name in a message:

```
/slopo-review
/slopo-review main
/slopo-analyze-one
```

Or drive the tool directly:

| Action | What it does |
| --- | --- |
| `status` | Everything: resolved settings, index state, the binary and its version, plus a live pre-flight against the embedding endpoint. |
| `configure` | Patch the settings namespace. |
| `prepare` | `slopo index` then `slopo embed` — the slow, possibly costly phase. |
| `review` | Similar code involving Git changes (`base` defaults to `HEAD`). |
| `analyze` | Triage the whole codebase. |
| `analyze-one` | One cluster, optionally by `cluster` hash. |
| `update` | Upgrade the slopo CLI. |

`prepare` — the warm-up — starts a **background job** by default, because embedding
takes minutes and slopo writes its progress to a log file rather than to stdout,
so there is nothing to stream; the job handle carries the log path instead. The
other actions return their report directly, and any of them can be backgrounded
with `run_in_background: true` — the same parameter name `dsh-tool-bash` uses.

That asymmetry is deliberate. A completion notice is best-effort: a **killed** job
deliberately produces none, and an agent whose turn has already ended may have the
notice injected rather than be woken for it. So the action that returns a report
the caller can use immediately is worth more than a job id it has to remember to
collect — and only the genuinely long step depends on being watched.

## Where things live

Everything is under `$DSH_HOME`, and nothing is written into your repository:

```
$DSH_HOME/dsh-slopo/
├── uv-tools/                    # the uv-managed slopo install and its shim
├── uv-python/                   # the CPython 3.12 that slopo requires
└── workspaces/<name>-<hash8>/
    ├── slopo.conf.yaml          # generated; edits here are overwritten
    ├── slopo.db                 # the index and embeddings
    ├── slopo.ignore.txt         # dismissed cluster hashes
    └── slopo.agent.log          # full errors, progress
```

`$DSH_HOME` is not an arbitrary choice. In a containerised harness it is usually
the only persistent mount, while `~/.local` — where `uv tool install` puts things
by default — lives on the container's writable layer and is lost when the
container is *recreated*, not merely restarted. Setting `UV_TOOL_DIR`,
`UV_TOOL_BIN_DIR` and `UV_PYTHON_INSTALL_DIR` into `$DSH_HOME` makes the install
survive that, and installation is lazy rather than on boot so a fresh container
self-heals on first use.

Point `ignoreFile` at e.g. `<repo>/slopo.ignore.txt` if you would rather those
decisions were versioned with the code — they are a judgement about *your*
codebase, and arguably belong in review.

## What gets indexed

Slopo does **not** read `.gitignore` — it walks the tree you point it at. Pointed
at a repository root with no exclusions, it will happily parse every file under
`node_modules`. That is not a review of your code; it is a review of your
dependencies, and it is slow enough to matter.

Measured on one real repository:

| scope | files | code units | embedding at the reference rate |
| --- | --- | --- | --- |
| nothing excluded | ~1,300 | **113,768** | **~7.9 hours** |
| default excludes | 562 | 8,607 | ~36 minutes |
| default excludes + a server-only scope | **165** | **1,519** | **~6 minutes** |

So there are two independent safeguards.

**Default excludes**, on unless you turn them off: `node_modules`, `.git`,
`dist`, `build`, `out`, `coverage`, `.venv`, `venv`, `__pycache__`, `target`,
`vendor`, and minified bundles. Your own `sourceDirExclude` is **added** to
these rather than replacing them — deliberately, because a settings patch
replaces a whole value, so setting one pattern would otherwise silently
re-enable the 13x blow-up.

**A budget on the backlog.** `maxIndexUnits` (default 5,000) refuses to start an
embed that exceeds it, naming the count, the estimate and the lever:

> 113,768 code units still need embedding across 1,300 files. At the reference
> rate that is about 7.9 hours, against a budget of 5,000.

Nothing has been embedded at that point. Raise the budget, or narrow the scope —
setting `maxIndexUnits` to 0 removes the limit. The count comes from reading
slopo's own database, not from parsing its output, because `Indexed N code units`
reports only what was *newly* indexed: on a warm database it reads zero while the
backlog is still full, which is exactly the state a refusal leaves behind.

Every action indexes first so the gate runs before anything expensive, and
`status` reports both the effective excludes and the budget.

## Things worth knowing

**The harness seam moved under this plugin once, and the failure was silent.**
dsh 0.1.7-rc.2 replaced three things this bundle used:

| was | is |
| --- | --- |
| `ctx.settings.installSection(ctx, ns, Config, config, …)` | the exported `Config` itself, projected onto the Loader entry; `ctx.settings.configure({ auto: false })` claims the entry's own page |
| a plugin-chosen settings namespace + a `settings.plugin.item` card | the entry id, plus a `plugins.bundle.config` card keyed by the bundle's package name |
| `ctx.shell.run(spec)` | `await ctx.shell.execute(ctx.shell.resolve(request)).result()` |

All three broke quietly rather than loudly. `installSection` was missing, so the
call threw into a `try/catch` that logged a warning nobody reads; the card's slot
and the `settingsScope` service it asked for no longer existed, so the card
simply never rendered; and `shell.run` threw only on the paths that actually run
a command. On top of that, a `Config` field that is not marked `volatile()` is
invisible to every settings UI, and the `jobs` registry wants `exec.agent.id`
(a session id) as its owner — handing it the agent object registered a job no
session could collect.

Two habits follow. First, this bundle's tests bind to the *installed* harness's
own slot catalog and package exports, so a rename fails a test instead of
quietly emptying a page. Second, the browser half keeps its slot and service
names in exactly one place each, so the next rename is a one-line edit.

**Five settings are frozen at first index.** `source_dir`, `embedding_model`,
`embedding_dimensions` and both node-count thresholds are written into
`slopo.db` on the first `index`, and every later run refuses to mix generations:

```
Error: Configuration mismatch.
```

Changing any of them means deleting the database and re-embedding everything —
real time, and real money on a hosted backend. The tool recognises this failure
and says so rather than relaying the two-word stderr line.

**A review cannot bootstrap the database.** `agent-review` opens the index with a
read-only open, so on a fresh workspace it fails with `No data in database. The
tool wasn't initialized.` Run `prepare` first. Indexing is pure local parsing and
free; only embedding reaches the network.

**Slopo refuses more than 1000 unembedded units** inside an agent call, to
protect agent latency. The tool translates that into an offer to run `prepare`
in the background.

**Slopo's stderr is deliberately unhelpful.** It prints one short non-revealing
line for an agent and writes the real error to the agent log. The tool reads the
log tail and maps the known failures — missing database, configuration mismatch,
oversized backlog, missing or invalid config, stale index, git failure,
embedding failure — onto a message and a next step.

**`--config-version 1` is an alarm, not boilerplate.** It makes slopo fail if its
agent-facing contract has moved on, which is the signal that the skill text in
`skills/` needs re-porting from upstream.
**The sandbox can block slopo's state.** Slopo's index, config and log live
under `$DSH_HOME`, which is outside the workspace that a `workspace-write`
sandbox permits writing to. The tool detects that and refuses *before* running,
naming both ways out, rather than letting a write fail halfway through an embed:
run the session under the `danger-full-access` preset, or set `stateDir` to a
directory inside the workspace.

**What the tool does with the sandbox policy, and why it matters.** The mounted
`ctx.shell` on a confined deployment is `dsh-bash-sandbox`, whose `resolve()`
stamps `request.sandboxPolicy ?? ctx.sandboxPolicy.resolve()`. Omitting the
policy therefore does *not* mean "unconfined" — it silently substitutes the
**deployment** default (`DSH_PERMISSION_MODE ?? 'workspace-write'`), which is not
the session's preset. That is what makes a `danger-full-access` session fail with
`SandboxUnavailableError` on a host with no sandbox runner. The tool resolves the
session's policy the way `dsh-tool-bash` does and passes it per call.

## Slopo is an external process, not a library

It is installed separately, invoked through its own CLI, and never imported or
modified. Slopo is AGPL-3.0-or-later and does not accept patches, so its
command-line interface is the only contract worth building on — and executing an
unmodified, separately installed program is a materially different proposition
from linking against it.

## Development

```sh
node --test test/*.test.js          # unit + integration, mock harness
DSH_SLOPO_E2E=1 npm test            # also drives the real CLI end to end
```

The integration tests load the real plugin module against a mock Cordis context
and drive the tool through a real shell into the real slopo binary, so a wrong
assumption about a dsh service surfaces as a failing test rather than as a plugin
that silently does nothing. Two real bugs were caught this way: a config
renderer that would write the API key to disk if handed one, and `--config-version`
placed before the subcommand, which slopo rejects with a usage error.

The mock's seams are shaped like the harness's, deliberately, because that is
what makes this file a contract test. `ctx.shell` exposes `resolve`/`execute`
with a handle whose `result()` settles the run — there is no `run(spec)`, which
is exactly the method dsh 0.1.7-rc.2 removed. `ctx.settings` has no
`installSection` either; it has `configure` and `update`, and the tests assert
that the plugin claims its own page and writes to the entry id the bundle patch
declares. The `jobs` fake records the spec, so the test can check that a
background job is owned by a **session id** and that its `done` resolves a
`JobOutcome` carrying `result`.

The browser half has no browser to run in, so its tests evaluate the bundle the
way the real module loader does — inside `window.__ModuleLoader__.load` — with a
fake `require` that answers `react` and `@deepseek-ai/dsh-client-ui-primitives`
and throws on anything else. That last part is deliberate: it checks the card
stays inside the platform's seed table, which is the constraint a hand-written
client half is most likely to break. The card is then rendered and its handlers
driven — including the field specs, run through a copy of the shipped
`SettingsFormModel` staging semantics — which is not a substitute for looking at
it but does cover the wiring it cannot work without. Three further tests read the
*installed* harness instead: that `plugins.bundle.config` is a declared slot and
`settings.plugin.item` is gone, that the slot is dispatched by bundle package
name, and that every primitive the card requires is still exported. They skip
when no global dsh installation is present.

`node_modules/@deepseek-ai` is a local symlink to the installed harness so the
`@deepseek-ai/*` imports resolve during tests. It is gitignored and is not a
declared dependency — `peerDependencies` states the compatibility contract
instead, and an out-of-tree bundle that depended on an in-box package would
install a second copy that drifts from the host's. The declared peers are what
the harness's compatibility gate reads: a plugin whose `@deepseek-ai/dsh-*`
ranges do not admit the running version is refused at startup rather than
half-loaded.

## Releases and updating

### The workflow

`.gitea/workflows/release.yml` runs on every push to `master`:

1. works out the range since the last tag;
2. decides whether the push releases at all;
3. bumps `package.json`, prepends a `CHANGELOG.md` section, commits, and tags
   `vX.Y.Z`;
4. creates the Gitea release with those notes as the body.

The bump is read from the commit subjects in the release, so the intent travels
with the change instead of living in a file someone has to remember to update:

| in a commit subject | effect |
|---|---|
| `feat`, as its own word (`feat:`, `feat(x):`) | minor — `0.1.0` → `0.2.0` |
| `feat!:` / `fix(x)!:` / `BREAKING` | major — `0.1.0` → `1.0.0` |
| `fix:`, `docs:`, prose, anything else | patch — `0.1.0` → `0.1.1` |
| `[minor]` / `[major]` | force the bump when the subject cannot carry it |
| `[skip release]` | this push publishes nothing |

Only the **subject** line is read, and `feat` has to be a word — `Fix the feature
flag leak` is a patch, not a minor.

With no tags yet, the version already in `package.json` is what gets tagged — the
first release should match what the code says, not invent a number.

Release notes come from the commit subjects in the range. Conventional prefixes
(`feat:`, `fix:`, `docs:`) are grouped under headings when at least half the
commits carry one; otherwise the notes are a flat list. Prose commit subjects get
no invented headings.

The logic lives in `.gitea/scripts/release.sh` so it runs without a runner, and
`npm run test:release` exercises it — an empty range, a repository with no tags
yet, a patch number already at 9, a `feat` that is only a word. `npm run
release:plan` says whether the current HEAD would release, and `npm run
release:notes` prints the notes it would publish, both without releasing.

The runner needs nothing but Gitea's own job token (`secrets.GITEA_TOKEN`), which
`permissions: contents: write` in the workflow is what unlocks. Two caveats worth
knowing before the first push: Actions has to be enabled on the repository, and a
runner with the `ubuntu-latest` label has to be registered against the instance —
otherwise the run sits queued forever.

### Making `dsh plugin update` work

A plugin installed as a bare `git+https://…/repo.git` is resolved to a **commit**
and pinned in the profile's lockfile. Verified against a local remote: `pnpm
update <name>` *does* move that to the branch head, so the mechanism is not
broken — but it tracks every commit, including half-finished ones, and it needs
fetch access to the remote.

Adding a version selector makes it track **releases** instead:

```jsonc
// $DSH_HOME/profiles/<profile>/package.json
"dsh-slopo": "git+https://git.notexpectedyet.com/Not-Expected-Yet/dsh-slopo.git#semver:^0.1.0"
```

`#semver:^0.1.0` resolves the highest tag in range. Verified locally: installed
`v0.1.1` where the branch was already at `v0.1.2`-plus-untagged-work, and
`pnpm update` then moved it to a newly pushed `v0.1.2`.

Two things to know, both of which look like "the update did nothing":

- **The remote must be fetchable.** `pnpm update` cannot see a new tag it cannot
  fetch. If `git ls-remote <url>` asks for a username, fix that first — the same
  credential push needs.
- **A restart is required.** Client bundle bytes are snapshotted when the profile
  activates, so a hot update installs new files but keeps serving the old ones
  until `dsh --profile <profile>` comes back up.

## Licence

AGPL-3.0-or-later, matching slopo. The three skill bodies are ports of slopo's
own agent configurations rather than new prose; `NOTICE.md` records exactly what
changed and why the licence follows.