feat: index extra source roots beside the workspace, for the code you write against

The workspace index cannot answer questions about an installed framework or
SDK, and in plugin development that is where most of the reading goes: 64 of
80 tool calls in one real session went to the harness checkout under
/usr/local/lib/node_modules to look up the API being implemented.

A root is entered whatever it is called, and its dependencies are not: a
path ending in node_modules/... is the normal way to name a package tree,
while a node_modules below the root is somebody else's code. A result
carries the label that says which tree it came from and an absolute
sourcePath, the only form a follow-up read can resolve.

The tool description now also claims the move this replaces - searching
before reading files to find out what a module contains.

lib/roots.js

@@ -0,0 +1,254 @@
+/**
+ * Extra index roots: code that is not in the workspace, indexed beside it.
+ *
+ * ## Why this exists
+ *
+ * A workspace index answers questions about the code you are writing. It does
+ * not answer questions about the code that code is written *against* — the
+ * library, the framework, the platform SDK. In plugin development that is not a
+ * side interest: measured on one real session, 64 of 80 tool calls went to the
+ * installed harness checkout under `/usr/local/lib/node_modules` to look up the
+ * contract being implemented, because that is the only authoritative copy of it.
+ * None of that is under the workspace, so none of it was indexed, and the agent
+ * read it into context the expensive way.
+ *
+ * An extra root is a second tree, indexed into the same index, whose results are
+ * distinguishable from the workspace's at a glance.
+ *
+ * ## The traversal rule is the whole feature
+ *
+ * An extra root is usually *a directory of packages* — the scoped-packages
+ * directory of a package manager, say — which is the one shape the workspace
+ * walk handles worst: it skips `node_modules` and friends at any depth, so the
+ * interesting files are exactly the ones it refuses to enter.
+ *
+ * So the rule is deliberately asymmetric, and it is worth stating precisely
+ * because the wrong version is silently wrong:
+ *
+ *   - **Below the root**, `node_modules`, build output and VCS directories are
+ *     skipped as usual. A dependency of a package is still somebody else's code,
+ *     and it is the same code the root already holds.
+ *   - **The root itself is entered whatever it is called.** A path that ends in
+ *     `node_modules/...` is the normal way to name a package-manager tree, and a
+ *     rule that skipped it would refuse the configuration this exists for.
+ *
+ * The cost of not applying the first half would be an index that embeds every
+ * dependency copy it can reach; the cost of not applying the second is that the
+ * feature cannot be configured at all. Only the root is exempt, never a nested
+ * directory that happens to share its name.
+ *
+ * ## Names, not just prefixes
+ *
+ * A result from an extra root is labelled `<name>/<path within the root>`. The
+ * name defaults to the root directory's basename — `@deepseek-ai` for a scope
+ * directory — and a deployment that wants something shorter says so with the
+ * `name=path` form, because `@deepseek-ai/dsh-agent/lib/index.js` is precise but
+ * long, and the label is on every result.
+ *
+ * ## Results from a root are not workspace-relative, and cannot pretend to be
+ *
+ * The paths a search returns are what the caller hands to `read`, which resolves
+ * against the workspace. A file under an extra root is not under the workspace,
+ * so no amount of prefixing makes `<name>/lib/index.js` readable. The honest
+ * answer is the absolute path, reported beside the relative label: the label
+ * says where the excerpt came from, the absolute path is what a follow-up read
+ * needs. Reading it may still need a scope grant from the caller's own policy —
+ * this module cannot grant it and does not pretend to.
+ */
+
+import { existsSync, statSync } from 'node:fs'
+import { basename, isAbsolute, resolve } from 'node:path'
+
+import { DEFAULT_SKIP_DIRS, listCodeFiles } from './walk.js'
+
+/**
+ * Directories skipped inside an extra root.
+ *
+ * The workspace default minus `node_modules`: a root is normally a
+ * package-manager tree whose *top* level is the thing being indexed, while a
+ * `node_modules` further down is a dependency of one of those packages and is
+ * skipped by the depth rule in {@link listRootFiles}.
+ *
+ * `node_modules` stays IN the set: it is skipped below a root like anything else,
+ * and it is {@link skipsRootDir} — not a shorter list — that lets the root itself
+ * be one. Dropping it here as well would index every dependency of every package
+ * in a package-manager tree, which is the cost the depth rule exists to avoid.
+ */
+export const ROOT_SKIP_DIRS = Object.freeze([...DEFAULT_SKIP_DIRS])
+
+/** Whether `path` exists and is a directory. */
+function statIsDirectory(path) {
+  try {
+    return statSync(path).isDirectory()
+  } catch {
+    return false
+  }
+}
+
+/**
+ * Read one `sourceRoots` entry.
+ *
+ * Accepted forms, both orderings of the same two facts:
+ *
+ * ```
+ * /abs/path/to/packages            # name defaults to the basename
+ * @dsh=/abs/path/to/packages       # explicit name
+ * ```
+ *
+ * A relative path is resolved against the workspace, which is what makes a
+ * checked-in configuration portable; an absolute one is taken as given, which is
+ * what a deployment's own installed tree needs.
+ *
+ * @param {string} entry one configured root
+ * @param {string} workspace absolute workspace root
+ * @returns {{name: string, path: string, workspaceRelative: boolean}}
+ */
+export function parseSourceRoot(entry, workspace) {
+  const text = String(entry ?? '').trim()
+  if (text === '') throw new Error('an empty entry')
+
+  // `name=path` only when a name is actually present: a Windows path or a
+  // relative path with an `=` in it must not be split by accident.
+  let name = ''
+  let target = text
+  const equals = text.indexOf('=')
+  if (equals > 0) {
+    name = text.slice(0, equals).trim()
+    target = text.slice(equals + 1).trim()
+  }
+  if (target === '') throw new Error(`"${text}" names no path`)
+  if (name.includes('/') || name.includes('\\')) {
+    throw new Error(`"${name}" is not a usable name: it contains a path separator`)
+  }
+
+  const workspaceRelative = !isAbsolute(target)
+  const absolute = workspaceRelative ? resolve(workspace, target) : resolve(target)
+  const label = sanitizeRootName(name === '' ? basename(absolute) : name)
+  if (label === '') throw new Error(`"${text}" does not yield a usable name`)
+  return { name: label, path: absolute, workspaceRelative }
+}
+
+/**
+ * Make a name safe to use as a path prefix.
+ *
+ * The name becomes the first segment of every result path, so a name carrying a
+ * separator would forge a deeper path than the root holds, and a name of `.` or
+ * `..` would forge one outside it.
+ */
+export function sanitizeRootName(name) {
+  const cleaned = String(name ?? '')
+    .trim()
+    .replace(/[^A-Za-z0-9._@-]+/g, '-')
+    .replace(/^[-.]+|[-.]+$/g, '')
+  return cleaned === '.' || cleaned === '..' ? '' : cleaned
+}
+
+/**
+ * Resolve every configured root, reporting each unusable one by name.
+ *
+ * A bad entry is skipped rather than fatal, and said out loud: a typo in one
+ * root must not cost the caller the other roots, and it must not look like the
+ * feature doing nothing. This mirrors how an unusable `focus` is reported.
+ *
+ * @param {readonly string[]} entries configured `sourceRoots`
+ * @param {object} [options]
+ * @param {string} options.workspace absolute workspace root
+ * @param {(path: string) => boolean} [options.isDirectory] injection point for tests
+ * @returns {{roots: {name: string, path: string}[], problems: string[]}}
+ */
+export function resolveSourceRoots(entries, options = {}) {
+  const { workspace, isDirectory = statIsDirectory } = options
+  const roots = []
+  const problems = []
+  const seen = new Set()
+
+  for (const entry of entries ?? []) {
+    let parsed
+    try {
+      parsed = parseSourceRoot(entry, workspace)
+    } catch (error) {
+      problems.push(String(error?.message ?? error))
+      continue
+    }
+    if (!isDirectory(parsed.path)) {
+      problems.push(`"${String(entry).trim()}" is not a directory: ${parsed.path}`)
+      continue
+    }
+    if (seen.has(parsed.name)) {
+      problems.push(`"${String(entry).trim()}" duplicates the name "${parsed.name}", which was already indexed`)
+      continue
+    }
+    seen.add(parsed.name)
+    roots.push({ name: parsed.name, path: parsed.path })
+  }
+
+  return { roots, problems }
+}
+
+/**
+ * Whether a directory is skipped during a *root* walk.
+ *
+ * Exported because it is the rule the feature turns on, and a test that asserts
+ * a nested `node_modules` is skipped is a test of this function, not of a walk.
+ *
+ * @param {number} depth level below the root; 0 is the root's own entries
+ * @param {string} name a directory entry's name
+ * @param {ReadonlySet<string>} [skip] directory names skipped below the root
+ */
+export function skipsRootDir(depth, name, skip = new Set(ROOT_SKIP_DIRS)) {
+  // The root is entered whatever it is called — that is the whole point of an
+  // explicit root. Everything below it obeys the normal skip list.
+  if (depth === 0) return false
+  return skip.has(name)
+}
+
+/**
+ * Label a file found under an extra root.
+ *
+ * @param {string} rootName the root's label
+ * @param {string} relPath path within the root, `/`-separated
+ */
+export function rootRelativePath(rootName, relPath) {
+  return relPath === '' ? rootName : `${rootName}/${relPath}`
+}
+
+/**
+ * List the indexable files under one extra root.
+ *
+ * Keyed by `ROOT_SKIP_DIRS` membership *below* the root and by nothing at all at
+ * it, because `listCodeFiles` reports the walk root at depth 0 and a directory it
+ * descends into at depth + 1. So the root's own name never decides anything,
+ * which is what lets a root be a `node_modules` tree, and a `node_modules`
+ * *inside* one of its packages is still skipped.
+ *
+ * `maxFiles` bounds each root rather than the whole index, and the result reports
+ * truncation, so a root too large to index says so instead of quietly answering
+ * from its first few files.
+ *
+ * @param {object} input
+ * @param {string} input.name label for the root, prefixed onto every path
+ * @param {string} input.path absolute directory to walk
+ * @param {readonly string[]} [input.extensions] allowed extensions, without dots
+ * @param {readonly string[]} [input.exclude] glob patterns over the root-relative path
+ * @param {number} [input.maxFiles] stop after this many candidates
+ * @param {AbortSignal} [input.signal]
+ * @returns {Promise<{files: {path: string, absolute: string}[], truncated: boolean, skippedFiles: number}>}
+ */
+export async function listRootFiles(input) {
+  const { name, path, extensions, exclude = [], maxFiles, signal } = input
+  const skip = new Set(ROOT_SKIP_DIRS)
+  const walk = await listCodeFiles(path, {
+    ...(extensions === undefined ? {} : { extensions }),
+    exclude,
+    ...(maxFiles === undefined ? {} : { maxFiles }),
+    isSkipped: (depth, dirName) => skipsRootDir(depth, dirName, skip),
+    signal,
+  })
+  return {
+    ...walk,
+    files: walk.files.map((file) => ({ path: rootRelativePath(name, file.path), absolute: file.absolute })),
+  }
+}
+
+/** One entry of the `sourceRoots` field, as the plugin's own docs describe it. */
+export const SOURCE_ROOT_EXAMPLE = '/usr/local/lib/node_modules/@deepseek-ai'

lib/walk.js

@@ -142,6 +142,11 @@ export function extensionOf(path) {
  * @param {readonly string[]} [options.extensions] allowed extensions, without dots
  * @param {readonly string[]} [options.skipDirs] directory names skipped at any depth
  * @param {readonly string[]} [options.exclude] glob patterns over the relative path
+ * @param {(depth: number, name: string, absolute: string) => boolean} [options.isSkipped]
+ *   decides directory traversal, replacing the `skipDirs` membership test. This is
+ *   the seam an extra index root uses: its top level is a package-manager
+ *   `node_modules` tree, which the built-in list exists to skip, so it must opt out
+ *   for the root and only the root. Depth 0 is the root's own entries.
  * @param {number} [options.maxFiles] stop after this many candidates
  * @param {number} [options.maxDepth] deepest directory level to enter
  * @param {AbortSignal} [options.signal]
@@ -152,6 +157,7 @@ export async function listCodeFiles(root, options = {}) {
     extensions = DEFAULT_EXTENSIONS,
     skipDirs = DEFAULT_SKIP_DIRS,
     exclude = [],
+    isSkipped,
     maxFiles = 20_000,
     maxDepth = 24,
     signal,
@@ -159,6 +165,12 @@ export async function listCodeFiles(root, options = {}) {
 
   const wanted = new Set(extensions.map((value) => String(value).toLowerCase()))
   const skip = new Set(skipDirs)
+  // Materialised once per walk rather than tested with `isSkipped ?? default` at
+  // every directory: the hot loop should not carry a branch that cannot change.
+  const shouldSkip =
+    isSkipped === undefined
+      ? (depth, name) => skip.has(name)
+      : (depth, name, absolute) => isSkipped(depth, name, absolute)
   const files = []
   let truncated = false
   let skippedFiles = 0
@@ -185,7 +197,7 @@ export async function listCodeFiles(root, options = {}) {
       const absolute = join(dir, entry.name)
       const relPath = relative(root, absolute).split(sep).join('/')
       if (entry.isDirectory()) {
-        if (skip.has(entry.name)) continue
+        if (shouldSkip(depth, entry.name, absolute)) continue
         if (depth >= maxDepth) continue
         if (matchesAny(relPath, exclude)) continue
         stack.push({ dir: absolute, depth: depth + 1 })