# Knowledge and RAG Knowledge gives the model one tool, `knowledge_search`, over a locally built index of the operator's own markdown and text files, so it can ground an answer in project documentation instead of its training data. The whole retrieval system, index and search, lives in the one `fisk-ai` binary and one process. It is opt-in and off by default, agent-mode only, and served over MCP only through an explicit allowlist, the same posture as [Memory]({{% relref "memory" %}}). {{% notice style="note" title="Where it lives" %}} `internal/rag`: the SQLite store in `store.go`, chunking in `chunk.go`, indexing in `index.go`, hybrid search in `search.go`, the embeddings client in `embed.go`, health checks in `doctor.go`, and the cross-process write lock in `lock_unix.go` and `lock_windows.go`. The model-facing `knowledge_search` tool is in `internal/util/builtin_rag.go`; the `fisk-ai knowledge` CLI is in `rag_command.go`; the `harness.knowledge` schema is in `config/config.go`. {{% /notice %}} ## Two tiers, one search A search always runs the lexical tier and, when the vector tier is on, runs it too and fuses the two. The lexical tier is an FTS5/BM25 full-text index; it is always active when knowledge is enabled and needs no model, no server, and no per-query cost. The vector tier is a separate opt-in: adding an `embeddings` block turns it on, each chunk is embedded through a local OpenAI-compatible server, and a query fuses the lexical and vector rankings with Reciprocal Rank Fusion. Lexical matches on shared words; vector matches on meaning, so a query finds the right section even when it shares no keywords with it. `Search` runs FTS5 first (`ftsSearch`, `search.go:104`), then, if an embedder is configured, embeds and normalizes the query and runs a KNN over the `sqlite-vec` table (`vecSearch`, `search.go:123`). Each retriever over-fetches `searchFanout` = 50 candidates (`search.go:25`), `rrf` fuses the two ranked lists with the constant `rrfK` = 60 (`search.go:20`, `search.go:286`), and the top_k survivors are hydrated into cited hits. RRF fuses by rank, not score, so the incomparable scales of BM25 and L2 distance never need normalizing against each other.
query text terms FTS5 / BM25 lexical, always on vector KNN opt-in, sqlite-vec RRF fuse by rank, K=60 top_k hits each cited embeddings unreachable: lexical only
Every query runs FTS5. When the vector tier is on, KNN runs too and RRF fuses both by rank. If the embeddings server is unreachable the query degrades to lexical and says so.
## Building the index `knowledge index` is the single writer. It is incremental and per-file: a file whose content hash is unchanged is skipped, a changed file is re-chunked, and a walk of a full configured root reconciles deletions. Only `.md`, `.markdown`, `.txt`, and `.text` files are walked, and the store directory and any `memory/` directory are always excluded.
  1. Walk and hash Each eligible file's SHA-256 is compared to the stored document row; an unchanged hash skips the file, a new or changed one is queued (`ingestOne`, `index.go`).
  2. Chunk `ChunkDocument` splits on ATX headings and packs blocks to roughly 1200 bytes, keeping fenced code blocks whole and folding the heading breadcrumb into each chunk (`chunk.go`).
  3. Embed outside the transaction When the vector tier is on, `EmbedDocuments` runs and each vector is L2-normalized (`index.go:274`, `index.go:283`), before any write transaction opens.
  4. Upsert in one short transaction The document, its chunks, and their vectors are written in a single `BeginTx` block (`index.go:287`); FTS5 triggers keep the full-text table in sync, and a foreign-key cascade clears old chunks first.
The embed step sits deliberately outside the transaction so a slow or hung embeddings call never holds the single writer slot open. The writer is serialized across processes by an advisory `flock` on Unix (`lock_unix.go`) and an `O_CREATE|O_EXCL` lock file on Windows (`lock_windows.go`), since WAL mode lets multiple processes open the file and `MaxOpenConns(1)` alone cannot serialize them. ## The knowledge_search tool When knowledge is enabled the agent run path opens the store read-only and builds one built-in tool (`RAGTools`, `builtin_rag.go:38`), tracked in its own slice beside the memory tools (`agent.go:189`). A start-of-run system note tells the model the tool exists and to consult it before answering project-specific questions (`RAGSystemNote`, `builtin_rag.go:50`). The tool takes a `query` and an optional `top_k`; the effective count is `min(requested or configured, 20)`, and the total returned text is capped at `max_injected_tokens` by `capHits` at roughly four characters per token (`builtin_rag.go:177`). Each result carries a citation token of the form `relpath#ordinal` plus the section's heading path. The same token is printed by `knowledge search` and `knowledge sources` and accepted verbatim by `knowledge show`, so a hit resolves back to its full text. When no index exists yet the tool returns a soft `index_not_built` status naming the fix rather than failing the run, so a missing index never bricks agent startup. Over MCP the tool is served only when it is allowlisted in `expose.agent.mcp.builtins` (`MCPExposesKnowledgeSearch`, `config.go:726`). `knowledge_search` is the one exposable built-in because it is read-only and needs no operator prompt; the memory and human-in-the-loop tools stay agent-only. The MCP path opens the same read-only store and closes it after the server stops (`mcp_command.go:109`). {{% notice style="warning" title="Load-bearing decision" %}} Transient and permanent failures are handled asymmetrically. An embeddings server that is unreachable at query time degrades to lexical and sets `Degraded` with a reason (`search.go:120`), so a search never fails just because the model is offline. A dimension, model, or prefix mismatch instead fails loud and demands a `--reindex`, because a silently wrong vector identity would return wrong rankings. An unreachable server at index time also fails loud, so an index asked to be semantic is never quietly built lexical-only. {{% /notice %}} {{% notice style="warning" title="Caveat" %}} The index holds the verbatim text of every indexed document, unencrypted on disk. The file and its `-wal` and `-shm` sidecars are created `0600` inside a `0700` directory, opened with `O_NOFOLLOW` on Unix, and retrieved chunks are framed as untrusted reference data, not instructions. This is the memory posture: do not index secrets, and bind localhost or add authentication before exposing `knowledge_search` over MCP. {{% /notice %}} {{% notice style="tip" title="Next" %}} Continue to [The Terminal UI]({{% relref "terminal-ui" %}}) for how a run is presented to an operator. {{% /notice %}}