Knowledge and RAG

Knowledge gives the model one tool, knowledge_search, over a locally built index of the operator’s own markdown and text files, so it can ground an answer in project documentation instead of its training data. The whole retrieval system, index and search, lives in the one fisk-ai binary and one process. It is opt-in and off by default, agent-mode only, and served over MCP only through an explicit allowlist, the same posture as Memory.

Two tiers, one search

A search always runs the lexical tier and, when the vector tier is on, runs it too and fuses the two. The lexical tier is an FTS5/BM25 full-text index; it is always active when knowledge is enabled and needs no model, no server, and no per-query cost. The vector tier is a separate opt-in: adding an embeddings block turns it on, each chunk is embedded through a local OpenAI-compatible server, and a query fuses the lexical and vector rankings with Reciprocal Rank Fusion. Lexical matches on shared words; vector matches on meaning, so a query finds the right section even when it shares no keywords with it.

Search runs FTS5 first (ftsSearch, search.go:104), then, if an embedder is configured, embeds and normalizes the query and runs a KNN over the sqlite-vec table (vecSearch, search.go:123). Each retriever over-fetches searchFanout = 50 candidates (search.go:25), rrf fuses the two ranked lists with the constant rrfK = 60 (search.go:20, search.go:286), and the top_k survivors are hydrated into cited hits. RRF fuses by rank, not score, so the incomparable scales of BM25 and L2 distance never need normalizing against each other.

Building the index

knowledge index is the single writer. It is incremental and per-file: a file whose content hash is unchanged is skipped, a changed file is re-chunked, and a walk of a full configured root reconciles deletions. Only .md, .markdown, .txt, and .text files are walked, and the store directory and any memory/ directory are always excluded.

1. Walk and hash Each eligible file's SHA-256 is compared to the stored document row; an unchanged hash skips the file, a new or changed one is queued (`ingestOne`, `index.go`).
2. Chunk `ChunkDocument` splits on ATX headings and packs blocks to roughly 1200 bytes, keeping fenced code blocks whole and folding the heading breadcrumb into each chunk (`chunk.go`).
3. Embed outside the transaction When the vector tier is on, `EmbedDocuments` runs and each vector is L2-normalized (`index.go:274`, `index.go:283`), before any write transaction opens.
4. Upsert in one short transaction The document, its chunks, and their vectors are written in a single `BeginTx` block (`index.go:287`); FTS5 triggers keep the full-text table in sync, and a foreign-key cascade clears old chunks first.

The embed step sits deliberately outside the transaction so a slow or hung embeddings call never holds the single writer slot open. The writer is serialized across processes by an advisory flock on Unix (lock_unix.go) and an O_CREATE|O_EXCL lock file on Windows (lock_windows.go), since WAL mode lets multiple processes open the file and MaxOpenConns(1) alone cannot serialize them.

The knowledge_search tool

When knowledge is enabled the agent run path opens the store read-only and builds one built-in tool (RAGTools, builtin_rag.go:38), tracked in its own slice beside the memory tools (agent.go:189). A start-of-run system note tells the model the tool exists and to consult it before answering project-specific questions (RAGSystemNote, builtin_rag.go:50). The tool takes a query and an optional top_k; the effective count is min(requested or configured, 20), and the total returned text is capped at max_injected_tokens by capHits at roughly four characters per token (builtin_rag.go:177).

Each result carries a citation token of the form relpath#ordinal plus the section’s heading path. The same token is printed by knowledge search and knowledge sources and accepted verbatim by knowledge show, so a hit resolves back to its full text. When no index exists yet the tool returns a soft index_not_built status naming the fix rather than failing the run, so a missing index never bricks agent startup.

Over MCP the tool is served only when it is allowlisted in expose.agent.mcp.builtins (MCPExposesKnowledgeSearch, config.go:726). knowledge_search is the one exposable built-in because it is read-only and needs no operator prompt; the memory and human-in-the-loop tools stay agent-only. The MCP path opens the same read-only store and closes it after the server stops (mcp_command.go:109).