Telemetry

Fisk AI exports OpenTelemetry traces and metrics over OTLP/HTTP, following the GenAI semantic conventions. One run is one trace: how long it took, which model calls it made, which tools it ran, and where the tokens went. It applies to fisk run, to the runs fisk serve hosts, and to knowledge searches served by fisk mcp. The a2a endpoint, which serves tools to other agents without running the loop, exports a span per served call and joins the caller’s trace.

A single session with two prompts:

Turning it on

Add a telemetry block and point it at a collector:

telemetry:
  enabled: true
  endpoint: http://127.0.0.1:4318

This build speaks OTLP/HTTP, port 4318. Pointing it at 4317 is the OTLP/gRPC port and is rejected at startup.

Every setting is in the reference. Transport credentials are never in the file: the standard OTEL_EXPORTER_OTLP_HEADERS and friends configure the connection, so the same configuration sends to a collector, Grafana Tempo, Honeycomb or any OTLP/HTTP endpoint.

--no-telemetry suppresses export: for one run on fisk run, and for the whole process on fisk serve, which reports whether telemetry is on in its startup banner.

A local collector

Save as otelcol.yaml and run otelcol-contrib --config otelcol.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      exporters: [debug]

Then run the agent. The run’s summary line ends with the trace id, and the full-screen UI shows it on the end card:

Run summary: model=claude-sonnet-5 llm_calls=2 tool_calls=1 tokens=1832/241 thinking=0 latency=4.1s trace=4bf92f3577b34da6a3ce929d0e0e4736

With --verbose the run also reports what reached the collector. An export that did not arrive is always reported.

What a trace looks like

A one-shot run:

invoke_agent <identity>            the whole run
├── startup <identity>             loading tools, opening stores, importing remote tools
│   └── memory_index               the start-of-run memory listing
├── chat <model>                   one model call, one event per HTTP attempt
├── execute_tool <name>            one tool call
│   ├── retrieval                  a knowledge_search
│   │   └── embeddings <model>     one request to the embeddings server
│   └── invoke_agent <remote>      a tool served by another agent
└── chat <model>

retrieval covers knowledge_search. knowledge_enumerate gets its own span of that name, since it never ranks and never uses vectors.

A full-screen run wraps the same work in a workflow, one agent invocation per turn:

invoke_workflow <identity>
├── startup <identity>
├── invoke_agent <identity>        turn 1
│   ├── chat <model>
│   └── execute_tool <name>
└── invoke_agent <identity>        turn 2
    └── chat <model>

A resumed session is a new trace, not a continuation. A trace spans two processes only across an a2a call, where the request carries the caller’s trace context. Group by gen_ai.conversation.id to see a session’s whole history. A resumed run’s first chat span continues the iteration numbering, so fisk.llm.iteration starting at 17 is expected.

Attributes

Standard gen_ai.* attributes carry the model, the token usage, the tool names and the stop reasons. Fisk-specific ones use a fisk. prefix.

Group tool calls by fisk.tool.outcome: a policy denial, an unknown tool and a failed command all return an error to the model, and only this tells them apart.

A run with content capture on carries the gen_ai.* content attributes and fisk.content.* alongside these.

On execute_tool, fisk.memory.backend and .location are present on memory tool calls only, so filtering on them selects the calls that reached the store. They describe the tool that ran, which a PreToolUse hook can change, and they are span attributes rather than metric labels: on the tool duration histogram the backend would be empty for every tool that is not a memory tool. They stay on startup as well, which is where a run that binds a store but never calls a memory tool reports it.

Model call attempts

One chat span can be several HTTP requests: the Anthropic SDK retries a rate limit or a transport failure inside the call. The span covers the whole call, and each attempt is recorded as an event on it.

A call that took ten seconds with http.request.resend_count of 3 spent most of it waiting on retries, not on the model. The status code is on the events rather than the span because a span attribute would be last-attempt-wins and report 200 for a call that spent most of its time being rate limited.

Nothing about the request or the response body is recorded: not the URL, which can carry credentials in its userinfo, not the headers, and not the response body or the error text. An attempt is a status code, a duration and an ordinal.

Remote agents

A tool served by another agent gets an invoke_agent <remote> span inside the execute_tool span that dispatched it. It covers the a2a call.

error.type separates the failures that look alike from the model’s side:

The request carries this span’s trace context, so a peer running Fisk AI puts its own span for the call in this trace and a slow remote call shows where the time went. A peer that exports nothing, or one that is not Fisk AI, still shows only as a slow span here.

The two sides can still disagree about when the call ended. A served call reports that it is running every ten seconds, so a caller gives up only when those stop or when harness.tool_timeout ends the whole call, and a span closed as remote_unavailable under a server span that is still open means the caller stopped hearing from a peer that kept working.

Knowledge

fisk.knowledge.tier.configured and .tier.effective differing means the vector tier was configured and did not run. fisk.knowledge.degraded_reason says why, from a fixed set: embeddings and timeout are the embeddings server, index_meta is the index’s own metadata failing to read, canceled is the run stopping. Only index_meta is a problem with the store rather than the server, and it is the one case that opens no embeddings child span.

fisk.knowledge.sections counts sections and fisk.knowledge.documents counts documents; several sections routinely come from one file, so the two are not comparable.

The embeddings span carries server.address, server.port and, when a response arrived, http.response.status_code. The dimension probe is lazy and cached per process, so the first search of a run makes two embeddings requests; fisk.embeddings.purpose tells them apart. A server that cannot be reached never lets the probe cache, so every later search makes a probe request and no query request.

retrieval and embeddings do not add up to the execute_tool span above them: the tool renders its tier banner and trims results to the injection budget after the store has returned.

Indexing is not instrumented. fisk knowledge index and fisk knowledge watch start no telemetry, so the knowledge spans are constructed and discarded there.

fisk mcp does export them. A knowledge_search that arrived over MCP opens the same retrieval and embeddings spans an agent run does, and each is its own trace with retrieval at the root: there is no run above it. Everything else fisk mcp serves is uninstrumented, so a config that exposes no knowledge tools exports nothing.

Metrics

gen_ai.invoke_agent.* is recorded per turn, treating a one-shot run as one turn.

fisk.session.append.duration times each write to the run journal, and is recorded only for a checkpointed run. It is a metric rather than a span because a run appends once per record, so a span each would outnumber every other span in the trace. The file backend writes locally and sits in the lowest buckets; the jetstream backend makes a network round trip per append, and this metric shows that difference. A failed append is recorded with its error.type, so the time spent before a failure is visible rather than missing.

fisk.knowledge.degraded_searches counts searches that fell back to lexical. It is a metric rather than a span attribute alone because spans are sampled: with sample_ratio below 1.0 most degraded searches never reach the backend, and an embeddings outage silently costs every search its vector tier. There is no knowledge duration metric; gen_ai.execute_tool.duration filtered on gen_ai.tool.name covers it.

gen_ai.token.type carries only input and output, so the histogram can be summed without grouping. Set no_metrics: true to export traces alone.

Working out cost

gen_ai.usage.input_tokens includes cached tokens, and cache reads bill at roughly a tenth of the uncached rate. So:

cost = (input_tokens - cache_read - cache_creation) x uncached rate
     + cache_read                                  x cache read rate
     + cache_creation                              x cache write rate
     + output_tokens                               x output rate

fisk.llm.uncached_input_tokens is that first term already worked out, and is the same number the run summary prints.

gen_ai.usage.reasoning.output_tokens is the share of output_tokens the model spent reasoning. It is already included in output_tokens, so it does not enter the calculation above; it is there because reasoning is not displayed by default, which makes a dashboard the only place its cost is visible.

On a resumed run, gen_ai.usage.* on the root covers that process alone, so summing it across a session’s traces gives the session total once. fisk.session.usage.* carries the cumulative view for comparison. The three fisk.run.*tool_calls counters work the same way: each covers that process alone, and the remote and MCP counts are subsets of fisk.run.tool_calls on every run, resumed or not.

Privacy

By default spans carry structure and timing: no prompt, tool argument value or tool result is exported, and error messages are reduced to a fixed error.type class rather than their text.

Content capture changes that. With it on, the system prompt, the conversation, the model’s replies, tool arguments and tool results are exported as span attributes. It is off unless the config turns it on.

The OpenTelemetry credential variables are stripped from tool subprocess environments whether or not telemetry is enabled, so --no-telemetry does not re-expose a collector token. See the reference Safety section for the full list and its limits.

Content capture

Off by default. With it on, a trace carries the conversation itself:

telemetry:
  enabled: true
  endpoint: https://otel.example.net:4318
  capture:
    enabled: true
    messages: delta     # delta | full
    max_bytes: 8192     # per attribute

A run with capture on shows OTEL Enabled + content on the full-screen startup card and marks its summary line:

Run summary: model=claude-sonnet-5 llm_calls=2 tool_calls=1 tokens=1832/241 thinking=0 latency=4.1s trace=4bf92f35 content=exported

fisk info shows what would be captured, including the derived export batch size. Plain http:// to a non-loopback host is rejected at startup while capture is on.

There is no command-line flag: capture is a config setting. --no-telemetry suppresses it along with the rest of the export.

The content attributes

Each is a JSON document in the shape the GenAI conventions define.

Never exported: a thinking block’s provider signature, and the payload of a provider-specific block such as a server-side tool search result. Reasoning text is exported.

A denied tool call still exports its arguments. The system prompt is on startup rather than on each model call because it does not change during a run.

Why gen_ai.input.messages holds only one message

messages: delta, the default, exports only what each model call added to the conversation, so no single span holds the whole thing. fisk.content.from_index says where a span’s messages start; add it to the number of messages exported and you get fisk.llm.messages on the same span. Consecutive model calls chain, so a gap means a span did not arrive.

Set capture.messages: full to put the whole conversation on every model call. That is quadratic in the length of the conversation: a thirty-iteration run exports thirty copies of a growing transcript.

Some content reaches no message attribute at all. A run that ends at the iteration cap, on the token budget, or on a hook abort leaves its last tool results with no model call after them; those survive as gen_ai.tool.call.result on the tool spans. Do not sum message attributes across the traces of one session either: a resumed run’s first model call carries the whole restored conversation.

Sizing

Each attribute is capped at capture.max_bytes (256 to 65536, default 8192), measured on the encoded JSON. Over the cap, whole messages are dropped oldest-first and then text is shortened, so the document always parses; fisk.content.truncated and fisk.content.dropped_messages say what happened.

OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT lowers the cap to match rather than being overridden, so the SDK never cuts a document mid-structure.

Capture raises the size of every export, so the batch size is reduced and gzip is turned on. A collector may still need its receive limit raised:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
        # Raise when content capture is on and exports are refused.
        max_request_body_size: 8388608

With sample_ratio below 1.0, content is exported only for sampled traces.

When it does not look right

The other run outputs