# RPC, retries, and upgrades Every call Provisioner makes to a server goes through one wrapper, so `rpcWrapper` holds the retries, timing, error counting, and pause check for every call. {{% notice style="note" title="Where it lives" %}} `host/rpc.go`, 404 lines. Every action calls `rpcWrapper` at `host/rpc.go:58`. `host/version.go` holds the RPM version comparison that the upgrade decision uses. {{% /notice %}} ## The shared client setup `rpcUtilClient` and `provisionClient` (`host/rpc.go:22` and `host/rpc.go:40`) build the generated go-choria clients for the `rpcutil` and `choria_provision` agents. Both narrow the client the same way before calling `rpcWrapper`: ```go client.OptionWorkers(1).OptionTargets([]string{h.Identity}) ``` The client uses one worker and one target because a `Host` is one machine. The action name passed to `rpcWrapper` is formatted as `agent#action`, and that string becomes the `rpc` label on both `choria_provisioner_rpc_time` and `choria_provisioner_rpc_errors`, so a dashboard can show which call is slow or failing. ## The retry loop

The callback enforces the retry limit by canceling the context that runs the loop.

`backoff.Default.For` retries until the callback returns nil or the context ends, so the try limit is enforced from inside: ```go err := backoff.Default.For(tctx, func(try int) error { if try > tries { cancel() return fmt.Errorf("maximum tries reached") } if h.cfg.Paused() { cancel() return fmt.Errorf("provisioning is paused, cannot perform %s", action) } obs := prometheus.NewTimer(rpcDuration.WithLabelValues(h.cfg.Site, action)) defer obs.ObserveDuration() return cb(tctx) }) ``` The pause flag is read twice: once before the loop and once on every attempt. A provisioner that loses an election mid-provision stops at the next attempt boundary rather than finishing the sequence against a node that another instance is now provisioning. The timer is deferred inside the callback, so `choria_provisioner_rpc_time` records each attempt separately rather than the total across retries. `choria_provisioner_rpc_errors` increments once per wrapper call, after the loop gives up. ## Retry limits per action Each caller picks its own limit, and the values reflect how expensive a repeat is. | Call | Tries | Reasoning | |------|-------|-----------| | `rpcutil#inventory` | 5 | Read-only and cheap to repeat. | | `choria_provision#gen25519` | 5 | Read-only. A repeat generates a fresh nonce. | | `choria_provision#configure` | 5 | The call that writes the configuration. Repeats are idempotent on the node. | | `choria_provision#jwt` | 3 | Read-only. | | `choria_provision#restart` | 3 | A repeat restarts a node that may already be restarting. | | `choria_provision#shutdown` | 3 | A repeat targets a node that may already be shutting down. | | `choria_provision#release_update` | 3 | A repeat downloads and rewrites the binary again. | | `choria_provision#gencsr` | 1 | Each call makes the node generate a new private key. | ## Reply validation Every action checks the response count first. It must be exactly one: ```go if res.Stats().ResponsesCount() != 1 { return fmt.Errorf("... received %d responses while expecting a response from %s", res.Stats().ResponsesCount(), h.Identity) } ``` Zero means the node did not answer within the client timeout. More than one means two machines answer to the same identity, and the wrapper cannot tell which machine the configuration would reach, so it fails the call. `EachOutput` then checks `ResultDetails().OK()` and assigns to a captured `err` that the enclosing closure returns. `shutdown` is the exception. It logs a non-OK result as a warning without setting `err`, and increments `choria_provisioner_helper_shutdown_requests` only on the OK path. The helper already rejected the node, so a failed shutdown confirmation does not fail the cycle. ## The upgrade decision `handleHostUpgrade` (`host/host.go:219`) runs before configuration and refuses the node when `upgrades_repository` is unset and `upgrades_optional` is off, when the inventory carried no version, or when the node did not report `upgradable`. The comparison itself uses `host/version.go`, a vendored copy of [go-rpm-version](https://github.com/knqyf263/go-rpm-version), MIT licensed, vendored because the upstream project is unmaintained. It implements RPM's `epoch:version-release` parsing and the `rpmvercmp` segment comparison, including tilde handling and the rule that numeric segments sort higher than alphabetic ones. ```go cv := NewVersion(h.version) tv := NewVersion(h.upgradeTargetVersion) if !cv.Equal(tv) { err := h.upgrade(ctx) ... } ``` {{% notice style="warning" title="Version comparison is an equality test" %}} `handleHostUpgrade` compares the two versions with `Equal` and acts on any difference. `Version` exposes `GreaterThan` and `LessThan`, and neither is called anywhere in the codebase. A helper that returns an `upgrade` value older than the node's current version triggers a release update that moves the node backwards. Whether a version change is an upgrade or a rollback is therefore the helper's decision. A helper that must prevent rollbacks compares the versions itself before setting the field. {{% /notice %}} `NewVersion` never returns an error. An epoch that does not parse as an integer silently becomes 0, which follows the upstream behavior. When the versions differ, `upgrade` sends `release_update` with the configured repository, the token, and the target version, and `Provision` returns with no delay so the node is re-provisioned on the next cycle. {{% notice style="tip" title="Next" %}} [Pausing and leader election]({{% relref "clustering" %}}) covers the pause flag that gates every call on this page. {{% /notice %}}