RPC, retries, and upgrades

Every call Provisioner makes to a server goes through one wrapper, so rpcWrapper holds the retries, timing, error counting, and pause check for every call.

The shared client setup

rpcUtilClient and provisionClient (host/rpc.go:22 and host/rpc.go:40) build the generated go-choria clients for the rpcutil and choria_provision agents. Both narrow the client the same way before calling rpcWrapper:

client.OptionWorkers(1).OptionTargets([]string{h.Identity})

The client uses one worker and one target because a Host is one machine. The action name passed to rpcWrapper is formatted as agent#action, and that string becomes the rpc label on both choria_provisioner_rpc_time and choria_provisioner_rpc_errors, so a dashboard can show which call is slow or failing.

The retry loop

backoff.Default.For retries until the callback returns nil or the context ends, so the try limit is enforced from inside:

err := backoff.Default.For(tctx, func(try int) error {
	if try > tries {
		cancel()
		return fmt.Errorf("maximum tries reached")
	}

	if h.cfg.Paused() {
		cancel()
		return fmt.Errorf("provisioning is paused, cannot perform %s", action)
	}

	obs := prometheus.NewTimer(rpcDuration.WithLabelValues(h.cfg.Site, action))
	defer obs.ObserveDuration()

	return cb(tctx)
})

The pause flag is read twice: once before the loop and once on every attempt. A provisioner that loses an election mid-provision stops at the next attempt boundary rather than finishing the sequence against a node that another instance is now provisioning.

The timer is deferred inside the callback, so choria_provisioner_rpc_time records each attempt separately rather than the total across retries. choria_provisioner_rpc_errors increments once per wrapper call, after the loop gives up.

Retry limits per action

Each caller picks its own limit, and the values reflect how expensive a repeat is.

Reply validation

Every action checks the response count first. It must be exactly one:

if res.Stats().ResponsesCount() != 1 {
	return fmt.Errorf("... received %d responses while expecting a response from %s",
		res.Stats().ResponsesCount(), h.Identity)
}

Zero means the node did not answer within the client timeout. More than one means two machines answer to the same identity, and the wrapper cannot tell which machine the configuration would reach, so it fails the call. EachOutput then checks ResultDetails().OK() and assigns to a captured err that the enclosing closure returns.

shutdown is the exception. It logs a non-OK result as a warning without setting err, and increments choria_provisioner_helper_shutdown_requests only on the OK path. The helper already rejected the node, so a failed shutdown confirmation does not fail the cycle.

The upgrade decision

handleHostUpgrade (host/host.go:219) runs before configuration and refuses the node when upgrades_repository is unset and upgrades_optional is off, when the inventory carried no version, or when the node did not report upgradable.

The comparison itself uses host/version.go, a vendored copy of go-rpm-version, MIT licensed, vendored because the upstream project is unmaintained. It implements RPM’s epoch:version-release parsing and the rpmvercmp segment comparison, including tilde handling and the rule that numeric segments sort higher than alphabetic ones.

cv := NewVersion(h.version)
tv := NewVersion(h.upgradeTargetVersion)

if !cv.Equal(tv) {
	err := h.upgrade(ctx)
	...
}

NewVersion never returns an error. An epoch that does not parse as an integer silently becomes 0, which follows the upstream behavior.

When the versions differ, upgrade sends release_update with the configured repository, the token, and the target version, and Provision returns with no delay so the node is re-provisioned on the next cycle.