# Pausing and leader election Running several Provisioners against one broker would have them race to configure the same servers. Instead, every instance starts paused and the election winner resumes. {{% notice style="note" title="Where it lives" %}} `config/pausable.go` holds the flag and its four methods. `hosts/election.go` calls `Pause` and `Resume` from the election callbacks. `config/stats.go` exports the flag as a gauge. {{% /notice %}} ## The pause flag `config.Config` embeds a `sync.Mutex` and carries an unexported `paused` bool. `Pause`, `Resume`, `Flip`, and `Paused` guard it, and each sets `choria_provisioner_paused` before returning: ```go func (c *Config) Pause() { c.Lock() defer c.Unlock() c.paused = true c.setPauseStat() } ``` `config.Load` sets the gauge to 0 at startup, so the metric exists before any election runs. Every action that leaves the process reads the flag first: | Reader | Effect when paused | |--------|--------------------| | `hosts/hosts.go:187` in `discover` | Logs and returns without sending the broadcast request. | | `hosts/event.go:63` in `handle` | Logs and drops the lifecycle event. | | `host/rpc.go:59` in `rpcWrapper` | Fails the call before the retry loop starts. | | `host/rpc.go:74` inside the retry loop | Cancels the context and fails at the next attempt. | | `host/helper.go:72` in `runHelper` | Refuses to run the helper. | {{% notice style="warning" title="The flag is checked inside the retry loop" %}} `rpcWrapper` reads the flag inside the retry loop as well as before it, so a Provisioner that loses an election mid-sequence stops at the next attempt boundary. Without the inner check, an instance could carry on sending configuration to a node that the new leader has already started working on. {{% /notice %}} `Flip` is defined alongside the other three and has no caller anywhere in the repository. All four methods carry comments saying they implement `backplane.Pausable`, but this codebase does not import the `backplane` package. The four methods satisfy the interface for a consumer that this repository does not contain. ## Election callbacks Setting `leader_election` makes `hosts.Process` call `conf.Pause()` before it starts any goroutine, so the instance is inert from its first line of work. `startElection` (`hosts/election.go:17`) then registers two callbacks with a Choria Streams election named `provisioner`.

A standby instance holds connections and goroutines but performs no work until it wins.

The `won` callback resumes work and then writes to `discoverTrigger`, a channel of capacity one that `hosts.Process` selects on alongside the interval ticker. A new leader therefore runs a discovery immediately instead of waiting up to a full interval, so provisioning continues within seconds of a failover. The `lost` callback pauses and calls `removeAllHosts`, which empties the map and then drains the `work` channel in a non-blocking loop until it is empty. A host already passed to a worker stays with that worker, but the worker's `isCurrent` check rejects it before provisioning starts. [Discovery and the work queue]({{% relref "discovery" %}}) covers that check. Campaigning uses `backoff.TwentySec`, so a failover can take up to a minute of standby time before a new leader starts working. ## Failure handling in the election path `hosts.Process` launches `startElection` in a goroutine, so its returned error is discarded. If `fw.NewElection` fails, the instance stays paused and logs nothing further. `elect.Start` runs in a nested goroutine and calls `log.Fatalf` on error, ending the process rather than leaving the instance paused: ```go go func() { err := elect.Start(ctx) if err != nil { log.Fatalf("Leader election failed to start: %s", err) } }() ``` `startElection` defers `wg.Done()` and returns once the election is registered, so its `WaitGroup` entry clears at setup rather than at shutdown. ## Single-instance deployment With `leader_election` unset, `paused` stays false for the life of the process and every gate passes. A single instance needs no election, and the documentation recommends that deployment unless high availability is required. [Configuration File]({{% relref "/configuration/provisioner" %}}) covers the operational side. {{% notice style="tip" title="Next" %}} [Reference and map]({{% relref "reference" %}}) collects the file map, configuration keys, and metrics in one place. {{% /notice %}}