Pausing and leader election

Running several Provisioners against one broker would have them race to configure the same servers. Instead, every instance starts paused and the election winner resumes.

The pause flag

config.Config embeds a sync.Mutex and carries an unexported paused bool. Pause, Resume, Flip, and Paused guard it, and each sets choria_provisioner_paused before returning:

func (c *Config) Pause() {
	c.Lock()
	defer c.Unlock()

	c.paused = true
	c.setPauseStat()
}

config.Load sets the gauge to 0 at startup, so the metric exists before any election runs.

Every action that leaves the process reads the flag first:

Flip is defined alongside the other three and has no caller anywhere in the repository. All four methods carry comments saying they implement backplane.Pausable, but this codebase does not import the backplane package. The four methods satisfy the interface for a consumer that this repository does not contain.

Election callbacks

Setting leader_election makes hosts.Process call conf.Pause() before it starts any goroutine, so the instance is inert from its first line of work. startElection (hosts/election.go:17) then registers two callbacks with a Choria Streams election named provisioner.

The won callback resumes work and then writes to discoverTrigger, a channel of capacity one that hosts.Process selects on alongside the interval ticker. A new leader therefore runs a discovery immediately instead of waiting up to a full interval, so provisioning continues within seconds of a failover.

The lost callback pauses and calls removeAllHosts, which empties the map and then drains the work channel in a non-blocking loop until it is empty. A host already passed to a worker stays with that worker, but the worker’s isCurrent check rejects it before provisioning starts. Discovery and the work queue covers that check.

Campaigning uses backoff.TwentySec, so a failover can take up to a minute of standby time before a new leader starts working.

Failure handling in the election path

hosts.Process launches startElection in a goroutine, so its returned error is discarded. If fw.NewElection fails, the instance stays paused and logs nothing further.

elect.Start runs in a nested goroutine and calls log.Fatalf on error, ending the process rather than leaving the instance paused:

go func() {
	err := elect.Start(ctx)
	if err != nil {
		log.Fatalf("Leader election failed to start: %s", err)
	}
}()

startElection defers wg.Done() and returns once the election is registered, so its WaitGroup entry clears at setup rather than at shutdown.

Single-instance deployment

With leader_election unset, paused stays false for the life of the process and every gate passes. A single instance needs no election, and the documentation recommends that deployment unless high availability is required. Configuration File covers the operational side.