Architecture

The source is four packages and about 2,400 lines.

The packages

cmd

The fisk command line, flag parsing, and process startup. Loads both configuration files, builds the Choria framework, starts the Prometheus listener and the signal handler, then calls hosts.Process. One file, cmd/provisioner.go.

config

The YAML configuration, its defaults, and its validation. Also holds the pause flag that rpcWrapper and runHelper check before every outbound action. Files: config.go, pausable.go, stats.go.

hosts

Fleet-level scheduling. Broadcast discovery, the lifecycle listener, a work queue, the worker pool, the finisher, and leader election. Files: hosts.go, event.go, provisioner.go, election.go, stats.go.

host

Everything that happens to one server. The provisioning sequence, the RPC wrappers, the helper invocation, and RPM version comparison. Files: host.go, rpc.go, helper.go, version.go, stats.go.

cmd imports hosts, hosts imports host, and all three import config, which imports none of them. host does not import hosts, so host/host_test.go builds a Host from a config.Config and a logger with no scheduler involved.

The boundaries

The helper, the pause flag, and the generated RPC clients each change without touching the other two.

1. The helper Provisioner marshals a Host to JSON, writes it to a child process on standard input, and unmarshals the reply into host.ConfigResponse (host/helper.go:38). The helper reaches certificate authorities, CMDBs, and naming services, in any language.
2. The pause flag config.Config declares Pause, Resume, Flip, and Paused (config/pausable.go). Leader election calls Pause and Resume, and both rpcWrapper and runHelper check the flag before sending an RPC request or starting the helper process.
3. Generated RPC clients host/rpc.go uses two clients generated in go-choria, provclient.ChoriaProvisionClient for the choria_provision agent and rpcutilclient.RpcutilClient for rpcutil. Every call to either goes through rpcWrapper, which holds the retries, timing, and pause check for both.

Package-level state in hosts

The hosts package declares its state as package variables rather than struct fields (hosts/hosts.go:24-33): the hosts map of identity to *host.Host, the buffered work and done channels, a mutex, and the logger, the Choria framework, and the configuration.

var (
	hosts = make(map[string]*host.Host)
	work  = make(chan *host.Host, 50000)
	done  = make(chan *host.Host, 50000)
	mu    = &sync.Mutex{}
	log   *logrus.Entry
	fw    *choria.Framework
	conf  *config.Config
	wg    = &sync.WaitGroup{}
)

Configuration and startup

cmd.run loads the Provisioner YAML from --config through config.Load, and the Choria client configuration from --choria-config, defaulting to choria.UserConfig(). cmd.run then overrides the Choria client configuration in code (cmd/provisioner.go:61-79): log level and log file come from the Provisioner configuration, and the collective is forced to provisioning for both Collectives and MainCollective.

Setting choria_insecure disables TLS, sets protocol.Secure to false, and forces the file security provider. Setting broker_provisioning_password switches the NATS credentials to user provisioner with that password. The broker’s dedicated provisioning account accepts that user.

config.Load fills defaults and rejects configurations that cannot work (config/config.go:64-142):

Observability

config/stats.go, hosts/stats.go, and host/stats.go each register their metrics in an init function: the pause gauge, the fleet-level counters, and per-server timing and errors. Every metric carries a site label taken from the site configuration key, so a query across installations can sum or group by site. cmd.setupPrometheus serves them on /metrics when monitor_port is set.

Discovery and the work queue

Broadcast discovery and lifecycle events both reach the same admission function, add. Everything after add is identical.

Startup order

hosts.Process (hosts/hosts.go:36) runs for the life of the process. It stores the framework, configuration, and logger in package variables, opens a connector, and publishes a startup lifecycle event for the provisioner component so a fleet operator can see the instance start.

It then starts goroutines in a fixed order:

1. Leader election, when enabled If leader_election is set, conf.Pause() runs first so the instance starts inert, then startElection begins campaigning.
2. The lifecycle listener One goroutine subscribed to startup events for the configured component.
3. The finisher One goroutine draining the done channel.
4. The worker pool cfg.Workers goroutines, each running provisioner().

hosts.Process then sets every Prometheus counter and gauge to zero, so the full metric set appears on /metrics before the first server arrives. It runs discover(ctx) once, then blocks on a select over the interval ticker, the election trigger channel, and context cancellation.

Broadcast discovery

discoverProvisionableNodes (hosts/hosts.go:201) builds a filter for the choria_provision agent and runs a broadcast discovery against the provisioning collective:

bd := broadcast.New(fw)
nodes, err := bd.Discover(ctx,
	broadcast.Collective("provisioning"),
	broadcast.Filter(f),
	broadcast.SlidingWindow(),
	broadcast.Timeout(2*time.Second))

The timeout is a fixed two seconds and the window slides, so the call returns as soon as replies stop arriving rather than always waiting the full two seconds. Every returned identity becomes a new host.Host and goes through add.

discover (hosts/hosts.go:186) checks the pause flag first. When paused it logs, returns before sending the broadcast request, and leaves the cycle counter unchanged, so a standby instance only campaigns for the election.

Lifecycle startup events

listen (hosts/event.go:26) queue-subscribes to choria.lifecycle.event.startup.<lifecycle_component>, where the component defaults to provision_mode_server. config.Load rejects a component containing ., >, or *, because the value is interpolated straight into a NATS subject and those characters would widen the subscription.

Each message goes through handle, which checks the pause flag, parses the event, and returns the identity only for events of type lifecycle.Startup. listen therefore admits a server that has just booted into provisioning mode without waiting for the next discovery cycle. Broadcast discovery and the listener increment separate counters, choria_provisioner_discovered and choria_provisioner_event_discovered, so their relative contribution is visible in monitoring.

Admission and deduplication

add (hosts/hosts.go:153) takes the mutex, and no other function writes to the work channel.

For an identity already in the map, add compares the age of the new sighting against conf.IntervalDuration:

if time.Since(host.DiscoveredTime()) < conf.IntervalDuration {
	return false
}
removeUnlocked(host)

In that comparison, host is the newly constructed Host, whose discovered field NewHost sets to time.Now(). The comparison therefore measures the age of the sighting being processed, not the age of the entry already in the map.

add then writes to the buffered work channel in a non-blocking select. If the buffer is full, add logs a failure and deletes the identity from the map, so the map holds only entries that are on the queue.

Workers

Each worker (hosts/provisioner.go:15) blocks on the work channel. It then calls isCurrent, which looks the identity up in the map, before calling provisionTarget:

if !isCurrent(host) {
	continue
}

An entry can be on the queue but no longer in the map, which happens when removeAllHosts runs after a lost election. The worker skips the entry and does not send it to done, because removeAllHosts already deleted the identity.

The worker then calls provisionTarget, which increments choria_provisioner_busy_workers for the duration and delegates to host.Provision. The return value carries a delay flag alongside the error:

The 60 second delay leaves the node in the map while it restarts, so the next discovery cycle skips it. After an in-place version upgrade the flag is false instead, and the node is re-provisioned on the next cycle.

The finisher

One goroutine drains done and calls remove, which deletes the identity from the map and updates both the choria_provisioner_waiting_nodes and choria_provisioner_work_queue_entries gauges. Only the finisher calls remove, so workers never take the mutex for deletion and one goroutine writes both gauges.

The provisioning cycle

Host.Provision runs on a worker goroutine, holds the host’s mutex for the duration of the call, and returns a delay flag and an error.

The sequence

The feature flags gate individual steps. The order never changes.

Checks before the first RPC

Provision checks provisioned and discovery age before it issues any RPC.

if h.provisioned {
	return true, nil
}

if !h.discovered.IsZero() {
	since := time.Since(h.discovered)
	if since > 2*h.cfg.IntervalDuration {
		return false, fmt.Errorf("skipping node that's been waiting %v", since)
	}
}

The provisioned short-circuit returns a delay, so the finisher holds a node that has already been through the sequence out of the map for a further 60 seconds.

The staleness check is the second half of the duplicate-provision trade-off described in Discovery and the work queue. Because the queue holds up to 50,000 entries, a host can wait long enough that the facts the helper would see no longer match the node. Provision refuses a host that has waited longer than two discovery intervals.

Gathering facts

Each fetch is a single RPC against one target, described in RPC, retries, and upgrades. fetchJWT, fetchInventory, and fetchEd25519PubKey each return early when the value they collect is already populated, so a retried Provision on the same Host does not re-fetch.

fetchInventory always runs. Besides the facts, it records h.version and h.upgradable from the reply, and the upgrade path requires both.

The helper’s answer

getConfig marshals the whole Host to JSON and runs the helper. Its reply, a host.ConfigResponse, can end the cycle:

defer

Returns an error carrying the helper's message. The node stays unconfigured. The finisher removes it from the map, and a later discovery cycle picks it up again.

shutdown

Issues a shutdown RPC to the node and returns a delay. The node exits with code 0, which systemd does not restart. When msg is empty, Provision logs a warning that carries no shutdown reason.

Provision copies the rest of the reply onto the Host: the configuration map, the CA, certificate, key, and SSL directory, the target upgrade version, and the two policy maps. The helper contract gives the full shape.

The upgrade interruption

When features.upgrades is on and the helper returned an upgrade version, handleHostUpgrade runs before configuration. It refuses to proceed if the inventory carried no version or the node reported upgradable as false, then compares versions with the RPM comparison in host/version.go.

The switch that consumes its result (host/host.go:189-202) handles these outcomes:

Configure and restart

configure sends the whole payload in one RPC: the configuration map as a JSON string, the token, the CA, certificate, key, SSL directory, the Provisioner’s ECDH public key, both policy maps, and the signed server JWT. It refuses to send an empty configuration map. If the CSR reply carried an SSL directory, that value overrides whatever the helper chose.

restart follows with a one second splay, and the node restarts into its new configuration. Provision sets h.provisioned to true after the restart returns, then returns (true, nil).

The helper contract

Provisioner ships no knowledge of any particular site. It cannot name a node, reach a certificate authority, pick a broker, or decide which policies a machine should carry, because every one of those answers differs between deployments and most of them live in systems Provisioner has never heard of. The helper supplies them. It is a program in any language, named by the helper configuration key, that reads a JSON description of one server on standard input and writes that server’s configuration back on standard output.

The helper runs at the midpoint of the provisioning cycle, once per server. By the time Provisioner calls it, the fetch steps have already collected everything the node can say about itself: its verified provisioning.jwt, its inventory, and its CSR or ed25519 public key, depending on the enrollment model. The helper turns that description into a decision. It can return a configuration map, x509 credentials, a signed-token request, Open Policy Agent and Action Policy documents, and a version to upgrade to. It can also refuse: defer leaves the node unconfigured for a later cycle, and shutdown stops it. Provisioner takes whatever comes back, sends it to the node in a single configure call, and restarts the node.

One exec per server

What the helper receives

getConfig marshals the Host itself, so the input shape is exactly the struct’s exported fields (host/host.go:29-51). The JSON-tagged fields are the only ones the helper sees; every other field on Host is unexported and stays in the process.

identity

The Choria Server identity, typically the fully qualified domain name.

csr

A provision.CSRReply, non-null only when features.pki is enabled.

ed25519_pubkey

A provision.ED25519Reply, non-null only when features.ed25519 is enabled.

inventory

The rpcutil#inventory result. A JSON string, not a nested object.

jwt

The parsed and already-verified provisioning.jwt claims, non-null only when features.jwt is enabled.

The token, the signing keys, and the helper path itself are not sent. A helper that needs a secret reads it from its own environment or files.

What the helper returns

The reply decodes into host.ConfigResponse (host/helper.go:23). Unknown keys are ignored and every field is optional, so a minimal helper answers with a configuration object alone.

type ConfigResponse struct {
	Defer          bool                 `json:"defer"`
	Shutdown       bool                 `json:"shutdown"`
	Msg            string               `json:"msg"`
	Key            string               `json:"key"`
	Certificate    string               `json:"certificate"`
	CA             string               `json:"ca"`
	SSLDir         string               `json:"ssldir"`
	ServerClaims   *tokens.ServerClaims `json:"server_claims"`
	Configuration  map[string]string    `json:"configuration"`
	ActionPolicies map[string]string    `json:"action_policies"`
	OPAPolicies    map[string]string    `json:"opa_policies"`
	UpgradeVersion string               `json:"upgrade"`
}

Provision checks defer and shutdown first, and either one ends the cycle. upgrade diverts to the upgrade path. Provision copies the rest onto the Host and sends it to the node in the single configure RPC. The provisioning cycle gives the order those checks run in, and Writing a helper gives worked examples in Ruby.

The child process

runHelper runs these steps in order.

1. Timing starts first A prometheus.NewTimer on choria_provisioner_helper_time is deferred before anything else, so a timed-out helper still records its duration.
2. The pause flag is checked A paused instance refuses to start the helper, and the error includes the helper path.
3. A 10 second timeout is applied context.WithTimeout(ctx, 10*time.Second) wraps the whole call. No configuration key sets the value.
4. The command string is split with shellquote shellquote.Split handles quoting, so helper can carry arguments, for example /opt/prov/helper.rb --site london. An empty result is rejected.
5. Input is written from a goroutine The write to stdin runs concurrently with the read from stdout and closes the pipe on completion. Doing both in one goroutine would deadlock on any helper whose output exceeds the pipe buffer.
6. Output is read to completion, then the process is waited on Reading before Wait is required: Wait closes the pipes.
7. Zero bytes is an error A helper that exits cleanly without writing anything fails with "zero bytes received" rather than producing an empty configuration.
8. The exit code is checked ProcessState.Success() must be true.

Provision counts the failures. It increments choria_provisioner_helper_errors when getConfig returns an error (host/host.go:134), and the successful shutdown path increments choria_provisioner_helper_shutdown_requests from host/rpc.go:169.

Security and enrollment

Provisioner talks to servers that have no credentials yet. It checks every value it accepts from a node before using it, and signs every credential it returns with keys that the node cannot reach.

Enrollment models

features.pki and features.ed25519 select mutually exclusive models, and config.Load rejects a configuration that sets both. features.jwt is independent, and enabling features.ed25519 turns it on implicitly.

Provisioning token verification

Runs when features.jwt is set. Setting features.ed25519 sets it too, so this step also runs in Organization Issuer deployments.

validateJWT (host/host.go:405) refuses to run without jwt_verify_cert. os.Stat decides whether that value is a key file path or a hex ed25519 public key:

if _, err = os.Stat(h.cfg.JWTVerifyCert); os.IsNotExist(err) {
	pk, err = hex.DecodeString(h.cfg.JWTVerifyCert)
	claims, err = tokens.ParseProvisioningToken(h.rawJWT, ed25519.PublicKey(pk))
} else {
	claims, err = tokens.ParseProvisioningTokenWithKeyfile(h.rawJWT, h.cfg.JWTVerifyCert)
}

A path that exists on disk is read as a key file. Anything else is decoded as a hex ed25519 public key, the form of an Organization Issuer public key. validateJWT stores the parsed claims on h.JWT, and getConfig sends them to the helper as the verified jwt field.

The ed25519 challenge

Runs when features.ed25519 is set. features.pki deployments skip it, and config.Load rejects a configuration that sets both.

fetchEd25519PubKey (host/rpc.go:228) generates a nonce with choria.NewRequestID(), sends it in the gen25519 request, and verifies the node’s signature over it:

if !ed25519.Verify(pk, []byte(h.nonce), sig) {
	err = fmt.Errorf("invalid nonce signature")
	return
}

Without the nonce, a node could present any public key, including one belonging to a different machine, and receive a server JWT bound to that key. The signature proves the node holds the matching seed. The call also records the directory that the node created the seed in, which becomes the SSL directory in the generated configuration.

Server token issuance

Runs when features.ed25519 is set, immediately after the helper returns. It also needs jwt_signing_key, and server_jwt_validity or its one year default.

generateServerJWT (host/host.go:251) builds the claims in increasing order of precedence: built-in defaults, the configuration that the helper returned, and the helper’s explicit server_claims.

org

Defaults to choria. Overridden by server_claims.ou.

collectives

Defaults to mcollective. Becomes choria when the returned configuration sets plugin.security.provider to choria, then a comma-split of the collectives configuration key, then server_claims.collectives.

validity

Defaults to server_jwt_validity, or one year. A server_claims expiry is converted with time.Until.

permissions

Null unless server_claims.permissions is set.

additional publish subjects

Empty unless server_claims.additional_publish_subjects is set.

A computed validity of one hour or less is discarded and replaced with the configured default, which covers both an expiry already in the past and a claim that failed to parse.

With a signing token present, the token must carry a TrustChainSignature claim, and claims.AddChainIssuerData attaches the chain before signing. tokens.SignTokenWithKeyFile signs the result, and the configure call sends it to the node as server_jwt.

CSR checks

Runs when features.pki is set. features.ed25519 deployments skip it, because no x509 certificate is issued.

validateCSR (host/host.go:435) parses the PEM that the node sent. The common name must equal the identity that Provisioner discovered, and no name on the request may match the denylist. validateCSR checks the common name and every DNS SAN.

The default denylist blocks the certificate names that Choria reserves for privileged clients:

cert_deny_list:
  - "\\.privileged.mcollective$"
  - "\\.privileged.choria$"
  - "\\.mcollective$"
  - "\\.choria$"

matchAnyRegex (host/host.go:468) treats a pattern wrapped in forward slashes as a bare regular expression and strips them, then matches with regexp.MatchString. Matching is unanchored, so a pattern without ^ or $ matches anywhere in the name.

Private key protection in transit

Runs when the helper returns a key, which no feature flag controls. It also needs features.jwt, because the server’s ECDH public key arrives on the jwt reply.

Some certificate authorities generate the key pair themselves rather than signing a CSR. For those, the helper can return a key, and it has to cross the network to a node that holds no credentials. encryptPrivateKey (host/host.go:354) uses an ephemeral Diffie-Hellman exchange over the two RPC calls that already happen.

The server’s ECDH public key arrives on the jwt reply as EcdhPublic, so key encryption requires features.jwt. Without it encryptPrivateKey fails with “private key received from helper but server did not start Diffie-Hellman exchange”. encryptPrivateKey stores Provisioner’s own public key on h.provisionPubKey, and the configure call sends it as EcdhPublic, so the node can derive the same secret and decrypt the PEM block.

x509.EncryptPEMBlock is deprecated in the standard library and the call carries a //lint:ignore SA1019 there is no alternative comment. The format is fixed by what Choria Server can decrypt.

RPC, retries, and upgrades

Every call Provisioner makes to a server goes through one wrapper, so rpcWrapper holds the retries, timing, error counting, and pause check for every call.

The shared client setup

rpcUtilClient and provisionClient (host/rpc.go:22 and host/rpc.go:40) build the generated go-choria clients for the rpcutil and choria_provision agents. Both narrow the client the same way before calling rpcWrapper:

client.OptionWorkers(1).OptionTargets([]string{h.Identity})

The client uses one worker and one target because a Host is one machine. The action name passed to rpcWrapper is formatted as agent#action, and that string becomes the rpc label on both choria_provisioner_rpc_time and choria_provisioner_rpc_errors, so a dashboard can show which call is slow or failing.

The retry loop

backoff.Default.For retries until the callback returns nil or the context ends, so the try limit is enforced from inside:

err := backoff.Default.For(tctx, func(try int) error {
	if try > tries {
		cancel()
		return fmt.Errorf("maximum tries reached")
	}

	if h.cfg.Paused() {
		cancel()
		return fmt.Errorf("provisioning is paused, cannot perform %s", action)
	}

	obs := prometheus.NewTimer(rpcDuration.WithLabelValues(h.cfg.Site, action))
	defer obs.ObserveDuration()

	return cb(tctx)
})

The pause flag is read twice: once before the loop and once on every attempt. A provisioner that loses an election mid-provision stops at the next attempt boundary rather than finishing the sequence against a node that another instance is now provisioning.

The timer is deferred inside the callback, so choria_provisioner_rpc_time records each attempt separately rather than the total across retries. choria_provisioner_rpc_errors increments once per wrapper call, after the loop gives up.

Retry limits per action

Each caller picks its own limit, and the values reflect how expensive a repeat is.

Reply validation

Every action checks the response count first. It must be exactly one:

if res.Stats().ResponsesCount() != 1 {
	return fmt.Errorf("... received %d responses while expecting a response from %s",
		res.Stats().ResponsesCount(), h.Identity)
}

Zero means the node did not answer within the client timeout. More than one means two machines answer to the same identity, and the wrapper cannot tell which machine the configuration would reach, so it fails the call. EachOutput then checks ResultDetails().OK() and assigns to a captured err that the enclosing closure returns.

shutdown is the exception. It logs a non-OK result as a warning without setting err, and increments choria_provisioner_helper_shutdown_requests only on the OK path. The helper already rejected the node, so a failed shutdown confirmation does not fail the cycle.

The upgrade decision

handleHostUpgrade (host/host.go:219) runs before configuration and refuses the node when upgrades_repository is unset and upgrades_optional is off, when the inventory carried no version, or when the node did not report upgradable.

The comparison itself uses host/version.go, a vendored copy of go-rpm-version, MIT licensed, vendored because the upstream project is unmaintained. It implements RPM’s epoch:version-release parsing and the rpmvercmp segment comparison, including tilde handling and the rule that numeric segments sort higher than alphabetic ones.

cv := NewVersion(h.version)
tv := NewVersion(h.upgradeTargetVersion)

if !cv.Equal(tv) {
	err := h.upgrade(ctx)
	...
}

NewVersion never returns an error. An epoch that does not parse as an integer silently becomes 0, which follows the upstream behavior.

When the versions differ, upgrade sends release_update with the configured repository, the token, and the target version, and Provision returns with no delay so the node is re-provisioned on the next cycle.

Pausing and leader election

Running several Provisioners against one broker would have them race to configure the same servers. Instead, every instance starts paused and the election winner resumes.

The pause flag

config.Config embeds a sync.Mutex and carries an unexported paused bool. Pause, Resume, Flip, and Paused guard it, and each sets choria_provisioner_paused before returning:

func (c *Config) Pause() {
	c.Lock()
	defer c.Unlock()

	c.paused = true
	c.setPauseStat()
}

config.Load sets the gauge to 0 at startup, so the metric exists before any election runs.

Every action that leaves the process reads the flag first:

Flip is defined alongside the other three and has no caller anywhere in the repository. All four methods carry comments saying they implement backplane.Pausable, but this codebase does not import the backplane package. The four methods satisfy the interface for a consumer that this repository does not contain.

Election callbacks

Setting leader_election makes hosts.Process call conf.Pause() before it starts any goroutine, so the instance is inert from its first line of work. startElection (hosts/election.go:17) then registers two callbacks with a Choria Streams election named provisioner.

The won callback resumes work and then writes to discoverTrigger, a channel of capacity one that hosts.Process selects on alongside the interval ticker. A new leader therefore runs a discovery immediately instead of waiting up to a full interval, so provisioning continues within seconds of a failover.

The lost callback pauses and calls removeAllHosts, which empties the map and then drains the work channel in a non-blocking loop until it is empty. A host already passed to a worker stays with that worker, but the worker’s isCurrent check rejects it before provisioning starts. Discovery and the work queue covers that check.

Campaigning uses backoff.TwentySec, so a failover can take up to a minute of standby time before a new leader starts working.

Failure handling in the election path

hosts.Process launches startElection in a goroutine, so its returned error is discarded. If fw.NewElection fails, the instance stays paused and logs nothing further.

elect.Start runs in a nested goroutine and calls log.Fatalf on error, ending the process rather than leaving the instance paused:

go func() {
	err := elect.Start(ctx)
	if err != nil {
		log.Fatalf("Leader election failed to start: %s", err)
	}
}()

startElection defers wg.Done() and returns once the election is registered, so its WaitGroup entry clears at setup rather than at shutdown.

Single-instance deployment

With leader_election unset, paused stays false for the life of the process and every gate passes. A single instance needs no election, and the documentation recommends that deployment unless high availability is required. Configuration File covers the operational side.

Reference and map

Each table links to the page that explains the area it covers.

Command line

main.go calls cmd.Run, which builds a fisk application named choria-provisioner. The run command is the default, so choria-provisioner --config ... and choria-provisioner run --config ... are equivalent.

--version prints config.Version, which is 0.0.0 in a plain go build. The release build injects it through -ldflags, mapped in packager/buildspec.yaml to github.com/choria-io/provisioner/config.Version.

Source map

Key types

config.Config

config/config.go:25. The parsed YAML plus derived durations, the config file path, and the pause flag. Passed by pointer to every package.

host.Host

host/host.go:29. One server. Five exported fields form the helper's input. The remaining fields hold state that the RPC sequence fills in.

host.ConfigResponse

host/helper.go:23. The helper's reply. Twelve optional fields.

host.Version

host/version.go:21. An RPM epoch, version, and release triple with a Compare method.

Exported metrics

Every metric carries a site label taken from the site configuration key. Three init functions register them, one per package.

The rpc label is the agent#action string, for example choria_provision#configure or rpcutil#inventory. Monitoring describes what each metric measures.

Configuration keys

Configuration File gives the full operational description. The following table gives the struct field and the reader for each key.

rego_policy is declared on Config as RegoPolicy and read nowhere. OPA policies reach a node through the helper’s opa_policies reply instead.

Choria RPC actions used

Glossary

fleet node

A machine running Choria Server. Provisioner's unit of work.

provisioning collective

The sub-collective that an unconfigured server joins. Provisioner forces both collectives and main_collective to this value.

provisioning.jwt

A token placed on a node that enables provisioning mode, carries the broker address, and holds the shared token.

helper

The site-supplied program that turns a server's identity and inventory into a configuration.

Organization Issuer

The ed25519 key at the root of a certificate-authority-free Choria deployment. Provisioner signs server tokens that chain to it.

server JWT

The token that Provisioner issues for a node in an Organization Issuer deployment, replacing an x509 certificate.

lifecycle event

A JSON event that Choria components publish on choria.lifecycle.event.>. Provisioner subscribes to the startup ones.

leader election

A Choria Streams primitive that names one instance in a cluster as active.

paused

The flag that a standby instance sets. Every outbound action checks it.

splay

A random delay that a server applies before acting on a restart or shutdown, spreading the load across a fleet.

site

A name for one installation, used as a label on every metric so a dashboard can aggregate across installations.