Fleet Management

Choria Server that is enabled for Provisioning expose a number of endpoints that can be used to interact with it in ways that is not otherwise possible.

These abilities are not possible without enabling Provisioning in the server, so they are off by default. You should ensure your site security policies have adequate control over these actions.

Checking if upgrades are supported

If you are running a new enough CLI and Server you can check if a running server supports version upgrades:

$ choria inventory 73ce620b5a21.choria.local
Inventory for 73ce620b5a21.choria.local

  Choria Server Statistics:

                    Version: 0.99.0.20221201 (upgradable during provisioning)
...

This will indicate if the JWT or custom build flags enable version upgrades and will be disabled by default, requiring the JWT was made with --upgrade option.

Restarting Servers

A fleet that is capable of provisioning will allow you to restart them over Choria RPC using CLI or other Client:

$ choria req choria_provision restart token=s3cret --batch 100 --batch-sleep 30

In this case the token is the same as supplied when creating the JWT in the --token argument. You can use any discovery filters and other options as shown here.

Servers will restart and continue as normal after a randomized sleep up to 10 seconds.

Initiating Provisioning

Once a fleet of nodes is deployed using Provisioner they will not again update their configuration unless instructed to or if they encounter a problem that might be fixed by provisioning again such as credentials expiring.

You can though initiate provisioning ad-hoc using the Choria CLI or other Client:

$ choria req choria_provision reprovision token=s3cret --batch 100 --batch-sleep 30

In this case the token is the same as supplied when creating the JWT in the --token argument. You can use any discovery filters and other options as shown here.

Servers will restart into provisioning mode after a randomized sleep up to 10 seconds.

Shutting down Servers

When provisioning is enabled the servers can be shut down remotely, this will trigger a exit code 0 shutdown that will not trigger a systemd restart.

This should be used with great caution, since, once shut down the machines will not return without manual intervention via reboot or service restart. Treat this like a Big Red Button to emergency shut down the service.

$ choria req choria_provision shutdown token=s3cret --batch 100 --batch-sleep 30

In this case the token is the same as supplied when creating the JWT in the --token argument. You can use any discovery filters and other options as shown here.

Servers will shut down after a short delay.