Server Setup

The Choria Server has the required RPC Agent embedded and is ready to be used in a Provisioner managed network but it is disabled by default. Custom binaries can enable provisioning at compile time for an always-on experience.

Provisioning is enabled in the Open Source server by means of a JWT token that you create and place on the server. The JWT token holds all of the information the server needs to find it’s provisioning server and will present that token also to the provisioning server for authentication.

The token is signed using a trusted private key, the provisioner will only provision nodes presenting a trusted key.

Credentials

If you are using a x509 based setup you really just need any RSA Key pair to sign the JWT with:

$ openssl genrsa -out provisioning-jwt-signer-key.pem 2048
$ openssl rsa -in provisioning-jwt-signer-key.pem -outform PEM -pubout -out provisioning-jwt-signer.pem

The public file will be placed on all your brokers to enable provisioning.

For the Organization Issuer based deploys you would sign it using your Issuer.

Creating the JWT

Choria JWTs are creating using the choria jwt command, for provisioning specifically choria jwt provisioning.

Creating the JWT:

$ choria jwt provisioning.jwt provisioning-jwt-signer-key.pem --srv choria.example.net --token toomanysecrets
Saved token to provisioning.jwt, use 'choria jwt view provisioning.jwt' to view it

$ choria jwt provisioning.jwt
Unvalidated Provisioning Token x.jwt

                         Token: *****
                        Secure: true
                    SRV Domain: choria.example.net
       Provisioning by default: false
               Standard Claims: {
                                  "purpose": "choria_provisioning",
                                  "iss": "Choria Tokens Package v0.26.2",
                                  "sub": "choria_provisioning",
                                  "nbf": 1669805752,
                                  "iat": 1669805752,
                                  "jti": "2c99227346f641bbba34faf0a6991d05"
                                }

Here we create a provisioning.jwt that will instruct Choria to look for _choria-provisioner._tcp.choria.example.net SRV records to find the server to connect to.

Other options can be set for example to hard code provisioning URLs, username and passwords and more.

When this file is placed in /etc/choria/provisioning.jwt and Choria starts without a configuration it will provision via these settings.

Choria also support provisioning plugins to resolve this information dynamically but this requires custom binaries and should in general be avoided.

Confirming

Without the JWT in place Provisioning is not enabled:

$ choria buildinfo
...
Server Settings:

    Provisioning Target Resolver: Choria JWT Resolver
           Supports Provisioning: false
           Provisioning JWT file: /etc/choria/provisioning.jwt
...

We can see it will look for the /etc/choria/provisioning.jwt file, lets move our newly created file there and try again:

$ sudo msg provisioning.jwt /etc/choria/provisioning.jwt
$ choria buildinfo
...
Server Settings:

    Provisioning Target Resolver: Choria JWT Resolver
           Supports Provisioning: true
           Provisioning JWT file: /etc/choria/provisioning.jwt
              Provisioning Token: *****
            Provisioning Default: false
                Provisioning TLS: true
      Default Provisioning Agent: true
         Provisioning SRV Domain: choria.example.net
...

Now provisioning is on with the settings we provided in the token.

Broker Setup

Choria Broker supports a concept called an Account or Organization, these are like a VLAN on your network switches.

Nodes partitioned into one Organization cannot communicate with nodes in any other Organizations unless special arrangements are made to facilitate that.

We use this feature to create an isolated network where unprovisioned servers wait for provisioning. When Choria Provisioner connects using fully verified credentials it gains access to these isolated Servers and can provision them.

Here we show one Choria Broker Cluster with a Choria Account and a Provisioning Account active. Unverified connections enter the Provisioning Account automatically. The Provisioner connects with its fully verified credentials that identify it as a Provisioner and gains access to the unprovisioned servers.

Restrictions in the Choria Broker authentication layer ensures this separation is not optional and enforce control over what software gains access to the unprovisioned fleet.

Enabling the System Account

We suggest enabling the Choria Broker system account to allow introspection of nodes waiting to be provisioned and more.

Configure this in both your Broker and Client configurations.

plugin.choria.network.system.user = system
plugin.choria.network.system.password = system

Enabling the Provisioning Account

The Choria Provisioner will connect using its normal Certificate or JWT token which means it will appear just like any other client. To differentiate it we force it to connect to a specific user and password.

The password is unique per environment and should be configured in the broker:

plugin.choria.network.provisioning.client_password = provSecret

x509 Based Choria

If you are using an x509 based network you have to set the Public certificate that was used to configure servers.

plugin.choria.network.provisioning.signer_cert = /etc/choria/provisioning-jwt-signer.pem

Organization Based Choria

If you are using an Organization Issuer you should already have the issuer configured and have used that issuer to sign the provisioning.jwt

plugin.security.issuer.names = choria
plugin.security.issuer.choria.public = 4ada2...cea4

Confirming

Once configured you can run the command choria broker server report conns --account provisioning to both confirm the account is active and it will show any connections made to this account. The Choria Provisioner will appear there as well as any machines being provisioned.

Writing a Helper

The Helper is a script or program written in any language that is in charge of handling the steps that are unique to your environment.

Essentially it receives Server Metadata on its STDIN and writes a response to STDOUT that is used to configure the server.

See the sections below for sample Helper scripts.

Input

Let’s look at what you might receive as input on STDIN. The specifics will vary a bit between scenarios which we will call out in specific sections. The data you will get will all already have been validated. For example the JWT would have been parsed already and known to be valid.

{
  "identity": "24bd22cdb279.choria.local",
  "csr": null,
  "ed25519_pubkey": {
    "public_key": "2ddc906446a935aefde69ceee2beb3b1d85f264153720d9a752aa88771c0594c",
    "directory": "/etc/choria",
    "signature": "34c5c4d0564747b1d0577ddc9c4084de547235d56e2803a754c76ba4eabbf17e31453ebd9ca4b4a7dd5e25ccf57afd4138519de4666c81d3ee3cd5381cc15a0a"
  },
  "inventory": "{\"agents\":[\"choria_provision\",\"choria_util\",\"discovery\",\"rpcutil\"],\"classes\":[],\"collectives\":[\"provisioning\"],\"data_plugins\":[],\"facts\":{},\"machines\":[],\"main_collective\":\"provisioning\",\"version\":\"0.99.0.20221129\",\"upgradable\":true}",
  "jwt": {
    "cht": "s3cret",
    "chs": false,
    "chu": "nats://broker.choria.local:4222",
    "chpd": true,
    "extensions": null,
    "ou": "choria",
    "v2": true,
    "purpose": "choria_provisioning",
    "iss": "Choria Tokens Package v0.26.2",
    "sub": "choria_provisioning",
    "nbf": 1669809657,
    "iat": 1669809657,
    "jti": "3a4f9896e013498daeedbcb9a82fcd3c"
  }
}

Output

The response your helper should write to STDOUT is also in JSON format.

Here is an example that configures Choria Server and uses the jwt feature to enroll a node into an Organization Issuer based network.

{
  "defer": false,
  "msg": "Done",
  "certificate": "",
  "ca": "",
  "configuration": {
    "identity": "eb873ce040d7.choria.local",
    "loglevel": "info",
    "plugin.choria.server.provision": "false",
    "plugin.choria.middleware_hosts": "nats://broker.choria.local:4222",
    "rpcauthorization": "0",
    "plugin.choria.status_file_path": "/var/log/choria-status.json",
    "plugin.choria.submission.spool": "/var/lib/choria/submission",
    "plugin.security.issuer.names": "choria",
    "plugin.security.issuer.choria.public": "e72cba5268b34627b75c5ceae9449ad16d62f15f862c30d4e0e7d2588e2e6259",
    "plugin.security.provider": "choria",
    "plugin.security.choria.token_file": "/etc/choria/server.jwt",
    "plugin.security.choria.seed_file": "/etc/choria/server.seed",
    "plugin.choria.machine.store": "/etc/choria/machine"
  },
  "server_claims": {
    "exp": 157680000,
    "permissions": {
      "streams": true,
      "submission": true
    }
  }
}

Response keys used by scenarios:

When using the pki feature used to enroll with a Certificate Authority:

When using the jwt feature to create server JWT tokens for Organization Issuer based networks:

When using the upgrades feature to in-place upgrade servers:

Enrolling nodes with a Certificate Authority

Most typically you have a Enterprise Certificate Authority or you made your own using something like cfssl.

In this mode Choria Server will generate a private key on its disk, create a CSR and your helper will receive the CSR in PEM format. In your helper you then simply interact with your CA to sign the CSR and respond with the signed Certificate and CA public key. This way once provisioned your server will be fully enrolled for mTLS.

To enable the Provisioner to request the CSR the pki feature needs to be enabled in the Provisioner Configuration.

The csr input field will then be a JSON Object with:

Once you have this data you can use your CA API to enroll the node and get a signed certificate back. Simply put the resulting PEM data in the certificateand ca keys in the reply. You can set a SSL directory but typically just set ssldir to what was received in the input.

In general the Private key stays on the node and you do not need it. Some Certificate Authorities require the private key to be accessible when signing a request. Provisioner support that, if you generate a key in the provisioner and add it to the reply in the key JSON field a single use Shared Secret negotiated using Diffie-Hellman will be used to encrypt the key in transit.

I would not suggest ever to use a CA that requires you to transmit the Private Key during enrollment, it’s best to assume your CA is unusable at that point and consider a Organization Issuer based deployment.

Enrolling nodes with an Organization Issuer

In cases where a Certificate Authority is not available or it is operated in a way that makes it unsuitable for mTLS use you might opt to deploy Choria in an Organization Issuer based setup. The basic setup of that mode is out of scope for this document.

To enrol nodes in an Organization Issuer based network you need to enable the ed25519 feature in the Provisioner Configuration.

Once enabled the ed25519_pubkey field will hold a JSON Object with these values:

Once you received these you can include a server_claims in your reply to give the server access to specific features in its JWT claims. The list here is correct for 0.27.0. For an up-to-date list see the Go Documentation for your version of Choria.

The claims can include permissions that have these properties.

You can also use them to restrict it to a specific sub collective and more. Most values will default to sane defaults when not given.

Upgrading Servers

Choria Server can be upgraded in-place to a new version. This is done by overwriting the binary at run-time with one downloaded from a specifically prepared repository.

This can only be done during Provisioning and requires the upgrades feature to be enabled.

Compatible nodes will have the upgradable key in the inventory received and will be set to true when upgrades are enabled. If you only want to support the latest nodes you can use this to determine if a upgrade is needed along with the version key in the inventory. On nodes that are too old set shutdown or defer.

You would have to configure the Provisioner with a repository location and set up a repository on a HTTP server as per the guidelines from go-updater.

With all of this in place you can add the upgrade key to the helper response that should just be a desired version like 0.28.0. Provisioner will then attempt to upgrade the node.

Configuration File

Provisioner is configured using /etc/choria-provisioner/choria-provisioner.yaml typically. It’s a YAML format file with a few required settings and a number of optional ones. Changes to the file requires the process to be restarted.

Choria Client Configuration

As the Provisioner connects to the Choria Broker as a client it needs a configuration that allows it access. Create /etc/choria-provisioner/choria.cfg with the following based on needs. These settings will augment those in /etc/choria/client.cfg.

x509 based Networks

plugin.security.provider = file
plugin.security.file.certificate = /etc/choria-provisioner/ssl/cert.pem
plugin.security.file.key = /etc/choria-provisioner/ssl/key.pem
plugin.security.file.ca = /etc/choria-provisioner/ssl/ca.pem

Obtain these certificates the same way you would obtain any other certificate, perhaps using choria enroll --certname provisioner.mcollective.

Organization Issuer Based Networks

In these setups you need a client JWT with --server-provisioner and --issuer set while creating the client JWT:

$ choria jwt keys /etc/choria-provisioner/signer.seed /etc/choria-provisioner/signer.public
$ choria jwt client /etc/choria-provisioner/signer.jwt provisioner_signer issuer \
     --public-key $(cat /etc/choria-provisioner/signer.public) \ 
     --server-provisioner \
     --validity 365d \
     --vault \
     --issuer

The /etc/choria-provisioner/choria.cfg would then have:

plugin.security.provider = choria
plugin.security.choria.token_file = /etc/choria-provisioner/credentials/signer.jwt
plugin.security.choria.seed_file = /etc/choria-provisioner/credentials/signer.seed
identity = provisioner_signer

Common Settings

These settings are required for all scenarios, below an example configuration followed by explanation:

workers: 4
interval: 1m
logfile: /dev/stdout
loglevel: warn
helper: /etc/choria-provisioner/provisioner/helper.rb
token: s3cret
site: testing
broker_provisioning_password: s3cret
jwt_verify_cert: e72cba5268b34627b75c5ceae9449ad16d62f15f862c30d4e0e7d2588e2e6259
jwt_signing_key: /etc/choria-provisioner/credentials/signer.seed
jwt_signing_token: /etc/choria-provisioner/credentials/signer.jwt

features:
  jwt: true
  ed25519: true

PKI / x509 Enrollment

When enabled the Provisioner will fetch a CSR from the node and ask the node to create a private key that stays on the node. The helper can then get the certificate signed and the signed certificate will be sent to the node.

To enable set features.pki to true.

Here we reference our x509 key and certificate

Organization Issuer based Enrollment

When enabled the Provisioner will sign and issue server JWTs with custom claims and signatures. No x509 steps will be done.

To enable set features.ed25519 to true.

Here we reference our JWT that gives us the right to provision and issue new JWTs

Choria Server Upgrades

Choria Provisioner can upgrade Choria Servers using a go-updated repository.

To enable set features.upgrades to true.

Here we configure the updates repository and say how failures are handled

Clustered Deployments

The Provisioner is generally fast enough to not need a cluster of active-active servers, so we support deploying multiple instances of Provisioner and using Leader Elections to elect one in the cluster as a leader that will be actively provisioning nodes.

To enable this the client needs access to Streams and Leader Elections so need --stream-user and --elections-user passed when creating its JWT.

Campaigning will be on a backoff schedule up to 20 second between campaigns, this means there can be up to a minute of downtime during a failover scenario, generally that’s fine for the Provisioner.

If a Provisioner was on standby and becomes leader it will immediately perform a discovery to pick up any nodes ready for provisioning.

Your broker must therefor have Choria Streams enabled.