Transformations

Transformations are like a shell pipe defined in App Builder. A number of transformations are available, and using them is entirely optional - often a shell pipe would be much better.

The reason for adding transformations like jq to App Builder itself is to have it function in places where that 3rd party dependency is not met. Rather than require everyone to install JQ - and handle that dependency, App Builder includes a JQ dialect directly.

A basic example of transformations can be seen here:

name: ghd
description: Gets the description of a Github Repo
type: exec
command: |
  curl -s -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/choria-io/appbuilder

transform:
  jq:
    query: .description

Here we call out a REST API that returns JSON payload using curl and then extract the description field from the result using a JQ transform

$ demo ghd
Tool to create friendly wrapping command lines over operations tools

Not every command supports transforms, so the individual command documentation will call out it out.

JQ Transform

The jq transform uses a dialect of JQ called GoJQ, most of your JQ knowledge is transferable with only slight changes/additions. This is probably the most used transform so there is a shortcut to make using it easier:

transform:
  jq:
    query: .description

The query supports Templating.

Since version 0.5.0 an optional yaml_input boolean can be set true to allow YAML input to be processed using JQ.

To JSON Transform

The to_json transform can convert YAML or JSON format input into JSON format output. By default the output JSON will be compact unindented JSON, prefix and indent strings can be configured.

# unindented JSON output
transform:
  to_json: {}

# Indented JSON output with a custom prefix
transform:
  to_json:
    indent: "  "
    prefix: "  "

To YAML Transform

The to_yaml transform can convert JSON format input into YAML format output.

transform:
  to_yaml: {}

The to_yaml transform has no options.

Bar Graph Transform

This transform takes a JSON document like {"x": 1, "y": 2} as input and renders bars for the values.

Here is an example that draws the sizes of the assets of the latest release:

name: bargraph
description: Draws an ASCII bar graph
type: exec
transform:
  pipeline:
    - jq:
        query: |
          .assets|map({(.name): .size})|reduce .[] as $a ({}; . + $a)

    - bar_graph:
        caption: "Release asset sizes"
        bytes: true
script: |
  curl -s https://api.github.com/repos/choria-io/appbuilder/releases/latest

This uses a pipeline (see below) to transform a GitHub API request into a hash and then a bar_graph to render it:

$ demo bargraph
Latest release asset sizes

    appbuilder-0.1.1-windows-arm64.zip: ▏ (2.0 MiB)
            appbuilder-0.1.1-arm64.rpm: ██ (2.0 MiB)
   appbuilder-0.1.1-linux-arm64.tar.gz: ███ (2.0 MiB)
            appbuilder-0.1.1-arm64.deb: ███ (2.0 MiB)
     appbuilder-0.1.1-windows-arm7.zip: █████████████ (2.1 MiB)
     appbuilder-0.1.1-windows-arm6.zip: ██████████████ (2.1 MiB)
             appbuilder-0.1.1-arm7.rpm: ██████████████ (2.1 MiB)
             appbuilder-0.1.1-arm7.deb: ███████████████ (2.1 MiB)
    appbuilder-0.1.1-linux-arm7.tar.gz: ███████████████ (2.1 MiB)
             appbuilder-0.1.1-arm6.rpm: ███████████████ (2.1 MiB)
             appbuilder-0.1.1-arm6.deb: ███████████████ (2.1 MiB)
    appbuilder-0.1.1-linux-arm6.tar.gz: ███████████████ (2.1 MiB)
    appbuilder-0.1.1-windows-amd64.zip: ███████████████████████ (2.2 MiB)
  appbuilder-0.1.1-darwin-arm64.tar.gz: █████████████████████████ (2.2 MiB)
           appbuilder-0.1.1-x86_64.rpm: ███████████████████████████ (2.2 MiB)
   appbuilder-0.1.1-linux-amd64.tar.gz: ███████████████████████████ (2.2 MiB)
            appbuilder-0.1.1-amd64.deb: ███████████████████████████ (2.2 MiB)
  appbuilder-0.1.1-darwin-amd64.tar.gz: ████████████████████████████████████████ (2.3 MiB)

The transform supports a few options, all are optional:

Option	Description
`width`	The width of the bar, defaults to 40
`caption`	The caption to show above the graph, supports Templating
`bytes`	When set to true indicates that the numbers rendered after the bars will be bytes like in the example

Line Graph

This transform takes input of floats per line or a JSON document (array of floats) and turns it into a line graph.

Here we find the hourly forecast for our location and graph it:

description: Hourly weather forecast
type: exec
transform:
  pipeline:
    - jq:
        query: |
          .weather[0].hourly|.[]|.FeelsLikeC
    - line_graph:
        width: 40
        height: 10
        caption: Hourly weather forecast (C)
command: |
  curl -s wttr.in/?format=j1

When run this produces:

$ demo linegraph
 30.00 ┤                      ╭─────╮
 29.90 ┤                     ╭╯     │
 29.80 ┤                    ╭╯      ╰╮
 29.70 ┤                    │        │
 29.60 ┤                   ╭╯        ╰╮
 29.50 ┤                   │          │
 29.40 ┤                  ╭╯          ╰╮
 29.30 ┤                  │            ╰╮
 29.20 ┤                 ╭╯             │
 29.10 ┤                ╭╯              ╰╮
 29.00 ┼────────────────╯                ╰─────
              Hourly weather forecast (C)

The transform supports a few options, all are optional:

Option	Description
`width`	The width of the graph, defaults to 40
`height`	The height of the graph, defaults to 20
`precision`	The decimal precision to consider and render
`json`	When true expects JSON input like `[1,2,3,4]` rather than a float per line
`caption`	The caption to show below the graph, supports Templating

Templates

The template transform uses Golang templates and the sprig functions to facilitate creation of text output using a template language.

name: template
type: exec
description: Demonstrates template processing of JSON input
command: |
  echo '{"name": "James", "surname":"Bond"}'
transform:
  template:
    contents: |
      Hello {{ .Input.name }} {{ .Input.surname | swapcase }}

Option	Description
`contents`	The body of the template embedded in the application yaml file, mutually exclusive with `source`
`source`	The file name holding the template, the file name is parsed using Templating. Mutually exclusive with `contents`

Writing to a file

Data entering the write_file transform is written to disk and also returned, but optionally a message can be returned.

name: template
type: exec
description: Demonstrates template processing of JSON input
command: |
  echo '{"name": "James", "surname":"Bond"}'
transform:
  pipeline:
    - write_file:
        file: /tmp/name.txt
        replace: true

    - template:
      contents: |
        Hello {{ .Input.name }} {{ .Input.surname | swapcase }}

Above the /tmp/name.txt would hold the initial JSON data.

If the write_file is the only transform or in a pipeline like here the data received is simply passed on to the next step, this can be annoying when writing large files as they will be dumped to the screen.

transform:
  write_file:
    file: /tmp/report.txt
    replace: true
    message: Wrote {{.IBytes}} to {{.Target}}

In this case the message Wrote 1.8 KiB to /tmp/report.txt would be printed. You can use .Bytes, .IBytes, .Target and .Contents in the message.

Option	Description
`file`	The file to write, the file name is parsed using Templating
`message`	A message to emit from the transform instead of the contents received by it
`replace`	Set to `true` to always overwrite the file

Row orientated Reports

These reports allow you to produce text reports for data found in JSON files. It reports on Array data and produce paginated reports with optional headers and footers.

name: report
type: exec
description: Demonstrates using a report writer transform
command: curl -s https://api.github.com/repos/choria-io/appbuilder/releases/latest
transform:
  report:
    name: Asset Report
    initial_query: assets
    header: |+
      @|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
      data.name
      --------------------------------------------------------------------------------

    body: |
      Name: @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Size: @B###### Downloads: @##
      row.name, row.size,              row.download_count
    footer: |+2
      
                                                                  ====================
                                                                  Total Downloads: @##
      report.summary.download_count

Here we fetch the latest release information from GitHub and produce a report with header, footer and body. Since the JSON data from GitHub is a object we use the assets GJSON query to find the rows of data to report on.

See the goform project for a full reference to the formatting language.

                                 Release 0.2.1                                  
--------------------------------------------------------------------------------

Name: appbuilder-0.2.1-amd64.deb                   Size: 2.3 MiB  Downloads: 2  
Name: appbuilder-0.2.1-arm6.deb                    Size: 2.2 MiB  Downloads: 0  
Name: appbuilder-0.2.1-arm6.rpm                    Size: 2.1 MiB  Downloads: 0  
....
Name: appbuilder-0.2.1-windows-arm7.zip            Size: 2.1 MiB  Downloads: 0  
Name: appbuilder-0.2.1-x86_64.rpm                  Size: 2.2 MiB  Downloads: 2    

                                                            ====================
                                                            Total Downloads: 20

Option	Description
`name`	The name of the report, parsed using Templating
`header`	The report header
`body`	The report body
`footer`	The report footer
`rows_per_page`	How many rows to print per page, pages each have `header` and `footer`
`initial_query`	The initial GJSON query to use to find the row orientated data to report
`source_file`	A file holding the report rather than inline, `name`, `header`, `body` and `footer` are read from here. File name parsed using Templating

Scaffold

The scaffold transform takes JSON data and can generate multiple files using that output.

This is essentially the Scaffold Command in transform form. The Command documentation provides full details on the underlying feature. This section covers only what makes the transform unique.

Version Hint

This was added in version 0.9.0

Option	Description
`target`	The directory to write the data into
`source_directory`	The directory where the template files can be found, cannot be used with `source`
`source`	Map holding file names and content, if a value is another object a directory is created instead
`post`	Post processing directives
`skip_empty`	Skips files that would be empty when written
`left_delimiter`	Custom template delimiter
`right_delimiter`	Custom template delimiter

These settings all correspond to the same ones in the command so we won’t cover them in full detail here.

The scaffold transform returns the input JSON on its output.

Pipelines

Several example transform pipelines appear above, like this one:

type: exec
transform:
  pipeline:
    - jq:
        query: |
          .weather[0].hourly|.[]|.FeelsLikeC
    - line_graph:
        width: 80
        caption: Hourly weather forecast (C)
command: |
  curl -s wttr.in/?format=j1

This runs the output of the curl command (JSON weather forecast data) through a jq transform that produces results like:

That data is then fed into a line_graph and rendered; the output from the jq transform is used as input to the line_graph.

Any failure in the pipeline will terminate processing.

CCM Manifest

The ccm_manifest transform executes a Choria Config Manager manifest using the input data as manifest data. Arguments and flags are merged into the manifest data along with any JSON input.

The manifest URL is parsed using Templating.

transform:
  ccm_manifest:
    manifest: https://example.net/manifests/deploy.yaml
    nats_context: CCM
    render_summary: true

Option	Description
`manifest`	The manifest URL to execute, required
`nats_context`	The NATS context to use for connections, defaults to `CCM`
`render_summary`	When true renders a text summary of the session to STDOUT
`no_render_messages`	When true suppresses pre and post manifest messages

When render_summary is false the transform outputs the session summary as JSON for further processing.