yutil

Common functionality for working with YAML files

Table of contents

• yutil
  • Table of contents
  • Features
  • Getting started
    • Installation
      • deb/rpm/apk:
      • Manual
      • Go users
    • Test installation
    • Quick Start
      • Format
      • Merge
      • Replace
      • External configuration
  • Development
  • Release Process
  • CHANGELOG
  • License

Features

• Format yaml files
• Merge yaml files
• Replace files in a directory with a template engine (golang or jinja2) with the replacements of one or more yaml files.

Getting started

Installation

Install yutil with your preferred method:

deb/rpm/apk:

Download the .deb, .rpm or .apk from the latest release and install them with the appropriate tools.

Manual

Download the .tar.gz from the latest release and add the binary to your path.

Go users

go install github.com/amplia-iiot/yutil@latest

Test installation

yutil version

Quick Start

yutil help

Format

This outputs a formatted YAML file or files. That includes sorting its nodes alphabetically and cleaning the format of the values:

• Strings that do not need quotes to remain a primitive string lose the quotes. When quotes are needed, single quotes are preferred for strings with special characters. For strings containing a number, boolean or null values, double quotes are used. Unicode escape sequences in a string are replaced with the character.
• The proper formatting for null is null, not Null. The same happens to boolean values, lowercase is used when formatting.
• Arrays maintain the order of elements, and each element appears on a new line.
• Comments are removed.

Check the tests for examples.

To see and format the content of a YAML file use:

yutil format file.yml

Use -o (--output) option if you want to output to a file instead of stdout.

yutil format file.yml -o file.formatted.yml

By default yutil uses stdin as YAML content if available:

cat file.yml | yutil format > file.formatted.yml

You may ignore this input (--no-input) if you can't control what's piped to yutil:

echo "this is not a yaml" | yutil --no-input format file.yml > file.formatted.yml

If you want to format a YAML file and store the result in the same file use in-place formatting with -i (--in-place). You may pass as many YAML files as desired:

yutil -i file.yml
yutil -i file1.yml file2.yml file3.yml

Use -s (--suffix) to make a backup before formatting the file/s:

yutil -s .bak file1.yml file2.yml file3.yml

This will make a copy the files (file1.yml.bak, file2.yml.bak and file3.yml.bak) and save the formatted YAML in the original files.

When using -s, in-place formatting is implicit and there's no need to also use -i.

In-place formatting does not allow for stdin to be used as input, if something is piped to yutil an error will be displayed. Use --no-input to ignore stdin input.

Merge

This outputs a formatted (ordered and cleaned) YAML file resulting of merging the passed yaml files (or content).

The files are merged in ascending level of importance in the hierarchy. A yaml node in the last file replaces values in any previous file. You may pass as many YAML files as desired:

yutil merge base.yml changes.yml
yutil merge base.yml changes.yml important.yml

Use -o (--output) option if you want to output to a file instead of stdout.

yutil merge base.yml changes.yml -o merged.yml

By default yutil uses stdin as the first YAML content:

cat base.yml | yutil merge changes.yml > merged.yml

You may ignore this input (--no-input) if you can't control what's piped to yutil:

echo "this is not a yaml" | yutil --no-input merge base.yml changes.yml

Replace

This searches files and passes them through a template engine using the replacement files as variables (multiple replacement files will be merged in ascending level of importance in the hierarchy).

Available template engines:

• Golang (default): Unless configured otherwise, uses each *.tmpl and *.tmpl.* file as go text/template with slim-sprig functions, writing the result to a file with the same name but without that extension. A file named test.tmpl.txt will result in test.txt
• Jinja2: Unless configured otherwise, uses each *.j2 and *.j2.* file as jinja2 template, writing the result to a file with the same name but without that extension. A file named test.j2.txt will result in test.txt

If no engine is picked the default golang template with slim-sprig functions will be used.

yutil replace -r config.yml
yutil replace -r base.yml -r changes.yml

To use the jinja2 engine:

yutil replace --jinja2

By default yutil uses stdin as the first YAML replacement file:

cat changes.yml | yutil replace
cat base.yml | yutil replace -r changes.yml
cat base.yml | yutil replace -r changes.yml -r important.yml

The extension config is used for including those files by default (unless include config is passed) and renaming replaced files accordingly. Use include and exclude to filter files (the glob patterns are tested with both file name and file path):

yutil replace -r config.yml -e .go -e .gotempl
# This command uses the go template engine on files with both .go and .gotempl extensions including those files.
yutil replace -r config.yml -e .go -e .gotempl --include '*.go.*'
# With the previous command .gotempl files are not used because the include filter has been overriden and allows only files with .go. in its name or path.
yutil replace -r config.yml --include 'directory/*.conf'
yutil replace -r config.yml --jinja2 --exclude '*/secret/*'

By default only the files inside the current directory and its subdirectories are passed through the template engine. Use the directory config to change it:

yutil replace -r base.yml -r changes.yml -d directory

Normally, all replacements inside the replacement files are passed, to use a subset of variables the root node can be adjusted:

yutil replace -n root_node
# Only nodes inside root_node are directly available, without root_node as first element

To include environment variables as variables for the template engine use the env flag:

yutil replace --env
# Env vars are available inside the env node

External configuration

You may want to always use the same config without writting the flags, yutil reads a YAML file to configure itself from the current folder or the user home dir in these order of precedence:

• .yutil.yaml in current folder
• .yutil.yml in current folder
• .yutil in current folder
• .yutil.yaml in user home dir
• .yutil.yml in user home dir
• .yutil in user home dir

Sample configuration file:

# Disable stdin
no-input: true
# Merge specific config
merge:
  # Merge output file
  output: /tmp/merged.yml

You may pass as argument the desired config file:

# Including extension to support multiple config types
./yutil version --config config.properties

Supported formats: JSON, TOML, YAML, HCL, envfile and Java properties config files

Development

1. Use Golang version >= 1.22
2. Fork (https://github.com/amplia-iiot/yutil)
3. Run make set-up to install dev tools
4. Create a feature branch
5. Check changes (test & lint) with make check
6. Commit your changes following Conventional Commits
7. Rebase your local changes against the upstream main branch
8. Create a Pull Request

You are welcome to report bugs or add feature requests and comments in issues.

Release Process

make version contains the steps to generate a new version. It uses svu to calculate the next version number based on the git log and generates the CHANGELOG.md with git-chglog

Push the generated tag and the release workflow will generate the release.

Or you can trigger a manual run of the release workflow with preview set to false to generate the commit and tag automatically.

CHANGELOG

See CHANGELOG.md

License

MIT © amplia-iiot