In a recent blog post, I explained how one could set up a modern-day engineer’s command center on Ubuntu using cloud-init. While writing and testing the cloud-init configuration, I came across few issues that warranted me to write up a post on how to troubleshoot the cloud-init problems. This blog post describes:

  1. Use of syntax checkers to validate cloud-init configuration.
  2. Log files generated by cloud-init.
  3. Sample errors

Validating cloud-init configuration

cloud-init configuration files are written in YAML format. YAML is a user-friendly data format for various programming languages. YAML based syntax is commonly used for configuration files.

Validating the syntax of cloud-init configuration using an online YAML validator is one of the best things to do to avoid any issues related to the YAML syntax.  I use http://yamllint.com/. It’s free to use and easy to use. You enter the configuration in the form and click on Go. Once the input has been processed by the website, it will show you errors (if any) and highlight the line numbers on which errors were found.

The image below shows an example from yamllint website where line number 23 has a single leading space instead of being double-spaced.

yaml-validator

cloud-init logs

Log files are my best friends when it comes to troubleshooting. cloud-init does a fantastic job here by providing two log files that are detailed and are quite helpful.

/var/log/cloud-init.log

The /var/log/cloud-init.log file contains details on the complete cloud-init process and is an ideal location to view any YAML syntax related issues.

/var/log/cloud-init-output.log

The output from all cloud-init stages are logged under /var/log/cloud-init-output.log. This log contains both errors and success messages. If a download didn’t work or the package manager didn’t install a package, you will see the messages here.

Sample Errors

Let’s look at few errors in this section of the blog post.

Error 1 – Watch for spacing and syntax

yaml.parser.ParserError: while parsing a block collection
in "<unicode string>", line 21, column 3:
- [pip, install, -U, pip, ansibl ...
^
expected <block end>, but found '<block sequence start>'
in "<unicode string>", line 26, column 4:
- [curl, -o, /tmp/gclisdk.tar.gz ...
^

The error shown above was due to incorrect formatting as the line had single space instead of double space. Single space marks the start of a block while double space marks the config inside a block.

Error 2 – Watch out for special characters

yaml.scanner.ScannerError: while scanning for the next token
found character '|' that cannot start any token
in "<unicode string>", line 27, column 95:
... epos/azure-cli/ $AZ_REPO main", |, tee, /etc/apt/sources.list.d/ 
                                    ^

Special characters need to be escaped within a single quote. If the | character would have been written as ‘|’, there wouldn’t have been an issue.

Error 3 – Watch out for missing quotes

"Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.")
yaml.scanner.ScannerError: while scanning a plain scalar
in "<unicode string>", line 28, column 17:
- [ curl, -L, https://packages.microsoft.com/k ...
                   ^
found unexpected ':'
in "<unicode string>", line 28, column 22:
- [ curl, -L, https://packages.microsoft.com/keys/m ...

When using URLs, it’s always safe to have them enclosed in double quotes. E.g.:

"https://packages.microsoft.com"

instead of

http://packages.microsoft.com

Error 4 – Misconfigured options in commands

deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main | tee /etc/apt/sources.list.d/azure-cli.list
curl: option -: is unknown
curl: try 'curl --help' or 'curl --manual' for more information

The best way to do this is to try the commands manually on your server and then use them in cloud-init configuration.

Error 5 – unexpected error when running command

Sometimes you might encounter the “unexpected error when running command” error, which would show in the cloud-init.log file.

cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['/var/lib/cloud/instance/scripts/runcmd']
Exit code: 2
Reason: -
Stdout: -
Stderr: -

cloud-init-output.log shows

/var/lib/cloud/instance/scripts/runcmd: 5: /var/lib/cloud/instance/scripts/runcmd: Syntax error: redirection unexpected

This error shows in cloud-init on Ubuntu because /bin/sh is a symbolic link to dash and not bash. dash does not know about ‘<<<‘ as that is not POSIX.

Instead of writing runcmd as

debconf-set-selections <<< "mariadb-server-10.0 mysql-server/root_password password $MDBPASS"

write as

echo "mariadb-server-10.0 mysql-server/root_password password $MDBPASS" | debconf-set-selections

Happy troubleshooting!

print