In a recent blog post, I explained how one could set up a modern-day engineer’s command center on Ubuntu using cloud-init. While writing and testing the cloud-init configuration, I came across few issues that warranted me to write up a post on how to troubleshoot the cloud-init problems. This blog post describes:
- Use of syntax checkers to validate cloud-init configuration.
- Log files generated by cloud-init.
- Sample errors
Validating cloud-init configuration
cloud-init configuration files are written in YAML format. YAML is a user-friendly data format for various programming languages. YAML based syntax is commonly used for configuration files.
Validating the syntax of cloud-init configuration using an online YAML validator is one of the best things to do to avoid any issues related to the YAML syntax. I use http://yamllint.com/. It’s free to use and easy to use. You enter the configuration in the form and click on Go. Once the input has been processed by the website, it will show you errors (if any) and highlight the line numbers on which errors were found.
The image below shows an example from yamllint website where line number 23 has a single leading space instead of being double-spaced.
Log files are my best friends when it comes to troubleshooting. cloud-init does a fantastic job here by providing two log files that are detailed and are quite helpful.
The /var/log/cloud-init.log file contains details on the complete cloud-init process and is an ideal location to view any YAML syntax related issues.
The output from all cloud-init stages are logged under /var/log/cloud-init-output.log. This log contains both errors and success messages. If a download didn’t work or the package manager didn’t install a package, you will see the messages here.
Let’s look at few errors in this section of the blog post.
Error 1 – Watch for spacing and syntax
yaml.parser.ParserError: while parsing a block collection in "<unicode string>", line 21, column 3: - [pip, install, -U, pip, ansibl ... ^ expected <block end>, but found '<block sequence start>' in "<unicode string>", line 26, column 4: - [curl, -o, /tmp/gclisdk.tar.gz ... ^
The error shown above was due to incorrect formatting as the line had single space instead of double space. Single space marks the start of a block while double space marks the config inside a block.
Error 2 – Watch out for special characters
yaml.scanner.ScannerError: while scanning for the next token found character '|' that cannot start any token in "<unicode string>", line 27, column 95: ... epos/azure-cli/ $AZ_REPO main", |, tee, /etc/apt/sources.list.d/ ^
Special characters need to be escaped within a single quote. If the | character would have been written as ‘|’, there wouldn’t have been an issue.
Error 3 – Watch out for missing quotes
"Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.") yaml.scanner.ScannerError: while scanning a plain scalar in "<unicode string>", line 28, column 17: - [ curl, -L, https://packages.microsoft.com/k ... ^ found unexpected ':' in "<unicode string>", line 28, column 22: - [ curl, -L, https://packages.microsoft.com/keys/m ...
When using URLs, it’s always safe to have them enclosed in double quotes. E.g.:
Error 4 – Misconfigured options in commands
deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main | tee /etc/apt/sources.list.d/azure-cli.list curl: option -: is unknown curl: try 'curl --help' or 'curl --manual' for more information
The best way to do this is to try the commands manually on your server and then use them in cloud-init configuration.
Error 5 – unexpected error when running command
Sometimes you might encounter the “unexpected error when running command” error, which would show in the cloud-init.log file.
cloudinit.util.ProcessExecutionError: Unexpected error while running command. Command: ['/var/lib/cloud/instance/scripts/runcmd'] Exit code: 2 Reason: - Stdout: - Stderr: -
/var/lib/cloud/instance/scripts/runcmd: 5: /var/lib/cloud/instance/scripts/runcmd: Syntax error: redirection unexpected
This error shows in cloud-init on Ubuntu because /bin/sh is a symbolic link to dash and not bash. dash does not know about ‘<<<‘ as that is not POSIX.
Instead of writing runcmd as
debconf-set-selections <<< "mariadb-server-10.0 mysql-server/root_password password $MDBPASS"
echo "mariadb-server-10.0 mysql-server/root_password password $MDBPASS" | debconf-set-selections