SaltyVagrant8.5 Our core configuration tool
In the previous section we converted our initial configuration steps into a Bash script. This decouples our initial configuration from a Vagrant specific format (the Vagrantfile) and places it into a more portable form that can be used to provision not only Vagrant machines but also cloud servers or physical machines. This has the benefit of making our configuration something defined independent of the underlying implementation of our server.
We also showed the separation of the configuration from how to achieve that configuration. This is an important abstraction allowing us to focus on ‘what’ our system should look like rather than ‘how’ to make our system look the way we want. (This is obviously an ideal and reality being the complex mess it is seldom this clean cut in real life. But, hey, that’s what we’re here to learn about!)
The Bash script is certainly a move in the right direction and there will be many more such scripts required to set up our servers, but Bash scripts are tough to get right and are seldom concise in expressing all the minor variations required when configuring multiple servers. Fortunately there are specialised tools for managing our server configurations.
There are many tools to choose from in the configuration management space. Which you choose may depend on a number of factors.
- Does your team already have experience using a particular tool? This could result in a default decision based on current experience, the tool may be adopted because people are comfortable using it, or rejected because of past problems with the tool.
- Does your team have deep knowledge of a particular language? This can influence tool choice because particular tools are written using, or are designed to integrate with, specific languages. For example Puppet is Ruby based, while Saltstack is Python based.
- Cost.
- Supported platforms.
- Legacy configuration. Either a need to adopt existing configuration or to migrate from a legacy configuration.
We will use Saltstack for the following reasons:
- It is freely available.
- It is based on Python and Python is pretty much the defacto standard scripting language on Linux (and is preinstalled6 on most Debian installations, including this debian/bullseye64).
- It is so much more than a configuration management tool, it can be used for monitoring and self-healing of systems, features we will use much later in this course.
- It has tools for deploying cloud servers, which we will use later in this course.
- It can be used standalone, over SSH, or as a full bus-oriented master/minion system. These options are discussed briefly in later sections and more fully in Saltstack from Scratch[Boo20d].
8.5.1 Installing Salt
We have seen how trivial installing a Debian package can be when we installed Git. Debian repositories do have a set of Salt packages but they tend to be older versions of Salt and we would prefer to have a more recent version (and have the option to keep our systems up-to-date with the latest releases), so we will not be using the Debian package repository version.
Fortunately SaltStack provide a shell script for installing various SaltStack components. In a similar approach to that used in §8.3.2 we can provide the SaltStack script and run it to install the Salt components we require. We will start with a naive implementation and gradual refine it into a more robust implementation, along with discussion of each refinement.
Edit the Vagrantfile.
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "debian/bullseye64" 6 config.vm.box_version = "v11.20220912.1" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11 config.vm.provision "shell", path: "scripts/configure"" 12 config.vm.provision shell, inline: "cd /tmp && curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com && sh bootstrap-salt.sh -M git master" 13end
We are adding just one line (line 12) but it’s doing a lot of work. There are three commands to be run; change to the /tmp directory, download (curl) the script from the SaltStack website, and finally run that script.
As before we can run this additional provision line using the --provision option. On your host computer, in the same directory as the Vagrantfile.
1vagrant up --provision
This run will take some time as the Salt installation script does a lot of work for us.
While it works, let’s consider what we just did (and why it’s not a particularly good approach).
We downloaded a script from the internet and ran it into our VM without any checks. This is a security risk on two levels; the source of the script (the bootstrap.saltstack.com website) could have been compromised and the script could have been tampered with, secondly, we have no idea what the script is actually doing, where is it sourcing the installation from, what (if any) precautions are taken to ensure the script installs the proper files.
This level of trust may be okay for our ‘quick and dirty’ development environment, but they are unacceptable for a production environment.
In Chapter 15 we discuss repositories in more detail but for now we should note that sourcing directly from a public website is a security risk. Whether we consider it a reasonable risk comes down to our risk tolerance (see §12.1.1) and this will vary according to context (we may accept more risk in an isolated development environment but less in a live production environment).
The first step in reducing our risk is simple in principle, we download the script to a local file system, review it, and then use this reviewed copy as the source for our configuration. Simple in principle, more complex in practice. Performing such a review is a non-trivial exercise. It is important that this process results in a controlled copy of the script that can be used for delivery but also for comparison when updates to the upstream (original source) are made.
Furthermore, in this case the script itself refers to other repository objects and if we are to be highly risk averse these must also be vetted and local repositories used. Chapter 15 discussed these observations in more detail, for now we will set most of them aside in order to progress with our configuration work (rest assured though, we will return to this issue).
8.5.2 Additional Salt setup
The current Salt setup will not work!
Salt, as installed, is running a Master/Minion setup where each Minion controls a target configuration (in this case the new VM) and the Master coordinates a set of Minions. In order for Minions to find the appropriate Master they use a DNS lookup. By default the Minion will look for the domain name salt.
The Master and Minion are running as two services on the VM. We can view their current state using systemctl.
1systemctl status salt-master 2systemctl status salt-minion
You will see that the Minion has failed to start, this is because we have not yet configured a domain name salt, so the Minion will be unable to find the Master (even though it happens in this case to be the same VM).
To fix this we need to define a domain name salt that the Minion can find and resolve, and then we need to restart the Minion so that is can find and connect to the Master.
To fix this quickly we can add a line to our /etc/hosts file to resolve domain name salt to the VM itself. As the Master and Minion are running on the same VM we can use the local loopback device lo, this has the IP Address 127.0.0.1 (this IP Address is a standard ‘this machine’ IP address7). We could simply edit our /etc/hosts file, but this would be useless to anyone building this VM from our Vagrantfile in the future. Following the principals of infrastructure as code, this change must be written into our configuration. The modification to the /etc/hosts should exist before we install Salt so that it is available before the Minion tries to find the Master for the first time. The obvious place to do this in our current configuration is the configure script we started earlier.
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4set -euo pipefail 5 6apt-get install -y git sed 7 8sed -i -e '/[[:space:]]salt\([[:space:]]\|$\)/ {:l;n;bl}' -e '/localhost/ s/$/ salt/' /etc/hosts
sed is a fairly standard Linux tool available in most distributions (it is available in the debian/bullseye64 box already). So, why add it to the install on line 6? This is a precaution. If our configure script is run on a distribution that does not have sed installed we want to ensure that it is installed before trying to use it on line 8. If Git or sed are already installed attempting to install them a second time will not cause any error.
Line 8 needs some explanation. This is the line that adds salt as a domain name. It may be tempting to try something line echo "127.0.0.1 salt" >> /etc/hosts to append a suitable line to the /etc/hosts file. But consider what this would do if the configure script were run multiple times (as it has been already). Every run would add another 127.0.0.1 salt line to our /etc/hosts file. Not good.
The more complex sed command avoids this problem. The salt domain name is added only if no suitable entry already exists.
This idea that the script should result in the same output each time is called ‘idempotence’. A fancy word meaning ‘repeated application without change to the result beyond the first run’. Put another way, any idempotent operation has the same result on our system as the first time we run it.
Our configuration is not entirely idempotent yet. If we run this repeatedly there is a possibility of different results for the following reasons.
- apt-get install will install the latest available version of each package, so if either Git or sed packages are updated in the repository between runs then the version installed on our VM will be upgraded; our configuration results in different versions of these packages being installed, breaking idempotence.
- We have not specified a version to the Salt installation script. As with the packages this may result in the Salt version changing if the source repository is updated between our runs of the installation script.
- We have not reviewed the salt installation script to identify if it is idempotent.
The bootstrap-salt.sh and configure scripts can be combined into one script by adding the call to the bootstrap-salt.sh into the configure script, then remove the config.vm.provision line that calls this script from the Vagrantfile. As a final bit of cleanup for this version we change the name of the configure script to bootstrap-masterserver. The resulting files are shown next.
1#!/usr/bin/env bash 2# vi :set ft=bash: 3 4set -euo pipefail 5 6apt-get install -y git sed 7 8sed -i -e '/[[:space:]]salt\([[:space:]]\|$\)/ {:l;n;bl}' -e '/localhost/ s/$/ salt/' /etc/hosts 9 10pushd /tmp 11curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com 12sh bootstrap-salt.sh -M git master 13popd
1# -*- mode: ruby -*- 2# vi: set ft=ruby : 3 4Vagrant.configure("2") do |config| 5 config.vm.box = "debian/bullseye64" 6 config.vm.box_version = "v11.20220912.1" 7 config.vm.provider "virtualbox" do |vb| 8 vb.memory=8192 9 vb.cpus=4 10 end 11 config.vm.provision "shell", path: "scripts/bootstrap-masterserver" 12end
We now have one clean script to take a base Linux (Debian) install and add in tools to facilitate our full configuration.
On the plus side, this script is simple enough that it can be modified for alternate base systems without much difficulty and it does so little that debugging it, should the need arise, will be trivial.
On the downside, we are reliant on bootstrap-salt.sh which, while not catastrophic, leaves us open to some risks. We are currently pulling the latest version direct from SaltStack’s server which means it could change without warning, it also means any compromise to this script or the SaltStack domain could compromise any server built from this bootstrap-masterserver script. For now I think these risks are within my comfort zone so let us proceed.