Salt on Graviton

Jan 27, 2023 · 6 min read · Graviton Saltstack AWS

It will be nice to solve another real-world problem, isn’t it? For example machines in private subnets, behind the NAT, or on rack shelves in an office. All of them need to be managed - at least from time to time.

Ansible is great, I appreciated the creators of this tool. It helps me a lot! In different cases, environment. I remember when I was a bit younger, a very popular interview question was - what is the difference between Ansible and Chef/Puppet ?. If we skip the part, that Chef/Puppet isn't very popular in Poland, the main and always correct answer was that Ansible is agentless.

Then I met a guy. Do you know what he said? Agentless? Phi, what about all these community roles and packages, python packages, etc? Your workstation is an agent. That for an unknown reason was a game-changer for me. From that point in time I desired an agent.

It solves the ingress issue, but the need for an experiment was stronger!

Issue

Let's talk about the ingress issue for a moment. According to security best practices we need to take care of a lot of incoming traffic. CIDR ranges, NATs, VPN, private subnets, bastion hosts, or System Manager which is awesome (however good luck with running Ansible through it!). Security is great, unfortunately, it makes our life only harder. That's why I as a smart guy decided to use agent-based SaltStack. I started with one machine behind the NAT, now I have over 100 agents around the world. In the cloud, as well as in data centers.

Tools used in this episode

git
saltstack
terrafrom

Scenario

Let's assume that we're running 5 machines in AWS. To make it fancier, let's use Gravitons2 - AWS Arm64-based instances. Also, two of our machines will be placed Data Center, ah and one small server in the office. Do you need a diagram for it? Probably, not. However diagrams are awesome, so here we go:


Mondodraw based picture

First problem

In the beginning went I started playing with Salt, there were no pre-built agents. Seems that today it's the same. What should be done then? Compile from the source - nothing special in the ARM ecosystem. How to do it? It's very simple:

1echo ${host} > /etc/salt/minion_id
2curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh
3sh install_salt.sh -x python3 -i ${host} git

That's it! It is that simple. In full picture it looks like that:

Spin new EC2 instance with pre-baked AMI
Provide user-data script
accept agent on salt-master side
Run all needed formulas

And the code:

1# hansolo.tf
2[...]
3user_data = templatefile("./init.sh.tpl", {
4    host       = "hansolo"
5    saltmaster = "204.0.1.12"
6  })
7[...]

 1#!/bin/bash
 2# install satlstack minion
 3sudo su
 4apt-get update
 5apt-get install git curl
 6hostname ${host}
 7echo ${host} > /etc/salt/minion_id
 8curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh
 9sh install_salt.sh -x python3 -i ${host}  git
10sed -i "/#master: salt/c\master: ${saltmaster}" /etc/salt/minion
11# temporary fix on templates rendering
12sed -i -e '101,107d' /usr/local/lib/python3.8/dist-packages/salt/utils/templates.py
13systemctl restart salt-minion

Tips

As we're running an agent-based solution, we need to take care of versions of our components. In general, we have two options here:

Update master often
Point install_salt.sh to dedicated tag

In general, I prefer the first option, however, it's worth remembering.

Basic Salt syntax

In the begining directory structure, in my case looks like that:

1root@saltmaster:/srv/salt# tree -L 1
2.
3├── configs # keeps configs
4├── setups # formulas
5├── top.sls # top file
6└── users # user

Then we can take a look at the basic setup, for example, Jenkins control plane.

  1{% set plugin_manager_version = '2.12.3' %}
  2
  3Add Jenkins Repo:
  4  pkgrepo.managed:
  5    - humanname : Jenkins LTS
  6    - name: deb [arch=all] https://pkg.jenkins.io/debian-stable binary/
  7    - file: /etc/apt/sources.list.d/jenkins.list
  8    - gpgcheck: 1
  9    - key_url: https://pkg.jenkins.io/debian-stable/jenkins.io.key
 10
 11Update Repos:
 12  pkg.uptodate:
 13    - refresh : True
 14
 15Install Jenkins Packages:
 16  pkg.installed:
 17    - pkgs:
 18      - openjdk-11-jdk
 19      - jenkins
 20      - nginx
 21      - certbot
 22      - python3-certbot-nginx
 23
 24Set JAVA_HOME:
 25  file.append:
 26    - require:
 27      - pkg: Install Jenkins Packages
 28    - name: /var/lib/jenkins/.bash_profile
 29    - text: export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
 30
 31Get Plugin Manager:
 32  file.managed:
 33    - name: /var/lib/jenkins/jenkins-plugin-manager.jar
 34    - source: https://github.com/jenkinsci/plugin-installation-manager-tool/releases/download/{{ plugin_manager_version }}/jenkins-plugin-manager-{{ plugin_manager_version }}.jar
 35    - skip_verify: true
 36    - user: jenkins
 37    - group: jenkins
 38
 39Copy plugin file:
 40  file.managed:
 41    - require:
 42      - file: /var/lib/jenkins/jenkins-plugin-manager.jar
 43    - source:
 44      - salt://configs/plugins.txt
 45    - name: /var/lib/jenkins/plugins.txt
 46    - user: jenkins
 47    - group: jenkins
 48    - mode: 644
 49
 50Generate latest plugin:
 51  cmd.run:
 52    - name: java -jar /var/lib/jenkins/jenkins-plugin-manager.jar -f /var/lib/jenkins/plugins.txt --war /usr/share/java/jenkins.war --available-updates --output txt > /var/lib/jenkins/plugins-new.txt
 53    - runas: jenkins
 54
 55Install plugin:
 56  cmd.run:
 57    - name: java -jar /var/lib/jenkins/jenkins-plugin-manager.jar -f /var/lib/jenkins/plugins-new.txt --war /usr/share/java/jenkins.war --plugin-download-directory /var/lib/jenkins/plugins/
 58    - runas: jenkins
 59
 60Copy JaaC:
 61  file.managed:
 62    - source:
 63      - salt://configs/jenkins.yaml
 64    - name: /var/lib/jenkins/jenkins.yaml
 65    - user: jenkins
 66    - group: jenkins
 67    - mode: 644
 68    - template: jinja
 69    - defaults:
 70        akv_secret: {{ pillar['akv_secret'] }}
 71
 72Copy Jenkins server settings:
 73  file.managed:
 74    - source:
 75      - salt://configs/jenkinsai.master.config
 76    - name: /etc/default/jenkins
 77
 78Restart Jenkins:
 79  service.running:
 80    - name: jenkins
 81    - enable: True
 82    - watch:
 83      - file: /var/lib/jenkins/jenkins.yaml
 84
 85Create cert dir:
 86  file.directory:
 87    - name: /etc/nginx/certs
 88    - user: root
 89
 90Copy fullchain:
 91  file.managed:
 92    - source:
 93      - salt://configs/wildcard.3sky.fullchain.pem
 94    - name: /etc/nginx/certs/fullchain.pem
 95
 96Copy privkey:
 97  file.managed:
 98    - source:
 99      - salt://configs/wildcard.3sky.privkey.pem
100    - name: /etc/nginx/certs/privkey.pem
101
102Copy SSL config:
103  file.managed:
104    - source:
105      - salt://configs/options-ssl-nginx.conf
106    - name: /etc/nginx/options-ssl-nginx.conf
107
108Add dhparam:
109  cmd.run:
110    - name: openssl dhparam -out /etc/nginx/ssl-dhparams.pem 2048
111    - creates: /etc/nginx/ssl_dhparam.pem
112
113/etc/nginx/sites-available/jenkinsai.3sky.com:
114  file.managed:
115    - require:
116      - pkg: Install Jenkins Packages
117    - source:
118      - salt://configs/jenkinsai.3sky.com
119    - user: root
120    - group: root
121    - mode: 644
122
123
124/etc/nginx/sites-enabled/jenkinsai.3sky.com:
125  file.symlink:
126    - require:
127      - file: /etc/nginx/sites-available/jenkinsai.3sky.com
128    - name: /etc/nginx/sites-enabled/jenkinsai.3sky.com
129    - target: /etc/nginx/sites-available/jenkinsai.3sky.com
130
131/etc/nginx/sites-enabled/default:
132  file.absent:
133    - name: /etc/nginx/sites-enabled/default
134
135reload_jenkins:
136  service.running:
137    - name: jenkins
138    - enable: True
139    - reload: True
140
141reload_nginx:
142  service.running:
143    - name: nginx
144    - enable: True
145    - reload: True
146    - watch:
147      - file: /etc/nginx/sites-enabled/jenkinsai.3sky.com

Yp, that's true - the whole standalone Jenkins setup in one file. Awesome right?
Then I need to configure my top.sls.

1base:
2  'hansolo':
3    - setups.upgrade
4    - setups.init_setup
5    - setups.jenkins_main

And execute one simple command:

1salt -L 'hansolo' state.apply

That will apply all formulas on my brand-new VM. Cool!

Summary

What I can say about this article? I hope it will be useful if some would like to play a bit with ARM-based Salt agents. It takes me some time to figure out, how all this needs to be set up in a clear and easy-to-use way.

I'm also aware, that I can run it over dedicated AMI. The thing is that my fleet is rather stable, I treat them as pets with a name, etc. That's related to the specific workflow of one of my clients. We like stable long-living and very stable machines, that's why we have bare metal servers as well.

That is the first article this year, so Happy New Year!. I'd like to bring my blog to life again and write one post per month. Where will go? We will see.