Again self-hosting! on k3s

Over some time I was really happy with my podman + ansible setup. It was great, but do you know what wasn’t such great? Deployment rollbacks. It all started with linkwarden. On my miniflux, I received a notification - that a new minor release is ready.

You can use GitHub repos as RSS links and received notification about new releases. If you're using miniflux just past https://github.com/ansible/ansible/releases.atom

Let's say version 2.8.0. When the time came, I just changed my variable and executed ansible-playbook command. Then after 3 minutes, my uptime-kuma started screaming, and saying that my app was dead. Initially, I ignored that, due to my experience with podman. During the service restart it makes an app unavailable, then downloads the new image and tries to run it. If the package is big, it could take a while. It wasn't great, but that is what could be acceptable for self-hosting service. Unfortunately, the main problem was with something different. Maintainers of Linkwarden sometimes inform users about a new release, however, it's only delivered as an rpm/deb package. The container image could be still in the building, or the building process could fail. It doesn't matter, sometimes my typo could occur as well. Then podman can't handle that, nor Ansible as it just restarts the service. Podman on another hand just stops service and does not roll back to the old version automatically. So sometimes it becomes annoying as I need to log into the server with ssh and manually fix it. That is why I decided to switch to some Kubernetes distribution.

k3s

So why is the based version of small Kubernetes supported by Suse? Hmm mostly due to popularity, stability, and small resource consumption. It was good enough to run on a single host, and it's shipped as just a binary. This fits my needs, a small, easy-to-use package with build-in ingress, local-path storage provider, and flannel that implements Kubernetes CNI. Here is also a really nice project for installation and configuration base cluster, which I used for my setup.

The migration process

I did not have much time for migration, as I really enjoy spending time with my family, also with a full-time job it is just hard to put in the calendar. That is why I decided to join the "5 AM Club", and start re-learning Kubernetes again. I have a full log of my activities with dates, but probably that is not what you could be interested in. Let's say my regular everyday process look like this:

1. Apply terraform code

   1cd hetzner
   2terrafrom apply
   

2. Run k3s-ansible project, (check IP in inventory.yaml).

   1cd k3s-ansible
   2ansible-playbook playbooks/site.yml -i inventory.yml
   

3. With ready cluster update KUBECONFIG.

   1export KUBECONFIG=~/.kube/config.new
   2kubectl config use-context k3s-ansible
   

4. Patch k3s setup with traefik config.

   1cd cluster-config
   2k apply -f traefik-ext-conf.yaml
   

5. Create External Secret Operator main token for doppler

   1kubectl create namespace external-secrets
   2kubectl create secret generic \
   3    -n external-secrets \
   4    doppler-token-argocd \
   5    --from-literal dopplerToken="dp.st.xx"
   

6. Install EOS operator

   1helm install external-secrets \
   2   external-secrets/external-secrets \
   3    -n external-secrets \
   4    --create-namespace \
   5    --set installCRDs=true
   

7. Check if the webhook is up and running (sometimes is not)

   1k logs -l app.kubernetes.io/name=external-secrets-webhook -n external-secrets
   

It always works with helm, but due to differences of applying changes to kustomize sometimes does not work.

1. If yes, apply the overlay and create ClusterSecretStore

   1cd ESO/
   2kubectl apply -k overlay/
   

2. Install tailscale

   1cd tailscale
   2kubectl apply -k .
   

3. Install argo

   1cd argocd
   2kubectl apply -k base/
   3kubectl apply -k overlay/
   

4. Get Argo init admin password

   1kubectl --namespace argocd get \
   2    secret argocd-initial-admin-secret \
   3    -o json \
   4    | jq -r '.data.password' \
   5    | base64 -d
   

So far so good. Now let's break down used services.

Operators

I started by extending my traefik configuration to be able to handle regular requests from the internet. If you decide to do it, as well please be aware that k3s for today (07-Feb-2025) is using version 2+, not 3+, which is why not everything straight from documentation will work. For example, my code is:

 1kind: HelmChartConfig
 2metadata:
 3  name: traefik
 4  namespace: kube-system
 5  # we're still on with k3s https://github.com/traefik/traefik-helm-chart/blob/v27.0.2/traefik/values.yaml
 6spec:
 7  valuesContent: |-
 8    additionalArguments:
 9      - "[email protected]"
10      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
11      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
12      # - "--certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory"    

With that, and duckdns domain (Hetzner, very often gave you the same IP address. Thanks folks!). I was able to expose my service directly to the internet with the usage HTTPS.

 1apiVersion: networking.k8s.io/v1
 2kind: Ingress
 3metadata:
 4  name: argocd
 5  namespace: argocd
 6  annotations:
 7    traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
 8    traefik.ingress.kubernetes.io/router.tls: "true"
 9    traefik.ingress.kubernetes.io/router.tls.certresolver: "letsencrypt"
10    traefik.ingress.kubernetes.io/service.serversscheme: "h2c"
11spec:
12  ingressClassName: "traefik"
13  tls:
14    - hosts:
15        - nginx997.duckdns.org
16  rules:
17  - host: nginx997.duckdns.org
18    http:
19      paths:
20      - path: /
21        pathType: Prefix
22        backend:
23          service:
24            name: nginx
25            port:
26              number: 80

Then we have a challenge called:

How to store secrets with the usage of external sources?

I decided to use two products: External Secret Operator, and Doppler. In the beginning, I thought about Bitwarden's "not-so-new" Secret Manager, however, after a short investigation, the product seems to be not so well-supported by the ESO, which IMO is useful as it allows me to have one cluster-wide secret for getting secrets from external sources, which is great.

Doppler and ESO combo requires another post, so check my website from time to time.

Then I wanted to add Tailscle which besides being a "best in class VPN" for the homelabbers, allows you to add k8s services directly into your tailnet. What does it mean? The Tailscale operator allows you to access your k8s applications only when you are logged into your private network (tailnet), with the usage of your domain for ended with ts.net. You can configure it in two ways on the resource side, with ingress or with service annotation.

1. Ingress

 1apiVersion: networking.k8s.io/v1
 2kind: Ingress
 3metadata:
 4  name: hello
 5  namespace: hello
 6spec:
 7  ingressClassName: tailscale
 8  tls:
 9    - hosts:
10        - hello.john-kira.ts.net
11  rules:
12    - host: hello.john-kira.ts.net
13      http:
14        paths:
15          - path: /
16            pathType: Prefix
17            backend:
18              service:
19                name: nginx
20                port:
21                  number: 80

1. Service

 1---
 2apiVersion: v1
 3kind: Service
 4metadata:
 5  name: hello
 6  annotations:
 7    tailscale.com/expose: "true"
 8    tailscale.com/tailnet-fqdn: "hello.john-kira.ts.net"
 9    tailscale.com/hostname: "hello"
10spec:
11  ports:
12    - name: http
13      port: 80
14      targetPort: 80
15      protocol: TCP
16  selector:
17    app: nginx

Also please be aware that service by default exposes your service into tailnet over HTTP, where ingress provides a TLS certification as well.

Nice, at this point I was able to access my public service, as well as internal. What was the next step? ArgoCD.

Deploying ArgoCD was simple. During the first iteration, I decided to split my repo into base and overlay folders. The first directory contains the file for deploying an instance of Argo, exposing it to the tailnet. To archive it I just created a simple kustomization.yaml file as below:

 1---
 2apiVersion: kustomize.config.k8s.io/v1beta1
 3kind: Kustomization
 4
 5namespace: argocd
 6resources:
 7- namespace.yaml
 8- https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.2/manifests/install.yaml
 9
10# add --insecure flag to deployment, to avoid 307 redirection loop
11patches:
12- target:
13    kind: ConfigMap
14    name: argocd-cmd-params-cm
15  path: configmap-patch.yaml
16- target:
17    kind: Service
18    name: argocd-server
19  path: patch-argocd-server-annotations.yaml

Then two patches were:

• patch-argocd-server-annotations.yaml

1apiVersion: apps/v1
2kind: Service
3metadata:
4  name: argocd-server
5  annotations:
6    tailscale.com/expose: "true"
7    tailscale.com/tailnet-fqdn: "argo.john-kira.ts.net"
8    tailscale.com/hostname: "argo"

• configmap-patch.yaml

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: argocd-cmd-params-cm
5  namespace: argocd
6data:
7  server.insecure: "true"

Then my overlay directory contains only Argo's objects definitions:

1overlay
2├── applications.yaml
3├── kustomization.yaml
4├── projects.yaml
5└── repositories.yaml

Summary

That was my first iteration of the "5 AM Club" Kubernetes migration. It takes me a bit longer than one or two mornings. Based on my notes it was around 7 days of work, ~1h per day. Not bad at all, however, there were a few things that are missing and which I should improve to make my setup much more flexible, stable, and easier to test. Where "to test" I understand deploying a new version of the operator and watching how the cluster is burning. Or not, I hope rather not.

With that in mind, thanks for your time, hope you enjoy reading it. If you would like to reach me or you have questions please use about page infos.