Coreos on Major Hayden

Connect Caddy to Porkbun

major@mhtx.net (Major Hayden) — Thu, 29 Feb 2024 00:00:00 +0000

I recently told a coworker about Caddy, a small web and proxy server with a very simple configuration. It also has a handy feature where it manages your TLS certificate for you automatically.

However, one problem I had at home with my CoreOS deployment is that I don’t have inbound network access to handle the certificate verification process. Most automated certificate vendors need to reach your web server to verify that you have control over your domain.

This post talks about how to work around this problem with domains registered at Porkbun.

DNS validation

Certificate providers usually default to verifying domains by making a request to your server and retrieving a validation code. If your systems are all behind a firewall without inbound access from the internet, you can use DNS validation instead.

The process looks something like this:

You tell the certificate provider the domain names you want on your certificate
The certificate provider gives you some DNS records to add wherever you host your DNS records
You add the DNS records
You get your certificates once the certificate provider verifies the records.

You can do this manually with something like acme.sh today, but it’s painful:

# Make the initial certificate request
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please

# Add your DNS records manually.

# Verify the DNS records and issue the certificates.
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew

# Copy the keys/certificates and configure your webserver.

We don’t want to live this way.

Let’s talk about how Caddy can help.

Adding Porkbun support to Caddy

Caddy is a minimal webserver and Porkbun support doesn’t get included by default. However, we can quickly add it via a simple container build:

FROM caddy:2.7.6-builder AS builder

RUN xcaddy build \
 --with github.com/caddy-dns/porkbun

FROM caddy:2.7.6

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

This is a two stage container build where we compile the Porkbun support and then use that new caddy binary in the final container.

We’re not done yet!

Automated Caddy builds with updates

I created a GitHub repository that builds the Caddy container for me and keeps it updated. There’s a workflow to publish a container to GitHub’s container repository and I can pull containers from there on my various CoreOS machines.

In addition, I use Renovate to watch for Caddy updates. New updates come through a regular pull request and I can apply them whenever I want.

Connecting to Porkbun

We start here by getting an API key to manage the domain at Porkbun.

Log into your Porkbun dashboard.
Click Details to the right of the domain you want to manage.
Look for API Access in the leftmost column and turn it on.
At the top right of the dashboard, click Account and then API Access.
Add a title for your new API key, such as Caddy, and click Create API Key.
Save the API key and secrey key that are displayed.

Open up your Caddy configuration file (the Caddyfile) and add some configuration:

{
 email me@example.com

 # Uncomment this next line if you want to get
 # some test certificates first.
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

 acme_dns porkbun {
 api_key pk1_******
 api_secret_key sk1_******
 }
}

example.com {
 handle {
 respond "Hello world!"
 }
}

Save the Caddyfile and restart your Caddy server or container. Caddy will immediately begin requesting your TLS certificates and managing your DNS records for those certificates. This normally finishes in less than 30 seconds or so during the first run.

If you don’t see the HTTPS endpoint working within a minute or two, be sure to check the Caddy logs. You might have a typo in a Porkbun API key or the domain you’re trying to modify doesn’t have the API Access switch enabled.

[!NOTE] Remember that Porkbun requires you to enable API access for each domain. API access is disabled at Porkbun by default.

That’s it! 🎉

Renewals

Caddy will keep watch over the certificates and begin the renewal process as the expiration approaches. It has a very careful retry mechanism that ensures your certificates are updated without tripping any rate limits at the certificate provider.

How I learned to stop worrying and love the CoreOS

major@mhtx.net (Major Hayden) — Fri, 13 Oct 2023 00:00:00 +0000

It’s quite clear that I’ve been on a CoreOS blogging streak lately. I keep getting asked by people inside and outside my company about what makes CoreOS special and why I’ve switched over so many workloads to it.

The answer is pretty basic. It makes my life easier.

I’m a Dad. I’m on the PTC (Parent Teacher Club) at one of my children’s schools. I volunteer as an IT person for a non-profit. I write software. I have other time consuming hobbies, such as ham radio, reading, and becoming a longer distance runner¹.

My available time for my own IT projects is extremely limited and CoreOS plays a part in keeping that part of my life as efficient as possible.

That’s what this blog post is about!

Updates

First and foremost, I love how CoreOS does updates. I encourage you to read the docs on this topic, but here’s a short explanation:

Updates are automatically retrieved and they’re loaded into a slot.
Your system reboots into the new update but your original OS tree remains in place.
Did the update boot? Awesome. You’re good to go.
Did something break? The system reverts back to the known good tree.

In this way, it’s a lot like your smartphone.

You have full control over when a node looks for an update and how often it checks for them. Check out the Zincati docs for tons of controls over updates and reboots.

Some of mine are timed so well that I set maintenance windows with my monitoring provider when I know an update might take place. The updates come through, monitoring shuts off, the node reboots, and monitoring comes back. The nodes almost always come back before the monitoring even alerts me.

It also removes the reminders I would set for myself to update packages and run reboots. I know that my CoreOS nodes will do this automatically, so I don’t need to think about it.

Also, updates are rarely ever impactful to my workloads since all of them are running inside containers. My containers come right back up as soon as the node finishes its reboot.

Toolbox

To be fair, you can get toolbox running on lots of different Linux distributions outside of CoreOS, but that’s the first place I ever used it. Toolbox, also called toolbx, gives you a utility container on your CoreOS node for all kinds of adminsitrative and diagnostic capabilities.

You might need a certain package for diagnosting a hardware issue or you might want to install some helpful utilities for the command line. Do that in a toolbox container. Just run toolbox enter and if you’ve never created a toolbox container before, you’ll get a Fedora container that matches your CoreOS release.

But it gets better.

Toolbox automatically saves your container when you’re done with it so all of your installed packages stay there for next time. Also, these containers have seamless access to anything you have in your home directory, including sockets. You’re running inside a container, but it’s almost like you’re running on the host itself inside your home directory. You get the best of both worlds.

Don’t want Fedora? You have lots of distribution options through toolbox. Read the details on custom images to create your own!

Layering

Okay, there are those situations where you really want a package on CoreOS and toolbox might not be sufficient. My muscle memory for vim is so strong and CoreOS only comes with vi.

You have a couple of options here:

Run sudo rpm-ostree install vim, reboot, and you have vim
Run sudo rpm-ostree install --apply-live vim and you have vim right now! (And it’s there after a reboot as well.)

When a new update comes down for the base OS from CoreOS, any packages you’ve added will be layered on the base image and available after a reboot. Layering is generally chosen as a last resort option for adding packages to the system but you shouldn’t run into issues if you’re installing small utilities or command line tools.

Declarative provisioning

If you’ve provisioned Linux distributions on cloud instances in the past, you’ve likely provided metadata that cloud-init uses to provision your system. CoreOS has something that acts a lot earlier in the boot process and has more power to get things done: ignition.

There’s a handy butane file forma that you use for writing your configuration. You use the butane utility to get it into ignition format. The ignition format is highly compressed to ensure you can fit your configuration into most cloud providers’ metadata fields.

For a real example of what you can do with ignition, check out my quadlets post where I provisioned an entire Wordpress container stack using a single ignition file.

There’s lots of documentation for writing butane configuration files for common situations. It’s easy to add files, configure Wireguard, set up users, and launch containers immediately on the first boot.

Pets and cattle

CoreOS works well for systems that I only need online for a short time. These might be situations where I need to test a few containers and throw it away. There’s no OS to mess with and no updates to worry about.

It also works well for systems that I keep online for a long time. I have a few physical systems at home that run CoreOS and they’ve been extremely stable. I also have cloud instances on Hetzner, VULTR, and Digital Ocean that have run CoreOS for months without issues.

Quadlets might make me finally stop using docker-compose

major@mhtx.net (Major Hayden) — Mon, 25 Sep 2023 00:00:00 +0000

I’ve written a lot about containers on this blog. Why do I love containers so much?

They start quickly
They make your workloads portable
They disconnect your application stack from the OS that runs underneath
You can send your application through CI as a single container image
You can isolate workloads on the network and limit their resource usage much like a VM

However, I’m still addicted to docker-compose. Can podman’s quadlets change that?

Yes, I think they can.

What’s a quadlet?

Podman introduced support for quadlets in version 4.4 and it’s a simpler way of letting systemd manage your containers. There was an option in the past to have podman generate systemd unit files, but those were unwieldy and full of podman command line options inside a unit file. These unit files weren’t easy to edit or even parse with eyeballs.

Quadlets make this easier by giving you a simple ini-style file that you can easily read and edit. This blog post will include some quadlets later, but here’s an example one for Wordpress:

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Lots of the lines under [Container] should look familiar to most readers who have worked with containers before. However, there’s something new here.

Check out the AutoUpdate=registry line. This tells podman to keep your container updated on a regular basis with the upstream container registry. I’ve used watchtower in the past for this, but it requires a privileged container and it’s yet another external dependency.

Also, at the very end, you’ll see a WantedBy line. This is a great place to set up container dependencies. In this example, the container that runs caddy (a web server) can’t start until Wordpress is up and running.

So why not stick with docker-compose?

There’s no denying that docker-compose is an awesome tool. You specify the desired outcome, tell it to bring up containers, and it gets containers into the state you specified. It handles volumes, networks, and complicated configuration without a lot of legwork. The YAML files are pretty easy to read, too.

However, as with watchtower, that’s another external dependency.

My container deployments are often done at instance boot time and I don’t make too many changes afterwards. I found myself using docker-compose for the initial deployment and then I didn’t really use it again.

Why not remove it entirely and use what’s built into CoreOS already?

Quaint quadlets quickly!

Before we start, we’re going to need a few things:

An easy to read butane configuration which gets transformed into a tiny ignition configuration for CoreOS
Some quadlets
Extra system configuration
A cloud provider with CoreOS images (using VULTR for this)

I’ve packed all of these items into my quadlets-wordpress repository to make it easy. Start by looking at the config.butane file.

Let’s break it down here. First up, we add an ssh key for the default core user.

variant: fcos
version: 1.5.0
passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0

Next up, we enable the podman-auto-update.timer so we get container updates automatically:

storage:
 links:
 - path: /home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer
 target: /usr/lib/systemd/user/podman-auto-update.timer
 user:
 name: core
 group:
 name: core

Next is the long files section:

 files:
 # Ensure the `core` user can keep processes running after they're logged out.
 - path: /var/lib/systemd/linger/core
 mode: 0644
 
 # Allow caddy to listen on 80 and 443.
 # Allow it to ask for bigger network buffers, too.
 - path: /etc/sysctl.d/90-caddy.conf
 contents:
 inline: |
 net.ipv4.ip_unprivileged_port_start = 80
 net.core.rmem_max=2500000
 net.core.wmem_max=2500000

 # Set up an an environment file that containers can read to configure themselves.
 - path: /home/core/.config/containers/containers-environment
 contents:
 inline: |
 MYSQL_DATABASE=wordpress
 MYSQL_USER=wordpress
 MYSQL_ROOT_PASSWORD=mariadb-needs-a-secure-password
 MYSQL_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_HOST=mariadb
 WORDPRESS_DB_USER=wordpress
 WORDPRESS_DB_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_NAME=wordpress
 mode: 0644

 # Deploy the caddy configuration file from the repository.
 - path: /home/core/.config/caddy/Caddyfile
 contents:
 local: caddy/Caddyfile
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add some named volumes for caddy and wordpress.
 - path: /home/core/.config/containers/systemd/caddy-config.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/caddy-data.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/wordpress.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core

 # Create a network for all the containers to use and enable the
 # DNS plugin. This allows containers to find each other using
 # the container names.
 - path: /home/core/.config/containers/systemd/wordpress.network
 contents:
 inline: |
 [Network]
 DisableDNS=false
 Internal=false
 user:
 name: core
 group:
 name: core

 # Add the wordpress container.
 - path: /home/core/.config/containers/systemd/wordpress.container
 contents:
 local: quadlets/wordpress.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the MariaDB container.
 - path: /home/core/.config/containers/systemd/mariadb.container
 contents:
 local: quadlets/mariadb.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the caddy container.
 - path: /home/core/.config/containers/systemd/caddy.container
 contents:
 local: quadlets/caddy.container
 mode: 0644
 user:
 name: core
 group:
 name: core

The Caddyfile is also in the repository and will be deployed by the butane configuration shown above.

We can go through each quadlet in detail. First up is MariaDB. We tell systemd that the wordpress container will want to have this one started first.

[Unit]
Description=MariaDB Quadlet

[Container]
Image=docker.io/library/mariadb:11
ContainerName=mariadb
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=mariadb.volume:/var/lib/mysql
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=wordpress.service multi-user.target default.target

The wordpress quadlet is much the same as the MariaDB one, but we tell systemd that caddy will want wordpress started first.

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Finally, the caddy quadlet contains four volumes and some published ports. These ports will be published to the container host. Also, you’ll note that the wordpress volume is mounted here, too. This is because caddy can serve static files much faster than wordpress can.

[Unit]
Description=Caddy Quadlet

[Container]
Image=docker.io/library/caddy:latest
ContainerName=caddy
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=caddy-data.volume:/data
Volume=caddy-config.volume:/config
Volume=/home/core/.config/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=wordpress.volume:/var/www/html
PublishPort=80:80
PublishPort=443:443
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

Launch the quadlets

There’s a launch script that ships this configuration to VULTR and launches a CoreOS instance:

#!/bin/bash
# This command starts up a CoreOS instance on Vultr using the vultr-cli
vultr-cli instance create \
 --os 391 \
 --plan vhp-1c-1gb-amd \
 --region dfw \
 --notify true \
 --ipv6 true \
 -u "$(butane --files-dir . config.butane)" \
 -l "coreos-$(date "+%s")"

To launch an instance, get your VULTR API key first. Then install vultr-cli and butane:

$ sudo dnf -y install butane vultr-cli

After launch, check to see what your containers are doing:

[core@vultr ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afa2d6501593 docker.io/library/caddy:latest caddy run --confi... 54 seconds ago Up 53 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp caddy
460426f39e6c docker.io/library/mariadb:11 mariadbd 35 seconds ago Up 35 seconds mariadb
92ece6538d5a docker.io/library/wordpress:fpm php-fpm 28 seconds ago Up 29 seconds wordpress

We should be able to talk to wordpress through caddy on port 80:

[core@vultr ~]$ curl -si http://localhost/wp-admin/install.php | head -n 25
HTTP/1.1 200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Server: Caddy
X-Powered-By: PHP/8.0.30
Date: Mon, 25 Sep 2023 21:43:40 GMT
Transfer-Encoding: chunked

<!DOCTYPE html>
<html lang="en-US" xml:lang="en-US">
<head>
	<meta name="viewport" content="width=device-width" />
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
	<meta name="robots" content="noindex,nofollow" />
	<title>WordPress &rsaquo; Installation</title>
	<link rel='stylesheet' id='dashicons-css' href='http://localhost/wp-includes/css/dashicons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='http://localhost/wp-includes/css/buttons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='http://localhost/wp-admin/css/forms.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='http://localhost/wp-admin/css/l10n.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='http://localhost/wp-admin/css/install.min.css?ver=6.3.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui language-chooser">
<p id="logo">WordPress</p>

Awesome! 🎉

Managing containers

Containers will automatically update on a schedule and you can check the timer:

[core@vultr ~]$ systemctl status --user podman-auto-update.timer
● podman-auto-update.timer - Podman auto-update timer
 Loaded: loaded (/usr/lib/systemd/user/podman-auto-update.timer; enabled; preset: disabled)
 Active: active (waiting) since Mon 2023-09-25 21:41:31 UTC; 3min 14s ago
 Trigger: Tue 2023-09-26 00:04:46 UTC; 2h 20min left
 Triggers: ● podman-auto-update.service

Sep 25 21:41:31 vultr.guest systemd[1786]: Started podman-auto-update.timer - Podman auto-update timer.

Quadlets are just regular systemd units:

[core@vultr ~]$ systemctl list-units --user | grep -i Quadlet
 caddy.service loaded active running Caddy Quadlet
 mariadb.service loaded active running MariaDB Quadlet
 wordpress.service loaded active running Wordpress Quadlet

As an example, you can make changes to caddy’s config file and restart it easily:

[core@vultr ~]$ systemctl restart --user caddy
[core@vultr ~]$ systemctl status --user caddy
● caddy.service - Caddy Quadlet
 Loaded: loaded (/var/home/core/.config/containers/systemd/caddy.container; generated)
 Drop-In: /usr/lib/systemd/user/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Mon 2023-09-25 21:46:28 UTC; 5s ago
 Main PID: 2652 (conmon)
 Tasks: 18 (limit: 1023)
 Memory: 15.1M
 CPU: 207ms

If you need to change a quadlet’s configuration, just open up the configuration file in your favorite editor under ~/.config/containers/systemd, reload systemd, and restart the container:

$ vi ~/.config/containers/systemd/caddy.container

--- make your edits and save the quadlet configuration ---

$ systemctl daemon-reload --user
$ systemctl restart --user caddy

Enjoy!

Deploy a containerized Ghost blog 👻

major@mhtx.net (Major Hayden) — Tue, 27 Jun 2023 00:00:00 +0000

There’s no shortage of options for starting a self-hosted blog. Wordpress might be chosen most often, but I stumbled upon Ghost recently and their performance numbers really got my attention.

I prefer deploying most things in containers these days with Fedora CoreOS. Luckily, the Ghost stack doesn’t demand a lot of infrastructure:

Ghost itself
MySQL 8+ (I went with MariaDB 11.x)
A web server out front
TLS certificate

[!NOTE] Although I chose MariaDB for the database here, Ghost recommends MySQL and will throw a warning in the admin panel if you’re using something else. I haven’t had any issues so far, but you’ve been warned. 💣

I picked Caddy for the webserver since it’s so small and the configuration is tremendously simple.

Launch CoreOS

Fedora CoreOS offers lots of cloud options for launching it immediately. Many public clouds already have CoreOS images available, but I love Hetzner’s US locations and I already had a CoreOS image loaded up in my account.

🇩🇪 Want CoreOS at Hetzner? There’s a blog post for that!

Once your CoreOS instance is running, connect to the instance over ssh and ensure the docker.service starts on each boot:

sudo systemctl enable --now docker.service

This ensures that containers come up on each reboot. CoreOS has a podman socket that listens for docker-compatible connections, but that doesn’t help with reboots.

Perhaps I’m old fashioned, but I still enjoy using docker-compose for container management. I like how I can declare what I want and let docker-compose sort out the rest.

Let’s install docker-compose on the CoreOS instance now:

# Check the latest version in the GitHub repo before starting!
# https://github.com/docker/compose
curl -LO https://github.com/docker/compose/releases/download/v2.19.0/docker-compose-linux-x86_64

# Install docker-compose and make it executable.
sudo mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
sudo chown +x /usr/local/bin/docker-compose

Verify that docker-compose is ready to go:

$ docker-compose --version
Docker Compose version v2.19.0

Preparing Caddy

Caddy uses a configuration file called a Caddyfile and we need that in place before we deploy the other containers. Within my home directory, I created a directory called caddy:

mkdir caddy

Then I added the Caddyfile inside the directory:

{
 # Your email for LetsEncrypt warnings/notices.
 email youremail@domain.com 

 # Staging LetsEncrypt server to use while testing.
 # Uncomment this before going to production!
 acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

# Basic virtual host definition to feed traffic into the
# Ghost container when it arrives.
example.com {
 reverse_proxy ghost:2368
}

# OPTIONAL: Redirect traffic to 'www' to the bare domain.
www.example.com {
 redir https://example.com{uri}
}

This configuration sets up LetsEncrypt certificates automatically from the staging server for now. Once we know our configuration is working well, we can comment out the acme_ca line above and get production TLS certificates.

At this point, you need a DNS record pointed to your server so you can get a certificate. You have some options:

If the site is entirely new, just point the root domain name to your CoreOS instance. Use that domain in the configuration above and later in the deployment.
If you’re migrating from an existing site, choose a subdomain off your main domain to use. If your website is example.com, use something like test.example.com or new.example.com to get Ghost up and running. It’s really easy to change this later.

Now we’re ready for the rest of the deployment.

Deploying containers

Here’s the docker-compose.yml file I’m using:

---
version: '3.8'
services:

 # OPTIONAL
 # Watchtower monitors all running containers and updates
 # them when the upstream container repo is updated.
 watchtower:
 image: docker.io/containrrr/watchtower:latest
 container_name: watchtower
 restart: unless-stopped
 hostname: coreos-ghost-deployment
 environment:
 - WATCHTOWER_CLEANUP=true
 - WATCHTOWER_POLL_INTERVAL=3600
 command:
 - --cleanup
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 privileged: true

 # Caddy acts as our external-facing webserver and handles
 # getting TLS certs from LetsEncrypt.
 caddy:
 image: caddy:latest
 container_name: caddy
 depends_on:
 - ghost
 ports:
 - 80:80
 - 443:443
 restart: unless-stopped
 volumes:
 - ./caddy/Caddyfile:/etc/caddy/Caddyfile:Z
 - ghost:/var/www/html
 - caddy_data:/data
 - caddy_config:/config

 # The Ghost blog software itself
 ghost:
 image: docker.io/library/ghost:5
 container_name: ghost
 restart: always
 depends_on:
 - ghostdb
 environment:
 url: https://example.com
 database__client: mysql
 database__connection__host: ghostdb
 database__connection__user: ghost
 database__connection__password: GHOST_PASSWORD_FOR_MARIADB
 database__connection__database: ghostdb
 volumes:
 - ghost:/var/lib/ghost/content

 # Our MariaDB database
 ghostdb:
 image: docker.io/library/mariadb:11
 container_name: ghostdb
 restart: always
 environment:
 MYSQL_ROOT_PASSWORD: A_SECURE_ROOT_PASSWORD
 MYSQL_USER: ghost
 MYSQL_PASSWORD: GHOST_PASSWORD_FOR_MARIADB
 MYSQL_DATABASE: ghostdb
 volumes:
 - ghostdb:/var/lib/mysql

volumes:
 caddy_config:
 caddy_data:
 ghost:
 ghostdb:

I love watchtower but that step is completely optional. It does require some elevated privileges to talk to the podman socket, so keep that in mind if you choose to use it.

Our ghostdb container starts first, followed by ghost, and then caddy. That follows the depends_on configuration keys shown above.

There are two steps to take now:

Replace GHOST_PASSWORD_FOR_MARIADB and A_SECURE_ROOT_PASSWORD above with better passwords. 😉
Also, set the url parameter for the ghost container to your blog’s domain name.

Once all of that is done, let’s let docker-compose do the heavy lifting:

sudo docker-compose up -d

Let’s verify that our containers are running:

$ sudo docker-compose ps
NAME IMAGE COMMAND SERVICE 
caddy caddy:latest "caddy run --config …" caddy 
ghost docker.io/library/ghost:5 "docker-entrypoint.s…" ghost 
ghostdb docker.io/library/mariadb:11 "docker-entrypoint.s…" ghostdb 
watchtower docker.io/containrrr/watchtower:latest "/watchtower --clean…" watchtower

Awesome! 👏

Ghost initial setup

With all of your containers running, browse to https://example.com/ghost/ Just add /ghost/ to the end of your domain name to reach the admin panel. Create your admin account there with a good password.

If everything looks good, run back to your Caddyfile and comment out the acme_ca line:

{
 # Your email for LetsEncrypt warnings/notices.
 email youremail@domain.com 

 # Staging LetsEncrypt server to use while testing.
 # Uncomment this before going to production!
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

Restart the caddy container to get a production LetsEncrypt certificate on the site:

sudo docker-compose restart caddy

Customizing Ghost

Ghost looks for lots of environment variables to determine its configuration and you can set these in your docker-compose.yml file. Although some configuration items are easy, like url, some are nested and get more complicated. For these, you can use double underscores __ to handle the nesting.

As an example, we already used database__connection__host in the docker-compose.yaml, and that’s the equivalent to this nested configuration:

"database": {
 "connection": {
 "host": "..."
 }
}

If you’re deploying in containers, it’s a good idea to configure Ghost via environment variables. This ensures that your docker-compose.yml is authoritative for the Ghost deployment. You can exec into the container, adjust the config file on disk, and restart Ghost, but then you have to remember where you configured each item. 🥵

Switching to production domain

If you used a temporary domain to get everything configured and you’re ready to use your production domain, follow these steps:

Open your Caddyfile and replace all instances of the testing domain with the production domain
Restart caddy: sudo docker-compose restart caddy
Edit the docker-compose.yml and change the url key in the ghost container to the production domain
Apply the configuration with sudo docker-compose up -d

Enjoy your new automatically-updating Ghost blog deployment! 👻

Launch a watchtower container via podman quadlets

major@mhtx.net (Major Hayden) — Wed, 31 May 2023 00:00:00 +0000

Most of my container workloads run on independent CoreOS cloud instances that I treat like pets. Keeping containers update remains a constant battle, but it’s still easier than running kubernetes.

I wrote about using watchtower in the past to keep containers updated. It’s a simple container that does a few important things:

It monitors (via docker/podman socket) the running containers on the host
It tracks the versions/tags of each container image
It looks for updated versions of the container image in their upstream repositories
Based on a configurable schedule, it pulls a new container image and restarts the container for updates

I encourage you to read more about watchtower on GitHub. There’s plenty you can configure, including update intervals, how updates are handled, and how you can get notifications when an update happens.

My new deployments always need watchtower running. Luckily, we can combine Fedora CoreOS’ initial provisioning system, called ignition, with podman’s new quadlet feature and launch watchtower automatically on the first boot.

Quadlets

So what’s a quadlet?

The blog post explains it well by making containers more declarative via a familiar systemd syntax. Here’s an example .container file from the post:

[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

You can toss this into $HOME/.config/containers/systemd/mysleep.container for rootless user containers or in /etc/containers/systemd/mysleep.container for a container running as root.

Configure a quadlet on boot

As I mentioned earlier, I want a watchtower container running on my CoreOS nodes at first boot. Let’s start with a fairly basic butane file:

variant: fcos
version: 1.4.0
passwd:
 users:
 - name: major
 groups:
 - wheel
 - sudo
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0
storage:
 files:
 - path: /etc/containers/systemd/watchtower.container
 contents:
 inline: |
 [Unit]
 Description=Watchtower container updater
 Wants=network-online.target
 After=network-online.target

 [Container]
 ContainerName=watchtower
 Image=ghcr.io/containrrr/watchtower:1.5.3@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca
 Environment=WATCHTOWER_CLEANUP=true
 Environment=WATCHTOWER_POLL_INTERVAL=3600
 Volume=/var/run/docker.sock:/var/run/docker.sock
 SecurityLabelDisable=true

 [Install]
 WantedBy=multi-user.target default.target

Let’s break this file down:

I start by adding a user named major that has administrative privileges an an ssh key (this is optional, but I like using my own username rather than core)
The quadlet unit file lands in /etc/containers/systemd/watchtower.container and starts at boot time

The quadlet file has some important configurations:

I added environment variables to clean up outdated container images and check for updates once an hour
The podman socket is mounted inside the watchtower container
Security labels are disabled to allow for communication with the podman socket

[!NOTE] Mounting the podman socket and disabling security labels is not an ideal security approach. However, I’ve found that watchtower’s configuration and automation fits my needs really well and I retreive the image from a trusted source. If this won’t work for you, you can use podman’s built-in auto-update feature instead.

From here, we convert the butane configuration into an ignition configuration. I’m launching this CoreOS node on VULTR, so I’ve named my files accordingly:

$ butane vultr-coreos.butane > vultr-coreos.ign

Let’s go 🚀

I’m using VULTR’s CLI here in Fedora, but you can do the same steps via VULTR’s portal if needed. Just paste in the ignition configuration into the large text box before launch.

# Install vultr-cli in Fedora
sudo dnf install vultr-cli

# Launch the instance
vultr-cli instance create --region dfw --plan vhp-2c-2gb-amd \
 --os 391 --label coreos-dfw-1 --host coreos-dfw-1 \
 --userdata "$(cat vultr-coreos.ign)"

Let’s see how the container is doing:

$ ssh major@COREOS_HOST

Fedora CoreOS 38.20230430.3.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[major@coreos-dfw-1 ~]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0024712c95d ghcr.io/containrrr/watchtower@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca About a minute ago Up About a minute watchtower

[major@coreos-dfw-1 ~]$ sudo podman logs watchtower
time="2023-05-31T14:01:12Z" level=info msg="Watchtower 1.5.3"
time="2023-05-31T14:01:12Z" level=info msg="Using no notifications"
time="2023-05-31T14:01:12Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-05-31T14:01:12Z" level=info msg="Scheduling first run: 2023-05-31 15:01:12 +0000 UTC"
time="2023-05-31T14:01:12Z" level=info msg="Note that the first check will be performed in 59 minutes, 59 seconds"

Awesome! 🥳

My system rebooted for an ostree update shortly after provisioning and the container came up automatically both times.

CoreOS as a pet

major@mhtx.net (Major Hayden) — Thu, 25 May 2023 00:00:00 +0000

Anyone working with containers has likely heard of CoreOS by this point Haven’t heard about it? Don’t despair. I’ll catch you up on what you missed.

Fedora CoreOS offers a really fast pathway to running containers on hardware, in virtual machines, or in clouds. It delivers a lightweight operating system with all of the container technology that you need for running simple containers or launching a kubernetes deployment.

But that’s not the best part.

CoreOS really shines due to its immutable OS layer¹. The OS underneath your containers ships as a single unit and it automatically updates itself much like your mobile phone. An update rolls down, CoreOS sets it up as a secondary OS, reboots into that new update, and rolls back to the original one if there were any issues.

Many people use CoreOS as the workhorse underneath kubernetes. Red Hat uses it underneath OpenShift as well. It’s even supported by the super light weight kubernetes distribution k3s.

But can you use CoreOS as a pet type instance that you use and maintain for long periods of time just like any other server? Absolutely!

What’s this pet stuff about?

Whether you like it or not, there’s a cattle versus pets paradigm that took hold in the world of IT at some point. The basic ideas are these:

When you take care of cattle, you take care of them as a group. Losing one or more of them would make you sad, but you know you have many others.
As for pets, you spend a lot of time taking care of them and playing with them. If you lost one, it would be devastating.

A fleet of web servers could be treated like cattle. Keep lots of them online and replace any instances that have issues.

On the other hand, databases or tier zero systems (everyone feels if it they went down) are like pets. You carefully build, maintain, and monitor these.

How does CoreOS fit in?

Many people do use CoreOS as a container hosting platform as part of a bigger system. It works really well for that. But it’s great as a regular cloud server, too.

You can run a single node CoreOS deployment and manage containers via the tools that you know and love. For example, docker-compose works great on CoreOS. I even used it to host my own Mastodon deployment.

You can also load up more user-friendly tools such as portainer to manage containers in a browser.

My development tools are missing!

😱 No vim? This is too minimal! What are we going to do?

Luckily CoreOS comes with toolbox. 🧰

Toolbox gives you the ability to run a utility container on the system with some handy benefits:

Toolbox environments have seamless access to the user’s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc..

This means that the toolbox feels like a second OS on the system and it has all of the elevated privileges that you need to do your work. Simply run toolbox enter, follow the prompts, and you’ll end up with a Fedora toolbox that matches your CoreOS version. Need a different version, such as Fedora Rawhide? Just specify the Fedora release you want on the prompt:

$ toolbox enter --release 39 
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:39 (500MB)? [y/N]: y

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢[major@toolbox ~]$

Look at the toolbox create --help output to see how to create lots of different toolbox containers with different names and releases. If you go overboard and need to delete some toolboxes, just list your toolboxes with toolbox list and follow it up with toolbox rm.

My tool won’t work in the toolbox.

Some applications have issues running inside a container, even one that has elevated privileges on the system. CoreOS offers an option for layering packages on top of the underlying immutable OS.

Simply run rpm-ostree install PACKAGE to layer a package on top of the OS. When rpm-ostree runs, it creates a new layer and sets that layer to be active on the next boot. That means that you need to reboot before you can use the package.

Don’t want to reboot? There’s another option, but I recommend against it if you can avoid it².

You can apply a package layer live on the system without a reboot with the --apply-live flag. Installing a package like mtr would look like this:

$ sudo rpm-ostree install --apply-live mtr

As soon as rpm-ostree finishes its work, mtr should be available on the system for you to use.

How do updates work?

There are two main technologies at work here.

First, zincati checks for updates to your immutable OS tree. It runs on a configurable schedule that you can adjust based on your preferences.

Second, rpm-ostree handles the OS layers and switches between them at boot time. If you’re running off layer A and an update comes down (layer B), that layer is written to the disk and activated on the next boot. Should there be any issues booting up layer B later, rpm-ostree switches the system back to layer A. In these situations, your downtime might be extended a bit due to two reboots. Your system will come back up with the original OS layer activated.

You also get a choice of update streams. Want to live a bit more on the edge? Go for next or testing. You’re on the stable stream by default.

Although I haven’t landed in this situation, it’s possible that the system boots into a new update where you notice a problem that doesn’t affect the boot. You can manually roll back to fix it.

I have more questions.

Your first stop should be the Fedora CoreOS docs. There are also lots of ways to contact the development team and talk with the community.

Love the idea of an immutable OS but you wish you had it for your desktop or laptop? Go check out Fedora Silverblue. 💻

Okay, so it’s mostly immutable. You can edit configuration in /etc and you can layer more packages on top of the base OS layer if you need them. However, CoreOS maintainers discourage adding layered packages if you can avoid it. ↩︎
When you apply some packages and make them available immediately, you may lose track of which ones were applied live and which ones are available on the next reboot. Things can get a bit confusing if you suddenly change your mind about applying a package live or not. ↩︎

Automatic container updates with watchtower

major@mhtx.net (Major Hayden) — Wed, 04 Jan 2023 00:00:00 +0000

Keeping things updated quickly becomes a monotonous task. I’m surrounded by devices that demand updates on different frequencies. Phones, computers, tables, cloud instances, containers, and even my car need constant attention for updates that improve security or fix bugs. (Sometimes the updates cause bugs, but let’s forget about those for now) 😉

My container infrastructure runs on Fedora CoreOS and it updates itself. It has an immutable layer underneath my containers that updates using ostree.

However, keeping containers updated is a constant battle. Updating the containers themselves is fairly easy with a podman pull or docker pull followed by a stop and start. It’s a bit easier with docker-compose, but it’s still a nuisance to remember to update.

Enter watchtower

Watchtower is incredibly handy and surprisingly simple to operate. At a high level, it reads the container tags from each running container and watches the upstream repositories for updates. When an updated container appears, watchtower springs into action, pulls the new container, and replaces the old container with the new one.

It accepts arguments to configure all kinds of aspects of updating containers. You can exclude certain containers from updates, choose your update interval, and send notifications when updates occur.

Also, if manually updating containers is something you find fascinating, watchtower can notify you about updates. Then you get that great feeling of running lots of commands on your own. (Wait, surely nobody likes that!)

Deploy

There are plenty of configuration snippets in watchtower’s documentation, but you can start off with something as simple with this in your docker-compose.yaml:

services:
 watchtower:
 image: ghcr.io/containrrr/watchtower
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 restart: always
 privileged: true

My configuration on my mastodon deployment looks something like this:

services:
 watchtower:
 container_name: watchtower
 image: ghcr.io/containrrr/watchtower
 hostname: watchtower.tootchute.com
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 environment:
 - WATCHTOWER_CLEANUP=true
 - WATCHTOWER_POLL_INTERVAL=86400
 - WATCHTOWER_NOTIFICATIONS=shoutrrr
 - WATCHTOWER_NOTIFICATION_URL=discord://DISCORD_WEBHOOK_KEY@DISCORD_CHANNEL_ID
 restart: always
 privileged: true

This configuration sends me notifications via my Discord server whenever watchtower starts or when a container is updated.

The docker.sock volume mount allows watchtower to interact with the container daemon underneath watchtower. This could easily be done with docker, moby-engine, or podman.

I’d like to remove the privileged setting at some point soon but I haven’t figured out a way to allow watchtower to talk to the docker.sock without it. 🤔

Further considerations

If you don’t trust the upstream where you download your containers, be careful using watchtower. A malicious container could be uploaded to a particular container image repository and your system might update itself to the malicious container before the malicious container is found. Then again, if you don’t trust the upstream where you download your containers, you should be building these containers yourself. 😉

For more complex services that might need some extra care around updates, such as database services, you may want to exclude them from automatic updates. You can run multiple instances of watchtower with customized configurations for different sets of containers.

docker-compose on Fedora CoreOS

major@mhtx.net (Major Hayden) — Sat, 17 Dec 2022 00:00:00 +0000

Deploying applications in containers provides lots of flexibility and compatibility benefits.

Once you package your application and its dependencies in a container, that container runs almost anywhere without issues. Very few of the old “it worked on my machine!” problems remain. However, the challenge of running a container and linking it up with other helpful pieces of software still remains.

Web applications need something to serve HTTP requests and handle TLS. They also need databases, and those databases must be online and available first. All of these need reliable storage that is easily managed.

In my personal infrastructure, I keep coming back to docker-compose.

🐇 In a hurry? Skip to the last section of this post if you want to skip my reasons for using docker-compose and you just want to see the steps.

Why docker-compose?

The tried-and-true docker-compose is one of the original “set your desired state” systems for managing containers. You specify what your container deployment should look like and docker-compose finds a way to get your containers in order. Sometimes that’s a fresh start without any existing containers. Sometimes it involves managing an existing fleet of containers and adjusting their configuration.

As an example, deploying your first container with docker-compose is easy. Assemble a basic YAML file, point docker-compose at it, and your container is running!

Need to change the configuration? Just make your changes in the YAML file, re-run docker-compose, and it knows enough to make the right changes. If containers need to be restarted, it takes care of that.

Why not kubernetes?

I’ve run my own kubernetes deployment and I’ve also been a consumer of large kubernetes and OpenShift deployments. All of this experience taught me two things:

Kubernetes and OpenShift are great. Like really great. Once you learn them, they are incredibly powerful tools that make a developer’s life much easier.
Running my own kubernetes or OpenShift deployment on my own time (and own dime) is not enjoyable.

Deploying, maintaining, and troubleshooting kubernetes infrastructure on your own is time consuming. Shared storage and networking caused me the most headaches in the past.

What about k3s?

I love k3s. However, that still means I have to figure out networking for inter-container communication and load balancing. Shared storage is also needed.

I could argue that running my own k3s deployment is easier than full kubernetes, but in the end, there’s more extra stuff around it that I don’t want to maintain.

Why don’t you use managed kubernetes?

Great question! Several providers have some excellent kubernetes offerings out there. Smaller providers, such as Digital Ocean and VULTR, have affordable offerings that are packed with features.

The challenge is that kubernetes deployments have overhead for the control plane, so you can’t use all of the virtual machines that you rent. For example, you may get three virtual machines in your cluster, each with 2GB RAM, but you can really only use about 1GB of RAM from each instance for your containers.

In addition, you will eventually need shared storage and some type of load balancer. The costs add up quickly.

It’s easy to start with a $50-$70 kubernetes offering and later find yourself cracking $100 per month after adding on storage and load balancers.

What about podman?

Podman is a delight to use. You can toss some kubernetes YAML at it for deployments or pods and it will start them up for you. It also has a handy feature for exporting a container to a systemd unit file so you can manage it like any other systemd unit.

However, when I want to make quick adjustments to a container configuration, it can be frustrating to get that done. I also like to make adjustments to all of my containers in one place since some applications depend on multiple containers running in tandem.

The podman-compose project helps quite a bit, but it still lacks some of docker-compose’s features.

I use podman constantly on my laptop and desktop for development, testing, and toolbox containers.

What’s the big deal about CoreOS?

Fedora CoreOS provides the foundation for my container infrastructure. I love Fedora already, but here’s what makes CoreOS special to me:

Automatic updates arrive via ostree as a fully tested minimal unit
My system reboots automatically to apply the updates and it reverts back to the previous working update if the update fails
It comes with all of the container tools and configurations that I need
Whenever I need development or troubleshooting tools, toolbox containers are one step away

It’s a constantly-updated OS that is designed for containers. What could be better than that?

Adding docker-compose to CoreOS

Deploy CoreOS in your favorite cloud or on your favorite piece of hardware (I have a post about deploying it on Hetzner Cloud). Login as the core user via ssh.

Go to the releases page for docker-compose and get the latest release for your architecture. Move it into place once you download it:

$ curl -sLO https://github.com/docker/compose/releases/download/v2.14.1/docker-compose-linux-x86_64
$ sudo mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

Let’s add a really basic deployment of traefik’s whoami container. Start a new file called docker-compose.yaml:

services:
 whoami:
 container_name: whoami
 image: docker.io/traefik/whoami
 ports:
 - 8080:80
 restart: unless-stopped

If we try to bring up our containers now, we get an error:

$ docker-compose up -d
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3Dcore%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied

The core user is not in the docker group and does not have permissions to talk to the docker socket.

$ id
uid=1000(core) gid=1000(core) groups=1000(core),4(adm),10(wheel),16(sudo),190(systemd-journal) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Add the docker group to the core user as a supplementary group:

$ sudo usermod -a -G docker core

Log out of your ssh session and log in again to pick up the new group. Start the containers once more:

$ id
uid=1000(core) gid=1000(core) groups=1000(core),4(adm),10(wheel),16(sudo),190(systemd-journal),982(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ docker-compose up -d
[+] Running 4/4
 ⠿ whoami Pulled
 ⠿ 029cd1bf7e7c Pull complete
 ⠿ e73b694ead4f Pull complete
 ⠿ 99df6e9e9886 Pull complete
[+] Running 2/2
 ⠿ Network core_default Created
 ⠿ Container whoami Started

Success!

🤔 BUT WAIT! If we reboot our system, the containers won’t start up at boot time!

On CoreOS, there’s a docker.socket file that starts the actual docker service (which is actually moby-engine on Fedora and not full-fledged docker). As soon as something touches the socket, systemd starts the docker service. That keeps it out of the way until something actually asks to use it.

$ systemctl status docker.socket
● docker.socket - Docker Socket for the API
 Loaded: loaded (/usr/lib/systemd/system/docker.socket; enabled; preset: enabled)
 Active: active (running) since Sat 2022-12-17 22:48:42 UTC; 4min 47s ago
 Until: Sat 2022-12-17 22:48:42 UTC; 4min 47s ago
 Triggers: ● docker.service
 Listen: /run/docker.sock (Stream)
 Tasks: 0 (limit: 2207)
 Memory: 0B
 CPU: 596us
 CGroup: /system.slice/docker.socket

After a reboot, nothing pokes the socket and the associated service never starts. You can try it yourself! Reboot and the whoami container will be down. Run docker-compose ps one time and suddenly your containers are running!

Let’s fix this so containers come up on boot without any extra work (or socket poking):

$ sudo systemctl enable --now docker.service

The socket will still be handled by systemd, but now the docker service itself will always start at boot whether someone touched it or not.

Enjoy your new lightweight container infrastructure. ✨

Deploy Fedora CoreOS in Hetzner cloud

major@mhtx.net (Major Hayden) — Fri, 20 Aug 2021 00:00:00 +0000

Hetzner has always been a reliable and cost-effective hosting company for me for several years. I’ve run icanhazip.com on their dedicated servers and I run several small applications in their cloud.

When I run containers, I love using Fedora CoreOS for its easy updates and very small server footprint. Almost everything you need for hosting containers is provided right out of the box, but you can add extra packages via rpm-ostree layers and reboot to use them.

This post shows you how to deploy Fedora CoreOS in Hetzner’s cloud as an inexpensive and efficient method for hosting your container workloads.

Roadblocks

First off, Hetzner does not offer Fedora CoreOS as one of its cloud base images. That normally wouldn’t be a problem, but they don’t allow you to upload snapshots or base images, either. Don’t worry! We can think creatively and deploy the image via the cloud rescue environment.

Another challenge is that ignition, the CoreOS first boot configuration tool, does not support Hetzner’s metadata service at this time. I’m working on a pull request to add this support. We can get creative here and embed our ignition configuration inside the instance itself. That’s a bit annoying since all of your instances will get the same configuration, but that works out fine for my needs since I mainly care about ssh keys being present.

Generating ignition configuration

The ignition configuration allows us to specify ssh keys (and many other possible configurations) for the first boot. First, we start with a really basic butane configuration:

# hetzner-coreos.butane
variant: fcos
version: 1.4.0
passwd:
 users:
 - name: core
 groups:
 - wheel
 ssh_authorized_keys:
 - ssh-rsa AAAAB3NzaC1y...

You can use any username here that you prefer and add your public ssh key. Compile the configuration into an ignition file with butane:

butane hetzner-coreos.butane > config.ign

Keep this config.ign file handy because we need it before we snapshot our Fedora CoreOS image later.

Prepare the CoreOS snapshot

Although Hetzner doesn’t allow for uploading your own snapshot, you can replace the entire root disk of an instance from the rescue system. From there, we can snapshot the root disk storage and use that as our new base image.

Download and install the hcloud tool first. Follow the installation instructions to generate an API key and store it on your system.

Start by building a basic Fedora 34 instance where we can deploy the Fedora CoreOS image:

$ hcloud server create --datacenter nbg1-dc3 --image fedora-34 \
 --type cpx11 --name coreos-deployer

Activate the rescue system for the instance you just created and reboot into the rescue environment:

$ hcloud server enable-rescue coreos-deployer
1.148s [=================================] 100.00%
Rescue enabled for server 14168423 with root password: xxxxx
$ hcloud server reboot coreos-deployer
656ms [==================================] 100.00%
Server 14168423 rebooted

Use ssh to log into the rescue environment (user: root, password provided in the enable-rescue step). The rescue environment is in a ramdisk and we don’t have enough space to build the coreos-installer or download a raw disk, but we can get creative and stream the filesystem directly from a Fedora CoreOS image download.

I prefer to live dangerously and I run the testing release, but there are three releases available for download. Download, decompress, and write the image to the root disk all at once:

export COREOS_DISK="https://builds.coreos.fedoraproject.org/prod/streams/testing/builds/34.20210821.2.0/x86_64/fedora-coreos-34.20210821.2.0-metal.x86_64.raw.xz"
curl -sL $COREOS_DISK | xz -d | dd of=/dev/sda status=progress

This process should take one or two minutes to complete.

Embed the ignition configuration

We could hop out of rescue now and reboot right into Fedora CoreOS, but we need to provide SSH keys for our instance. The config.ign file we generated in the first section of this post will be deployed as /ignition/config.ign in the image.

Let’s find the boot partition:

# fdisk -l /dev/sda
Disk /dev/sda: 38.2 GiB, 40961572864 bytes, 80003072 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001

Device Start End Sectors Size Type
/dev/sda1 2048 4095 2048 1M BIOS boot
/dev/sda2 4096 264191 260096 127M EFI System
/dev/sda3 264192 1050623 786432 384M Linux filesystem « boot
/dev/sda4 1050624 5003230 3952607 1.9G Linux filesystem « root

Mount the boot partition and deploy the config.ign:

mount /dev/sda3 /mnt
mkdir /mnt/ignition
vi /mnt/ignition/config.ign « copy/paste your config.ign from earlier
umount /mnt

Let’s power off the instance to avoid booting it. We want all subsequent boots to be as clean as possible.

root@rescue ~ # poweroff

Make the snapshot

Create a snapshot from our powered off server:

hcloud server create-image --description fedora-34-coreos \
 --type snapshot coreos-deployer

This usually takes 1-2 minutes. Let’s get our image ID:

$ hcloud image list | grep fedora-34-coreos
46874212 snapshot - fedora-34-coreos 0.96 GB 40 GB Wed Sep 1 10:07:04 CDT 2021 -

Boot the instance

Now that we have a snapshot with our ignition configuration embedded in it, let’s make a new instance!

hcloud server create --datacenter nbg1-dc3 --image 46874212 --type cpx11 \
 --ssh-key personal_servers --name first-coreos-instance

Hetzner normally boots cloud images really quickly, but it takes a bit longer when booting from snapshots. I assume that they have the common base images cached on most hypervisors so they can provision them really quickly. The delay isn’t too bad here: the instances usually take about 90 seconds to boot.

Once it boots, you should see notes from ignition on bootup about your configuration:

Use ssh to login as the core user:

$ ssh core@INSTANCE_IP_ADDRESS
Fedora CoreOS 34.20210821.2.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

[core@localhost ~]$ podman --version
podman version 3.3.0

Wrapping up

Now that you have a snapshot made, you can delete your original instance (I called it coreos-deployer above) and just build off that snapshot whenever you need to. That should save you a few Euros per month. 💸

Once ignition has support for Hetzner’s metadata service, the extra step of embedding your configuration won’t be needed.

Also, if anyone from Hetzner is reading this post, I’d love to get Fedora CoreOS as one of the options for base images in your cloud! 🤗

Photo credit: Daniel Seßler on Unsplash