Containers on Major Hayden

Automatic container updates with Podman quadlets

major@mhtx.net (Major Hayden) — Fri, 19 Sep 2025 00:00:00 +0000

Running containers at home or in production often means juggling updates across multiple services. While orchestration platforms like Kubernetes handle this automatically, what about those simple deployments on a single host?

Podman’s quadlet system integrates containers directly with systemd, and when combined with automatic updates, you get a robust solution that keeps your containers fresh without manual intervention.

Let’s explore how to set up automatic container updates using Podman quadlets on Fedora, turning container management into a hands-off operation that just works.

Setting up a basic quadlet

First, let’s create a simple quadlet for running a Valkey database service under a user account. Quadlet files for user services live in ~/.config/containers/systemd/.

Create the directory if it doesn’t exist:

mkdir -p ~/.config/containers/systemd/

Then create your quadlet file:

# ~/.config/containers/systemd/valkey.container
[Container]
ContainerName=valkey
Image=docker.io/valkey/valkey:latest
Label=io.containers.autoupdate=registry
PublishPort=16379:6379
Volume=valkey_data:/data

[Service]
Restart=always

[Install]
WantedBy=default.target

The magic happens with the Label=io.containers.autoupdate=registry line. This label tells Podman that this container should be automatically updated when a newer image is available in the registry.

After creating the file, reload systemd and start your container:

systemctl --user daemon-reload
systemctl --user start valkey.service

Your container is now running as a user systemd service! Check its status with:

systemctl --user status valkey.service
podman ps

Enabling automatic updates

Podman ships with a systemd timer that checks for container updates. The podman-auto-update.timer runs daily by default, but you need to enable it for your user:

systemctl --user enable --now podman-auto-update.timer

You can check when the next update will run:

systemctl --user list-timers podman-auto-update.timer

When the timer triggers, it runs podman auto-update, which:

Checks all containers with the io.containers.autoupdate label
Pulls newer images if available
Restarts containers with the new image
Keeps the old image in case you need to roll back

Customizing update behavior

The io.containers.autoupdate label supports different values for various update strategies:

# Always pull the latest image from the registry
Label=io.containers.autoupdate=registry

# Only update if the local image changes (useful for locally built images)
Label=io.containers.autoupdate=local

You can also customize when updates occur by creating a timer override:

systemctl --user edit podman-auto-update.timer

Add these lines to run updates every 6 hours instead of daily:

[Timer]
OnCalendar=
OnCalendar=*-*-* 00,06,12,18:00:00

Monitoring updates

Track what’s happening with your automatic updates using journalctl:

# View recent auto-update logs
journalctl --user -u podman-auto-update.service -n 50

# Follow updates in real-time
journalctl --user -u podman-auto-update.service -f

# Check a specific container's restart history
journalctl --user -u valkey.service | grep Started

Connect Caddy to Porkbun

major@mhtx.net (Major Hayden) — Thu, 29 Feb 2024 00:00:00 +0000

I recently told a coworker about Caddy, a small web and proxy server with a very simple configuration. It also has a handy feature where it manages your TLS certificate for you automatically.

However, one problem I had at home with my CoreOS deployment is that I don’t have inbound network access to handle the certificate verification process. Most automated certificate vendors need to reach your web server to verify that you have control over your domain.

This post talks about how to work around this problem with domains registered at Porkbun.

DNS validation

Certificate providers usually default to verifying domains by making a request to your server and retrieving a validation code. If your systems are all behind a firewall without inbound access from the internet, you can use DNS validation instead.

The process looks something like this:

You tell the certificate provider the domain names you want on your certificate
The certificate provider gives you some DNS records to add wherever you host your DNS records
You add the DNS records
You get your certificates once the certificate provider verifies the records.

You can do this manually with something like acme.sh today, but it’s painful:

# Make the initial certificate request
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please

# Add your DNS records manually.

# Verify the DNS records and issue the certificates.
acme.sh --issue --dns -d example.com \
 --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew

# Copy the keys/certificates and configure your webserver.

We don’t want to live this way.

Let’s talk about how Caddy can help.

Adding Porkbun support to Caddy

Caddy is a minimal webserver and Porkbun support doesn’t get included by default. However, we can quickly add it via a simple container build:

FROM caddy:2.7.6-builder AS builder

RUN xcaddy build \
 --with github.com/caddy-dns/porkbun

FROM caddy:2.7.6

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

This is a two stage container build where we compile the Porkbun support and then use that new caddy binary in the final container.

We’re not done yet!

Automated Caddy builds with updates

I created a GitHub repository that builds the Caddy container for me and keeps it updated. There’s a workflow to publish a container to GitHub’s container repository and I can pull containers from there on my various CoreOS machines.

In addition, I use Renovate to watch for Caddy updates. New updates come through a regular pull request and I can apply them whenever I want.

Connecting to Porkbun

We start here by getting an API key to manage the domain at Porkbun.

Log into your Porkbun dashboard.
Click Details to the right of the domain you want to manage.
Look for API Access in the leftmost column and turn it on.
At the top right of the dashboard, click Account and then API Access.
Add a title for your new API key, such as Caddy, and click Create API Key.
Save the API key and secrey key that are displayed.

Open up your Caddy configuration file (the Caddyfile) and add some configuration:

{
 email me@example.com

 # Uncomment this next line if you want to get
 # some test certificates first.
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

 acme_dns porkbun {
 api_key pk1_******
 api_secret_key sk1_******
 }
}

example.com {
 handle {
 respond "Hello world!"
 }
}

Save the Caddyfile and restart your Caddy server or container. Caddy will immediately begin requesting your TLS certificates and managing your DNS records for those certificates. This normally finishes in less than 30 seconds or so during the first run.

If you don’t see the HTTPS endpoint working within a minute or two, be sure to check the Caddy logs. You might have a typo in a Porkbun API key or the domain you’re trying to modify doesn’t have the API Access switch enabled.

[!NOTE] Remember that Porkbun requires you to enable API access for each domain. API access is disabled at Porkbun by default.

That’s it! 🎉

Renewals

Caddy will keep watch over the certificates and begin the renewal process as the expiration approaches. It has a very careful retry mechanism that ensures your certificates are updated without tripping any rate limits at the certificate provider.

How I learned to stop worrying and love the CoreOS

major@mhtx.net (Major Hayden) — Fri, 13 Oct 2023 00:00:00 +0000

It’s quite clear that I’ve been on a CoreOS blogging streak lately. I keep getting asked by people inside and outside my company about what makes CoreOS special and why I’ve switched over so many workloads to it.

The answer is pretty basic. It makes my life easier.

I’m a Dad. I’m on the PTC (Parent Teacher Club) at one of my children’s schools. I volunteer as an IT person for a non-profit. I write software. I have other time consuming hobbies, such as ham radio, reading, and becoming a longer distance runner¹.

My available time for my own IT projects is extremely limited and CoreOS plays a part in keeping that part of my life as efficient as possible.

That’s what this blog post is about!

Updates

First and foremost, I love how CoreOS does updates. I encourage you to read the docs on this topic, but here’s a short explanation:

Updates are automatically retrieved and they’re loaded into a slot.
Your system reboots into the new update but your original OS tree remains in place.
Did the update boot? Awesome. You’re good to go.
Did something break? The system reverts back to the known good tree.

In this way, it’s a lot like your smartphone.

You have full control over when a node looks for an update and how often it checks for them. Check out the Zincati docs for tons of controls over updates and reboots.

Some of mine are timed so well that I set maintenance windows with my monitoring provider when I know an update might take place. The updates come through, monitoring shuts off, the node reboots, and monitoring comes back. The nodes almost always come back before the monitoring even alerts me.

It also removes the reminders I would set for myself to update packages and run reboots. I know that my CoreOS nodes will do this automatically, so I don’t need to think about it.

Also, updates are rarely ever impactful to my workloads since all of them are running inside containers. My containers come right back up as soon as the node finishes its reboot.

Toolbox

To be fair, you can get toolbox running on lots of different Linux distributions outside of CoreOS, but that’s the first place I ever used it. Toolbox, also called toolbx, gives you a utility container on your CoreOS node for all kinds of adminsitrative and diagnostic capabilities.

You might need a certain package for diagnosting a hardware issue or you might want to install some helpful utilities for the command line. Do that in a toolbox container. Just run toolbox enter and if you’ve never created a toolbox container before, you’ll get a Fedora container that matches your CoreOS release.

But it gets better.

Toolbox automatically saves your container when you’re done with it so all of your installed packages stay there for next time. Also, these containers have seamless access to anything you have in your home directory, including sockets. You’re running inside a container, but it’s almost like you’re running on the host itself inside your home directory. You get the best of both worlds.

Don’t want Fedora? You have lots of distribution options through toolbox. Read the details on custom images to create your own!

Layering

Okay, there are those situations where you really want a package on CoreOS and toolbox might not be sufficient. My muscle memory for vim is so strong and CoreOS only comes with vi.

You have a couple of options here:

Run sudo rpm-ostree install vim, reboot, and you have vim
Run sudo rpm-ostree install --apply-live vim and you have vim right now! (And it’s there after a reboot as well.)

When a new update comes down for the base OS from CoreOS, any packages you’ve added will be layered on the base image and available after a reboot. Layering is generally chosen as a last resort option for adding packages to the system but you shouldn’t run into issues if you’re installing small utilities or command line tools.

Declarative provisioning

If you’ve provisioned Linux distributions on cloud instances in the past, you’ve likely provided metadata that cloud-init uses to provision your system. CoreOS has something that acts a lot earlier in the boot process and has more power to get things done: ignition.

There’s a handy butane file forma that you use for writing your configuration. You use the butane utility to get it into ignition format. The ignition format is highly compressed to ensure you can fit your configuration into most cloud providers’ metadata fields.

For a real example of what you can do with ignition, check out my quadlets post where I provisioned an entire Wordpress container stack using a single ignition file.

There’s lots of documentation for writing butane configuration files for common situations. It’s easy to add files, configure Wireguard, set up users, and launch containers immediately on the first boot.

Pets and cattle

CoreOS works well for systems that I only need online for a short time. These might be situations where I need to test a few containers and throw it away. There’s no OS to mess with and no updates to worry about.

It also works well for systems that I keep online for a long time. I have a few physical systems at home that run CoreOS and they’ve been extremely stable. I also have cloud instances on Hetzner, VULTR, and Digital Ocean that have run CoreOS for months without issues.

Quadlets might make me finally stop using docker-compose

major@mhtx.net (Major Hayden) — Mon, 25 Sep 2023 00:00:00 +0000

I’ve written a lot about containers on this blog. Why do I love containers so much?

They start quickly
They make your workloads portable
They disconnect your application stack from the OS that runs underneath
You can send your application through CI as a single container image
You can isolate workloads on the network and limit their resource usage much like a VM

However, I’m still addicted to docker-compose. Can podman’s quadlets change that?

Yes, I think they can.

What’s a quadlet?

Podman introduced support for quadlets in version 4.4 and it’s a simpler way of letting systemd manage your containers. There was an option in the past to have podman generate systemd unit files, but those were unwieldy and full of podman command line options inside a unit file. These unit files weren’t easy to edit or even parse with eyeballs.

Quadlets make this easier by giving you a simple ini-style file that you can easily read and edit. This blog post will include some quadlets later, but here’s an example one for Wordpress:

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Lots of the lines under [Container] should look familiar to most readers who have worked with containers before. However, there’s something new here.

Check out the AutoUpdate=registry line. This tells podman to keep your container updated on a regular basis with the upstream container registry. I’ve used watchtower in the past for this, but it requires a privileged container and it’s yet another external dependency.

Also, at the very end, you’ll see a WantedBy line. This is a great place to set up container dependencies. In this example, the container that runs caddy (a web server) can’t start until Wordpress is up and running.

So why not stick with docker-compose?

There’s no denying that docker-compose is an awesome tool. You specify the desired outcome, tell it to bring up containers, and it gets containers into the state you specified. It handles volumes, networks, and complicated configuration without a lot of legwork. The YAML files are pretty easy to read, too.

However, as with watchtower, that’s another external dependency.

My container deployments are often done at instance boot time and I don’t make too many changes afterwards. I found myself using docker-compose for the initial deployment and then I didn’t really use it again.

Why not remove it entirely and use what’s built into CoreOS already?

Quaint quadlets quickly!

Before we start, we’re going to need a few things:

An easy to read butane configuration which gets transformed into a tiny ignition configuration for CoreOS
Some quadlets
Extra system configuration
A cloud provider with CoreOS images (using VULTR for this)

I’ve packed all of these items into my quadlets-wordpress repository to make it easy. Start by looking at the config.butane file.

Let’s break it down here. First up, we add an ssh key for the default core user.

variant: fcos
version: 1.5.0
passwd:
 users:
 - name: core
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0

Next up, we enable the podman-auto-update.timer so we get container updates automatically:

storage:
 links:
 - path: /home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer
 target: /usr/lib/systemd/user/podman-auto-update.timer
 user:
 name: core
 group:
 name: core

Next is the long files section:

 files:
 # Ensure the `core` user can keep processes running after they're logged out.
 - path: /var/lib/systemd/linger/core
 mode: 0644
 
 # Allow caddy to listen on 80 and 443.
 # Allow it to ask for bigger network buffers, too.
 - path: /etc/sysctl.d/90-caddy.conf
 contents:
 inline: |
 net.ipv4.ip_unprivileged_port_start = 80
 net.core.rmem_max=2500000
 net.core.wmem_max=2500000

 # Set up an an environment file that containers can read to configure themselves.
 - path: /home/core/.config/containers/containers-environment
 contents:
 inline: |
 MYSQL_DATABASE=wordpress
 MYSQL_USER=wordpress
 MYSQL_ROOT_PASSWORD=mariadb-needs-a-secure-password
 MYSQL_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_HOST=mariadb
 WORDPRESS_DB_USER=wordpress
 WORDPRESS_DB_PASSWORD=wordpress-needs-a-secure-password
 WORDPRESS_DB_NAME=wordpress
 mode: 0644

 # Deploy the caddy configuration file from the repository.
 - path: /home/core/.config/caddy/Caddyfile
 contents:
 local: caddy/Caddyfile
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add some named volumes for caddy and wordpress.
 - path: /home/core/.config/containers/systemd/caddy-config.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/caddy-data.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core
 - path: /home/core/.config/containers/systemd/wordpress.volume
 contents:
 inline: |
 [Volume]
 user:
 name: core
 group:
 name: core

 # Create a network for all the containers to use and enable the
 # DNS plugin. This allows containers to find each other using
 # the container names.
 - path: /home/core/.config/containers/systemd/wordpress.network
 contents:
 inline: |
 [Network]
 DisableDNS=false
 Internal=false
 user:
 name: core
 group:
 name: core

 # Add the wordpress container.
 - path: /home/core/.config/containers/systemd/wordpress.container
 contents:
 local: quadlets/wordpress.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the MariaDB container.
 - path: /home/core/.config/containers/systemd/mariadb.container
 contents:
 local: quadlets/mariadb.container
 mode: 0644
 user:
 name: core
 group:
 name: core

 # Add the caddy container.
 - path: /home/core/.config/containers/systemd/caddy.container
 contents:
 local: quadlets/caddy.container
 mode: 0644
 user:
 name: core
 group:
 name: core

The Caddyfile is also in the repository and will be deployed by the butane configuration shown above.

We can go through each quadlet in detail. First up is MariaDB. We tell systemd that the wordpress container will want to have this one started first.

[Unit]
Description=MariaDB Quadlet

[Container]
Image=docker.io/library/mariadb:11
ContainerName=mariadb
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=mariadb.volume:/var/lib/mysql
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=wordpress.service multi-user.target default.target

The wordpress quadlet is much the same as the MariaDB one, but we tell systemd that caddy will want wordpress started first.

[Unit]
Description=Wordpress Quadlet

[Container]
Image=docker.io/library/wordpress:fpm
ContainerName=wordpress
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=wordpress.volume:/var/www/html
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=caddy.service multi-user.target default.target

Finally, the caddy quadlet contains four volumes and some published ports. These ports will be published to the container host. Also, you’ll note that the wordpress volume is mounted here, too. This is because caddy can serve static files much faster than wordpress can.

[Unit]
Description=Caddy Quadlet

[Container]
Image=docker.io/library/caddy:latest
ContainerName=caddy
AutoUpdate=registry
EnvironmentFile=/home/core/.config/containers/containers-environment
Volume=caddy-data.volume:/data
Volume=caddy-config.volume:/config
Volume=/home/core/.config/caddy/Caddyfile:/etc/caddy/Caddyfile:Z
Volume=wordpress.volume:/var/www/html
PublishPort=80:80
PublishPort=443:443
Network=wordpress.network

[Service]
Restart=always
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target default.target

Launch the quadlets

There’s a launch script that ships this configuration to VULTR and launches a CoreOS instance:

#!/bin/bash
# This command starts up a CoreOS instance on Vultr using the vultr-cli
vultr-cli instance create \
 --os 391 \
 --plan vhp-1c-1gb-amd \
 --region dfw \
 --notify true \
 --ipv6 true \
 -u "$(butane --files-dir . config.butane)" \
 -l "coreos-$(date "+%s")"

To launch an instance, get your VULTR API key first. Then install vultr-cli and butane:

$ sudo dnf -y install butane vultr-cli

After launch, check to see what your containers are doing:

[core@vultr ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
afa2d6501593 docker.io/library/caddy:latest caddy run --confi... 54 seconds ago Up 53 seconds 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp caddy
460426f39e6c docker.io/library/mariadb:11 mariadbd 35 seconds ago Up 35 seconds mariadb
92ece6538d5a docker.io/library/wordpress:fpm php-fpm 28 seconds ago Up 29 seconds wordpress

We should be able to talk to wordpress through caddy on port 80:

[core@vultr ~]$ curl -si http://localhost/wp-admin/install.php | head -n 25
HTTP/1.1 200 OK
Cache-Control: no-cache, must-revalidate, max-age=0
Content-Type: text/html; charset=utf-8
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Server: Caddy
X-Powered-By: PHP/8.0.30
Date: Mon, 25 Sep 2023 21:43:40 GMT
Transfer-Encoding: chunked

<!DOCTYPE html>
<html lang="en-US" xml:lang="en-US">
<head>
	<meta name="viewport" content="width=device-width" />
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
	<meta name="robots" content="noindex,nofollow" />
	<title>WordPress &rsaquo; Installation</title>
	<link rel='stylesheet' id='dashicons-css' href='http://localhost/wp-includes/css/dashicons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='http://localhost/wp-includes/css/buttons.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='http://localhost/wp-admin/css/forms.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='http://localhost/wp-admin/css/l10n.min.css?ver=6.3.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='http://localhost/wp-admin/css/install.min.css?ver=6.3.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui language-chooser">
<p id="logo">WordPress</p>

Awesome! 🎉

Managing containers

Containers will automatically update on a schedule and you can check the timer:

[core@vultr ~]$ systemctl status --user podman-auto-update.timer
● podman-auto-update.timer - Podman auto-update timer
 Loaded: loaded (/usr/lib/systemd/user/podman-auto-update.timer; enabled; preset: disabled)
 Active: active (waiting) since Mon 2023-09-25 21:41:31 UTC; 3min 14s ago
 Trigger: Tue 2023-09-26 00:04:46 UTC; 2h 20min left
 Triggers: ● podman-auto-update.service

Sep 25 21:41:31 vultr.guest systemd[1786]: Started podman-auto-update.timer - Podman auto-update timer.

Quadlets are just regular systemd units:

[core@vultr ~]$ systemctl list-units --user | grep -i Quadlet
 caddy.service loaded active running Caddy Quadlet
 mariadb.service loaded active running MariaDB Quadlet
 wordpress.service loaded active running Wordpress Quadlet

As an example, you can make changes to caddy’s config file and restart it easily:

[core@vultr ~]$ systemctl restart --user caddy
[core@vultr ~]$ systemctl status --user caddy
● caddy.service - Caddy Quadlet
 Loaded: loaded (/var/home/core/.config/containers/systemd/caddy.container; generated)
 Drop-In: /usr/lib/systemd/user/service.d
 └─10-timeout-abort.conf
 Active: active (running) since Mon 2023-09-25 21:46:28 UTC; 5s ago
 Main PID: 2652 (conmon)
 Tasks: 18 (limit: 1023)
 Memory: 15.1M
 CPU: 207ms

If you need to change a quadlet’s configuration, just open up the configuration file in your favorite editor under ~/.config/containers/systemd, reload systemd, and restart the container:

$ vi ~/.config/containers/systemd/caddy.container

--- make your edits and save the quadlet configuration ---

$ systemctl daemon-reload --user
$ systemctl restart --user caddy

Enjoy!

Fixing a ghost database migration failure

major@mhtx.net (Major Hayden) — Thu, 31 Aug 2023 00:00:00 +0000

I love learning about the behind the scenes aspects of just about everything. I do ham radio, I self-host lots of my personal infrastructure, and I’ve been learning more about the math behind the stock market for the last year or two.

That led me to start a blog on Ghost to share my findings with others. I started Theta Nerd¹ earlier this summer.

My deployment looked great when I started! Everything was automatically updated with watchtower and running with docker-compose on Fedora CoreOS. (Click these links to read the posts on both topics!)

However, I woke up one morning to my monitoring going off and my site was down. 😱

Why is the site down?

Anyone who has worked in IT knows this sinking feeling. Something is down, you don’t know why, and you suspect the worst possible scenarios.

The instance hosting the blog was online and responsive, so I started digging into the logs with docker-compose logs. I suddenly found a wall of text in the logs for the Ghost container:

[2023-08-03 11:10:16] INFO Adding members.email_disabled column
[2023-08-03 11:10:16] INFO Setting email_disabled to true for all members that have their email on the suppression list
[2023-08-03 11:10:16] INFO Setting nullable: stripe_products.product_id
[2023-08-03 11:10:16] INFO Adding table: donation_payment_events
[2023-08-03 11:10:16] INFO Rolling back: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible..
[2023-08-03 11:10:16] INFO Dropping table: donation_payment_events
[2023-08-03 11:10:16] INFO Dropping nullable: stripe_products.product_id with foreign keys disabled
[2023-08-03 11:10:16] INFO Setting email_disabled to false for all members
[2023-08-03 11:10:16] INFO Removing members.email_disabled column
[2023-08-03 11:10:16] INFO Rollback was successful.
[2023-08-03 11:10:16] ERROR alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.

alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
{"config":{"transaction":false},"name":"2023-07-27-11-47-49-create-donation-events.js"}
"Error occurred while executing the following migration: 2023-07-27-11-47-49-create-donation-events.js"
Error ID:
 300
Error Code: 
 ER_FK_INCOMPATIBLE_COLUMNS
----------------------------------------
Error: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
 at /var/lib/ghost/versions/5.57.2/node_modules/knex-migrator/lib/index.js:1032:19
 at Packet.asError (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packets/packet.js:728:17)
 at Query.execute (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/commands/command.js:29:26)
 at Connection.handlePacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:478:34)
 at PacketParser.onPacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:97:12)
 at PacketParser.executeStart (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packet_parser.js:75:16)
 at Socket.<anonymous> (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:104:25)
 at Socket.emit (node:events:513:28)
 at addChunk (node:internal/streams/readable:315:12)
 at readableAddChunk (node:internal/streams/readable:289:9)
 at Socket.Readable.push (node:internal/streams/readable:228:10)
 at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

Ah, so a failed database migration in the upgrade to 5.57.2 is the culprit! 👏

I brought the site back online quickly by changing the container version for Ghost back to the previous version (5.55.2).

Why did the database migration fail?

The error message from above boils down to this:

Error: alter table `donation_payment_events` add constraint 
`donation_payment_events_member_id_foreign` foreign key (`member_id`)
references `members` (`id`) on delete SET NULL - Referencing column
'member_id' and referenced column 'id' in foreign key constraint 
'donation_payment_events_member_id_foreign' are incompatible.

Adjusting the donation_payment_events.member_id column to be a foreign key of members.id is failing because they are incompatible types. However, as I examined both tables, both were regular varchar(24) columns without anything special attached to them:

mysql> describe members;
+------------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------------+---------------+------+-----+---------+-------+
| id | varchar(24) | NO | PRI | NULL | |
| uuid | varchar(36) | YES | UNI | NULL | |
| email | varchar(191) | NO | UNI | NULL | |
| status | varchar(50) | NO | | free | |
| name | varchar(191) | YES | | NULL | |
| expertise | varchar(191) | YES | | NULL | |
| note | varchar(2000) | YES | | NULL | |
| geolocation | varchar(2000) | YES | | NULL | |
| enable_comment_notifications | tinyint(1) | NO | | 1 | |
| email_count | int unsigned | NO | | 0 | |
| email_opened_count | int unsigned | NO | | 0 | |
| email_open_rate | int unsigned | YES | MUL | NULL | |
| last_seen_at | datetime | YES | | NULL | |
| last_commented_at | datetime | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| created_by | varchar(24) | NO | | NULL | |
| updated_at | datetime | YES | | NULL | |
| updated_by | varchar(24) | YES | | NULL | |
+------------------------------+---------------+------+-----+---------+-------+
18 rows in set (0.00 sec)

Going upstream

I went to Ghost’s GitHub repository and opened an issue with as much data as I can find.

One of the first replies mentioned something about database collations. Long story short, collations describe how databases handle sorting and comparing data for different languages. Comparing some languages to other languages can be particularly challenging and this can lead to problems.

I made a switch from MariaDB to MySQL recently for the blog. Could that be related?

More searching

I figured that I wasn’t the first one to stumble into this problem, and sure enough – I wasn’t! There’s a great blog post about a broken migration from MySQL 5 to 8 with Ghost.

In short, it required several steps to fix it:

Stop the Ghost container
Back up the database first (always a good idea)
Do a quick find/replace on the dumped database to change the collations
Drop the ghost database from the database 😱
Import the database back into MySQL
Start Ghost again

Dropping databases always makes me pause, but that’s what backups are for! 😉

How I fixed it

In my case, my MySQL container is called ghostmysql and my Ghost database is ghostdb. Then I made a backup of the database using mysqldump:

sudo docker-compose exec ghostmysql mysqldump \
 -u root -psuper-secret-password ghostdb > backup-ghost-db.sql

Next, I copied the SQL file to another directory just in case I accidentally deleted this backup with an errant command.

cp backup-ghost-db.sql ../

Then I made a copy of the SQL file in the current directory and ran the find and replace on that copy. This changes the collations from the wrong one, utf8mb4_general_ci, to the right one, utf8mb4_0900_ai_ci²:

cp backup-ghost-db.sql backup-ghost-db-new.sql
sed -i 's/utf8mb4_general_ci/utf8mb4_0900_ai_ci/g' \
 backup-ghost-db-new.sql

Now I have the collations right for importing the database back into MySQL. But first, I have to drop the existing database. This is a good time to double check your backups!

sudo docker-compose exec ghostmysql mysql -u root \
 -psuper-secret-password
mysql> DROP DATABASE ghostdb;

Now we can import the modified backup:

cat backup-ghost-db-new.sql | sudo docker-compose exec -T \
 ghostmysql mysql -u root -psuper-secret-password ghostdb

Start all the containers:

sudo docker-compose up -d

Ghost was back online with the older version and everything looked good! I updated my docker-compose.yaml back to use latest for the Ghost version and ran sudo docker-compose up -d once more.

Within seconds, the new container image was in place and the container was running! Both migrations completed in seconds and the blog was back online with the newest version. 🎉

Theta is one of many financial Greeks that measure certain aspects of options contracts in the market. It’s also a letter in the Greek alphabet. ↩︎
The default collation in MySQL 8 is utf8mb4_0900_ai_ci. ↩︎

Deploy a containerized Ghost blog 👻

major@mhtx.net (Major Hayden) — Tue, 27 Jun 2023 00:00:00 +0000

There’s no shortage of options for starting a self-hosted blog. Wordpress might be chosen most often, but I stumbled upon Ghost recently and their performance numbers really got my attention.

I prefer deploying most things in containers these days with Fedora CoreOS. Luckily, the Ghost stack doesn’t demand a lot of infrastructure:

Ghost itself
MySQL 8+ (I went with MariaDB 11.x)
A web server out front
TLS certificate

[!NOTE] Although I chose MariaDB for the database here, Ghost recommends MySQL and will throw a warning in the admin panel if you’re using something else. I haven’t had any issues so far, but you’ve been warned. 💣

I picked Caddy for the webserver since it’s so small and the configuration is tremendously simple.

Launch CoreOS

Fedora CoreOS offers lots of cloud options for launching it immediately. Many public clouds already have CoreOS images available, but I love Hetzner’s US locations and I already had a CoreOS image loaded up in my account.

🇩🇪 Want CoreOS at Hetzner? There’s a blog post for that!

Once your CoreOS instance is running, connect to the instance over ssh and ensure the docker.service starts on each boot:

sudo systemctl enable --now docker.service

This ensures that containers come up on each reboot. CoreOS has a podman socket that listens for docker-compatible connections, but that doesn’t help with reboots.

Perhaps I’m old fashioned, but I still enjoy using docker-compose for container management. I like how I can declare what I want and let docker-compose sort out the rest.

Let’s install docker-compose on the CoreOS instance now:

# Check the latest version in the GitHub repo before starting!
# https://github.com/docker/compose
curl -LO https://github.com/docker/compose/releases/download/v2.19.0/docker-compose-linux-x86_64

# Install docker-compose and make it executable.
sudo mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
sudo chown +x /usr/local/bin/docker-compose

Verify that docker-compose is ready to go:

$ docker-compose --version
Docker Compose version v2.19.0

Preparing Caddy

Caddy uses a configuration file called a Caddyfile and we need that in place before we deploy the other containers. Within my home directory, I created a directory called caddy:

mkdir caddy

Then I added the Caddyfile inside the directory:

{
 # Your email for LetsEncrypt warnings/notices.
 email youremail@domain.com 

 # Staging LetsEncrypt server to use while testing.
 # Uncomment this before going to production!
 acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

# Basic virtual host definition to feed traffic into the
# Ghost container when it arrives.
example.com {
 reverse_proxy ghost:2368
}

# OPTIONAL: Redirect traffic to 'www' to the bare domain.
www.example.com {
 redir https://example.com{uri}
}

This configuration sets up LetsEncrypt certificates automatically from the staging server for now. Once we know our configuration is working well, we can comment out the acme_ca line above and get production TLS certificates.

At this point, you need a DNS record pointed to your server so you can get a certificate. You have some options:

If the site is entirely new, just point the root domain name to your CoreOS instance. Use that domain in the configuration above and later in the deployment.
If you’re migrating from an existing site, choose a subdomain off your main domain to use. If your website is example.com, use something like test.example.com or new.example.com to get Ghost up and running. It’s really easy to change this later.

Now we’re ready for the rest of the deployment.

Deploying containers

Here’s the docker-compose.yml file I’m using:

---
version: '3.8'
services:

 # OPTIONAL
 # Watchtower monitors all running containers and updates
 # them when the upstream container repo is updated.
 watchtower:
 image: docker.io/containrrr/watchtower:latest
 container_name: watchtower
 restart: unless-stopped
 hostname: coreos-ghost-deployment
 environment:
 - WATCHTOWER_CLEANUP=true
 - WATCHTOWER_POLL_INTERVAL=3600
 command:
 - --cleanup
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 privileged: true

 # Caddy acts as our external-facing webserver and handles
 # getting TLS certs from LetsEncrypt.
 caddy:
 image: caddy:latest
 container_name: caddy
 depends_on:
 - ghost
 ports:
 - 80:80
 - 443:443
 restart: unless-stopped
 volumes:
 - ./caddy/Caddyfile:/etc/caddy/Caddyfile:Z
 - ghost:/var/www/html
 - caddy_data:/data
 - caddy_config:/config

 # The Ghost blog software itself
 ghost:
 image: docker.io/library/ghost:5
 container_name: ghost
 restart: always
 depends_on:
 - ghostdb
 environment:
 url: https://example.com
 database__client: mysql
 database__connection__host: ghostdb
 database__connection__user: ghost
 database__connection__password: GHOST_PASSWORD_FOR_MARIADB
 database__connection__database: ghostdb
 volumes:
 - ghost:/var/lib/ghost/content

 # Our MariaDB database
 ghostdb:
 image: docker.io/library/mariadb:11
 container_name: ghostdb
 restart: always
 environment:
 MYSQL_ROOT_PASSWORD: A_SECURE_ROOT_PASSWORD
 MYSQL_USER: ghost
 MYSQL_PASSWORD: GHOST_PASSWORD_FOR_MARIADB
 MYSQL_DATABASE: ghostdb
 volumes:
 - ghostdb:/var/lib/mysql

volumes:
 caddy_config:
 caddy_data:
 ghost:
 ghostdb:

I love watchtower but that step is completely optional. It does require some elevated privileges to talk to the podman socket, so keep that in mind if you choose to use it.

Our ghostdb container starts first, followed by ghost, and then caddy. That follows the depends_on configuration keys shown above.

There are two steps to take now:

Replace GHOST_PASSWORD_FOR_MARIADB and A_SECURE_ROOT_PASSWORD above with better passwords. 😉
Also, set the url parameter for the ghost container to your blog’s domain name.

Once all of that is done, let’s let docker-compose do the heavy lifting:

sudo docker-compose up -d

Let’s verify that our containers are running:

$ sudo docker-compose ps
NAME IMAGE COMMAND SERVICE 
caddy caddy:latest "caddy run --config …" caddy 
ghost docker.io/library/ghost:5 "docker-entrypoint.s…" ghost 
ghostdb docker.io/library/mariadb:11 "docker-entrypoint.s…" ghostdb 
watchtower docker.io/containrrr/watchtower:latest "/watchtower --clean…" watchtower

Awesome! 👏

Ghost initial setup

With all of your containers running, browse to https://example.com/ghost/ Just add /ghost/ to the end of your domain name to reach the admin panel. Create your admin account there with a good password.

If everything looks good, run back to your Caddyfile and comment out the acme_ca line:

{
 # Your email for LetsEncrypt warnings/notices.
 email youremail@domain.com 

 # Staging LetsEncrypt server to use while testing.
 # Uncomment this before going to production!
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

Restart the caddy container to get a production LetsEncrypt certificate on the site:

sudo docker-compose restart caddy

Customizing Ghost

Ghost looks for lots of environment variables to determine its configuration and you can set these in your docker-compose.yml file. Although some configuration items are easy, like url, some are nested and get more complicated. For these, you can use double underscores __ to handle the nesting.

As an example, we already used database__connection__host in the docker-compose.yaml, and that’s the equivalent to this nested configuration:

"database": {
 "connection": {
 "host": "..."
 }
}

If you’re deploying in containers, it’s a good idea to configure Ghost via environment variables. This ensures that your docker-compose.yml is authoritative for the Ghost deployment. You can exec into the container, adjust the config file on disk, and restart Ghost, but then you have to remember where you configured each item. 🥵

Switching to production domain

If you used a temporary domain to get everything configured and you’re ready to use your production domain, follow these steps:

Open your Caddyfile and replace all instances of the testing domain with the production domain
Restart caddy: sudo docker-compose restart caddy
Edit the docker-compose.yml and change the url key in the ghost container to the production domain
Apply the configuration with sudo docker-compose up -d

Enjoy your new automatically-updating Ghost blog deployment! 👻

Launch a watchtower container via podman quadlets

major@mhtx.net (Major Hayden) — Wed, 31 May 2023 00:00:00 +0000

Most of my container workloads run on independent CoreOS cloud instances that I treat like pets. Keeping containers update remains a constant battle, but it’s still easier than running kubernetes.

I wrote about using watchtower in the past to keep containers updated. It’s a simple container that does a few important things:

It monitors (via docker/podman socket) the running containers on the host
It tracks the versions/tags of each container image
It looks for updated versions of the container image in their upstream repositories
Based on a configurable schedule, it pulls a new container image and restarts the container for updates

I encourage you to read more about watchtower on GitHub. There’s plenty you can configure, including update intervals, how updates are handled, and how you can get notifications when an update happens.

My new deployments always need watchtower running. Luckily, we can combine Fedora CoreOS’ initial provisioning system, called ignition, with podman’s new quadlet feature and launch watchtower automatically on the first boot.

Quadlets

So what’s a quadlet?

The blog post explains it well by making containers more declarative via a familiar systemd syntax. Here’s an example .container file from the post:

[Unit]
Description=The sleep container
After=local-fs.target

[Container]
Image=registry.access.redhat.com/ubi9-minimal:latest
Exec=sleep 1000

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target

You can toss this into $HOME/.config/containers/systemd/mysleep.container for rootless user containers or in /etc/containers/systemd/mysleep.container for a container running as root.

Configure a quadlet on boot

As I mentioned earlier, I want a watchtower container running on my CoreOS nodes at first boot. Let’s start with a fairly basic butane file:

variant: fcos
version: 1.4.0
passwd:
 users:
 - name: major
 groups:
 - wheel
 - sudo
 ssh_authorized_keys:
 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDyoH6gU4lgEiSiwihyD0Rxk/o5xYIfA3stVDgOGM9N0
storage:
 files:
 - path: /etc/containers/systemd/watchtower.container
 contents:
 inline: |
 [Unit]
 Description=Watchtower container updater
 Wants=network-online.target
 After=network-online.target

 [Container]
 ContainerName=watchtower
 Image=ghcr.io/containrrr/watchtower:1.5.3@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca
 Environment=WATCHTOWER_CLEANUP=true
 Environment=WATCHTOWER_POLL_INTERVAL=3600
 Volume=/var/run/docker.sock:/var/run/docker.sock
 SecurityLabelDisable=true

 [Install]
 WantedBy=multi-user.target default.target

Let’s break this file down:

I start by adding a user named major that has administrative privileges an an ssh key (this is optional, but I like using my own username rather than core)
The quadlet unit file lands in /etc/containers/systemd/watchtower.container and starts at boot time

The quadlet file has some important configurations:

I added environment variables to clean up outdated container images and check for updates once an hour
The podman socket is mounted inside the watchtower container
Security labels are disabled to allow for communication with the podman socket

[!NOTE] Mounting the podman socket and disabling security labels is not an ideal security approach. However, I’ve found that watchtower’s configuration and automation fits my needs really well and I retreive the image from a trusted source. If this won’t work for you, you can use podman’s built-in auto-update feature instead.

From here, we convert the butane configuration into an ignition configuration. I’m launching this CoreOS node on VULTR, so I’ve named my files accordingly:

$ butane vultr-coreos.butane > vultr-coreos.ign

Let’s go 🚀

I’m using VULTR’s CLI here in Fedora, but you can do the same steps via VULTR’s portal if needed. Just paste in the ignition configuration into the large text box before launch.

# Install vultr-cli in Fedora
sudo dnf install vultr-cli

# Launch the instance
vultr-cli instance create --region dfw --plan vhp-2c-2gb-amd \
 --os 391 --label coreos-dfw-1 --host coreos-dfw-1 \
 --userdata "$(cat vultr-coreos.ign)"

Let’s see how the container is doing:

$ ssh major@COREOS_HOST

Fedora CoreOS 38.20230430.3.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[major@coreos-dfw-1 ~]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0024712c95d ghcr.io/containrrr/watchtower@sha256:a924a9aaef50016b7e69c7f618c7eb81ba02f06711558af57da0f494a76e7aca About a minute ago Up About a minute watchtower

[major@coreos-dfw-1 ~]$ sudo podman logs watchtower
time="2023-05-31T14:01:12Z" level=info msg="Watchtower 1.5.3"
time="2023-05-31T14:01:12Z" level=info msg="Using no notifications"
time="2023-05-31T14:01:12Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2023-05-31T14:01:12Z" level=info msg="Scheduling first run: 2023-05-31 15:01:12 +0000 UTC"
time="2023-05-31T14:01:12Z" level=info msg="Note that the first check will be performed in 59 minutes, 59 seconds"

Awesome! 🥳

My system rebooted for an ostree update shortly after provisioning and the container came up automatically both times.

CoreOS as a pet

major@mhtx.net (Major Hayden) — Thu, 25 May 2023 00:00:00 +0000

Anyone working with containers has likely heard of CoreOS by this point Haven’t heard about it? Don’t despair. I’ll catch you up on what you missed.

Fedora CoreOS offers a really fast pathway to running containers on hardware, in virtual machines, or in clouds. It delivers a lightweight operating system with all of the container technology that you need for running simple containers or launching a kubernetes deployment.

But that’s not the best part.

CoreOS really shines due to its immutable OS layer¹. The OS underneath your containers ships as a single unit and it automatically updates itself much like your mobile phone. An update rolls down, CoreOS sets it up as a secondary OS, reboots into that new update, and rolls back to the original one if there were any issues.

Many people use CoreOS as the workhorse underneath kubernetes. Red Hat uses it underneath OpenShift as well. It’s even supported by the super light weight kubernetes distribution k3s.

But can you use CoreOS as a pet type instance that you use and maintain for long periods of time just like any other server? Absolutely!

What’s this pet stuff about?

Whether you like it or not, there’s a cattle versus pets paradigm that took hold in the world of IT at some point. The basic ideas are these:

When you take care of cattle, you take care of them as a group. Losing one or more of them would make you sad, but you know you have many others.
As for pets, you spend a lot of time taking care of them and playing with them. If you lost one, it would be devastating.

A fleet of web servers could be treated like cattle. Keep lots of them online and replace any instances that have issues.

On the other hand, databases or tier zero systems (everyone feels if it they went down) are like pets. You carefully build, maintain, and monitor these.

How does CoreOS fit in?

Many people do use CoreOS as a container hosting platform as part of a bigger system. It works really well for that. But it’s great as a regular cloud server, too.

You can run a single node CoreOS deployment and manage containers via the tools that you know and love. For example, docker-compose works great on CoreOS. I even used it to host my own Mastodon deployment.

You can also load up more user-friendly tools such as portainer to manage containers in a browser.

My development tools are missing!

😱 No vim? This is too minimal! What are we going to do?

Luckily CoreOS comes with toolbox. 🧰

Toolbox gives you the ability to run a utility container on the system with some handy benefits:

Toolbox environments have seamless access to the user’s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc..

This means that the toolbox feels like a second OS on the system and it has all of the elevated privileges that you need to do your work. Simply run toolbox enter, follow the prompts, and you’ll end up with a Fedora toolbox that matches your CoreOS version. Need a different version, such as Fedora Rawhide? Just specify the Fedora release you want on the prompt:

$ toolbox enter --release 39 
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/fedora-toolbox:39 (500MB)? [y/N]: y

Welcome to the Toolbox; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢[major@toolbox ~]$

Look at the toolbox create --help output to see how to create lots of different toolbox containers with different names and releases. If you go overboard and need to delete some toolboxes, just list your toolboxes with toolbox list and follow it up with toolbox rm.

My tool won’t work in the toolbox.

Some applications have issues running inside a container, even one that has elevated privileges on the system. CoreOS offers an option for layering packages on top of the underlying immutable OS.

Simply run rpm-ostree install PACKAGE to layer a package on top of the OS. When rpm-ostree runs, it creates a new layer and sets that layer to be active on the next boot. That means that you need to reboot before you can use the package.

Don’t want to reboot? There’s another option, but I recommend against it if you can avoid it².

You can apply a package layer live on the system without a reboot with the --apply-live flag. Installing a package like mtr would look like this:

$ sudo rpm-ostree install --apply-live mtr

As soon as rpm-ostree finishes its work, mtr should be available on the system for you to use.

How do updates work?

There are two main technologies at work here.

First, zincati checks for updates to your immutable OS tree. It runs on a configurable schedule that you can adjust based on your preferences.

Second, rpm-ostree handles the OS layers and switches between them at boot time. If you’re running off layer A and an update comes down (layer B), that layer is written to the disk and activated on the next boot. Should there be any issues booting up layer B later, rpm-ostree switches the system back to layer A. In these situations, your downtime might be extended a bit due to two reboots. Your system will come back up with the original OS layer activated.

You also get a choice of update streams. Want to live a bit more on the edge? Go for next or testing. You’re on the stable stream by default.

Although I haven’t landed in this situation, it’s possible that the system boots into a new update where you notice a problem that doesn’t affect the boot. You can manually roll back to fix it.

I have more questions.

Your first stop should be the Fedora CoreOS docs. There are also lots of ways to contact the development team and talk with the community.

Love the idea of an immutable OS but you wish you had it for your desktop or laptop? Go check out Fedora Silverblue. 💻

Okay, so it’s mostly immutable. You can edit configuration in /etc and you can layer more packages on top of the base OS layer if you need them. However, CoreOS maintainers discourage adding layered packages if you can avoid it. ↩︎
When you apply some packages and make them available immediately, you may lose track of which ones were applied live and which ones are available on the next reboot. Things can get a bit confusing if you suddenly change your mind about applying a package live or not. ↩︎

Automatic container updates with watchtower

major@mhtx.net (Major Hayden) — Wed, 04 Jan 2023 00:00:00 +0000

Keeping things updated quickly becomes a monotonous task. I’m surrounded by devices that demand updates on different frequencies. Phones, computers, tables, cloud instances, containers, and even my car need constant attention for updates that improve security or fix bugs. (Sometimes the updates cause bugs, but let’s forget about those for now) 😉

My container infrastructure runs on Fedora CoreOS and it updates itself. It has an immutable layer underneath my containers that updates using ostree.

However, keeping containers updated is a constant battle. Updating the containers themselves is fairly easy with a podman pull or docker pull followed by a stop and start. It’s a bit easier with docker-compose, but it’s still a nuisance to remember to update.

Enter watchtower

Watchtower is incredibly handy and surprisingly simple to operate. At a high level, it reads the container tags from each running container and watches the upstream repositories for updates. When an updated container appears, watchtower springs into action, pulls the new container, and replaces the old container with the new one.

It accepts arguments to configure all kinds of aspects of updating containers. You can exclude certain containers from updates, choose your update interval, and send notifications when updates occur.

Also, if manually updating containers is something you find fascinating, watchtower can notify you about updates. Then you get that great feeling of running lots of commands on your own. (Wait, surely nobody likes that!)

Deploy

There are plenty of configuration snippets in watchtower’s documentation, but you can start off with something as simple with this in your docker-compose.yaml:

services:
 watchtower:
 image: ghcr.io/containrrr/watchtower
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 restart: always
 privileged: true

My configuration on my mastodon deployment looks something like this:

services:
 watchtower:
 container_name: watchtower
 image: ghcr.io/containrrr/watchtower
 hostname: watchtower.tootchute.com
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock
 environment:
 - WATCHTOWER_CLEANUP=true
 - WATCHTOWER_POLL_INTERVAL=86400
 - WATCHTOWER_NOTIFICATIONS=shoutrrr
 - WATCHTOWER_NOTIFICATION_URL=discord://DISCORD_WEBHOOK_KEY@DISCORD_CHANNEL_ID
 restart: always
 privileged: true

This configuration sends me notifications via my Discord server whenever watchtower starts or when a container is updated.

The docker.sock volume mount allows watchtower to interact with the container daemon underneath watchtower. This could easily be done with docker, moby-engine, or podman.

I’d like to remove the privileged setting at some point soon but I haven’t figured out a way to allow watchtower to talk to the docker.sock without it. 🤔

Further considerations

If you don’t trust the upstream where you download your containers, be careful using watchtower. A malicious container could be uploaded to a particular container image repository and your system might update itself to the malicious container before the malicious container is found. Then again, if you don’t trust the upstream where you download your containers, you should be building these containers yourself. 😉

For more complex services that might need some extra care around updates, such as database services, you may want to exclude them from automatic updates. You can run multiple instances of watchtower with customized configurations for different sets of containers.

Second try at self-hosting Mastodon

major@mhtx.net (Major Hayden) — Mon, 02 Jan 2023 00:00:00 +0000

Mastodon caught my attention at the end of 2022 in the wake of all the Twitter shenanigans. At a high level, Mastodon is an implementation of ActivityPub and you can use it for “micro-blogging” much like you would use Twitter. (This is a really quick, high-level explanation and I skipped over plenty of detail.) 😉

This post covers my journey on Mastodon that led me to self-host my own Mastodon instance in a fairly reliable way.

Early start

My early Mastodon adventure started out much like the story of Goldilocks:

I started out on mastodon.social, but it was too big. There were so many people on the server that the federated timeline was flying by. Rules seemed to be enforced well, but it was a bit like Twitter all over again.
I deployed my own, but it was too small (the federated timeline was empty). Finding new people to talk to or following new topics was difficult.
Finally, I discovered Fosstodon after several friends in the open source community joined. It felt just right.

The admins of the Fosstodon instance are fantastic. Sure, there was downtime as the usage levels increased, but the admin team was quick to communicate the issues at hand along with future plans. My interactions with the community were almost all positive and it was fun to reconnect with some open source contributors that I had not spoken to in ages.

As time went on, I read various toots¹ about Mastodon servers changing owners, suddenly going offline, or altering rules abruptly. Someone talked about taking control of your online identity and that Mastodon should be included in that.

This aligned with my existing approach to hosting blogs on my own domains. Also, after the Twitter fiasco, I’d like people to find me via the systems where I have full control, such as my blog.

Self-hosted adventure

So far, there are three main deployment methods for Mastodon that I’ve found:

The official guide uses a custom Ruby, lots of steps, and systemd
Using docker-compose
Deploying in kubernetes using Mastodon’s charts or the ones from Bitnami

Official guide

Although the official guide looks fairly straightforward, it has a lot of steps. I struggled to get the right Ruby version compiled on Fedora 37 and I found spots where I needed to tweak the guide to make things work. Also, I wasn’t sure if I could get the steps done the same way again if I needed to migrate the instance or recover from a failure.

docker-compose

Next up was docker-compose. I use docker-compose quite often and I know my way around many of the rough edges. However, I couldn’t get the upstream compose file to work properly. Sometimes the database migrations would not run. Sometimes certain pieces of the Mastodon infrastructure couldn’t find each other. As soon as I tried to set passwords for postgres and redis, I couldn’t get Mastodon’s rails app to work again.

In addition, the docker-compose file from upstream builds containers on your local machine rather than pulling the official containers that were built and tested upstream. That’s a quick fix in the compose file, but I still had issues during the deployment.

kubernetes

Finally, I looked at kubernetes. Surely you can just add kubernetes to something and make it better, right? 😆

The Bitnami charts made it much further along than the charts from upstream, but I still had errors flowing about database migrations cut off during their run and occasionally unreachable postgres servers.

There must be a better way.

Deployment

For this Mastodon deployment to work well, I needed a few things:

The deployment should be mostly hands off. Said another way, moving it to another server or re-deploying should be a docker-compose up -d plus one or two commands maximum.
It should be relatively easy to back up and restore.
The big file of secret environment variables should be generated ahead of time and not at deploy time.

After plenty of trial and error, I came up with this plan:

Start with an empty secrets environment file. Deploy all of the containers and run the rake db:setup to generate the environment file.
Copy the environment file to .env.production so that it can be used along with upstream’s docker-compose file.
Delete the entire deployment. Remove all existing volumes and containers.
Add Caddy to the deployment to handle TLS and serving cached content.
Deploy again with docker-compose up -d and run rake db:setup to prepare the database with the environment file.

Without further ado, let’s get to the guide!

Generate the environment file

Here’s my initial docker-compose file:

version: '3'
services:
 postgres:
 restart: always
 container_name: postgres
 image: docker.io/library/postgres:14
 networks:
 - internal_network
 healthcheck:
 test: ['CMD', 'pg_isready', '-U', 'postgres']
 volumes:
 - postgres:/var/lib/postgresql/data
 environment:
 - POSTGRES_HOST_AUTH_METHOD=trust
 - POSTGRES_PASSWORD=my-super-secret-postgres-password
 - POSTGRES_USER=postgres

 redis:
 restart: always
 container_name: redis
 image: redis:7
 networks:
 - internal_network
 healthcheck:
 test: ['CMD', 'redis-cli', 'ping']
 volumes:
 - redis:/data

 web:
 image: tootsuite/mastodon
 container_name: web
 restart: always
 env_file: .env.production
 command: bash -c "rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000"
 networks:
 - external_network
 - internal_network
 healthcheck:
 test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
 ports:
 - '127.0.0.1:3000:3000'
 depends_on:
 - postgres
 - redis
 # - es
 volumes:
 - mastodon-public:/mastodon/public/system

 streaming:
 image: tootsuite/mastodon
 container_name: streaming
 restart: always
 env_file: .env.production
 command: node ./streaming
 networks:
 - external_network
 - internal_network
 healthcheck:
 test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1']
 ports:
 - '127.0.0.1:4000:4000'
 depends_on:
 - postgres
 - redis

 sidekiq:
 image: tootsuite/mastodon
 container_name: sidekiq
 restart: always
 env_file: .env.production
 command: bundle exec sidekiq -c 1
 depends_on:
 - postgres
 - redis
 networks:
 - external_network
 - internal_network
 volumes:
 - mastodon-public:/mastodon/public/system
 healthcheck:
 test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

networks:
 external_network:
 internal_network:
 internal: true

volumes:
 mastodon-public: {}
 postgres: {}
 redis: {}

I’ve made a few alterations to the upstream compose file:

I’m using the upstream containers from docker hub rather than building them on startup
My containers use docker volumes instead of mounting local directories
The sidekiq container only uses one worker (keeping resource usage low)

At this point, I can run docker-compose up -d and all of the containers are running. Now we can use Mastodon’s interactive configuration tool to generate our environments file:

docker-compose run --rm web bundle exec rake db:setup

Go through the interactive configuration and answer all of the questions there.

For SMTP, I used Mailgun since it’s very inexpensive for my Mastodon use case. Once you set up your account there, look for the SMTP credentials under your domain in Mailgun’s control panel. The Mastodon setup process will ask for those credentials.

Also, I keep all of my assets in Backblaze B2 to avoid clogging up all of the storage on my VM that runs Mastodon. Create a public bucket in Backblaze and create some access keys. When Mastodon asks for your S3 endpoint, use https://s3.us-west-001.backblazeb2.com. If it asks for a hostname, you can use s3.us-west-001.backblazeb2.com.

Once the setup completes, take the environments file that prints to the screen and store that as .env.production.

Delete the deployment (for real)

This is going to sound weird, but we need to throw everything away at this point. I like this step because it allows me to start fresh with a fully generated environments file. It’s a good simulation of how things might look in a brand new deployment or during a migration from one server to another.

[!NOTE] 💣 WARNING! This assumes that Mastodon’s containers are the only ones running on your system. If you are running other containers for other services, don’t run these commands. You must go through each container, remove it, and remove the associated volume carefully.

# Stop all of the current containers and delete them (see warning above!)
$ docker-compose rm -sfv

# Destroy all of the container volumes (see warning above!)
$ docker system prune --volumes

Add Caddy

For most container deployments, I’d use traefik here. Its configuration discovery abilities, especially when paired with docker-compose, are top-notch. There’s almost no little one-off configuration issues when you use traefik.

However, Mastodon has tons of static assets, such as images, stylesheets, and other media. Serving those through Mastodon’s rails web server is possible, but it’s horribly inefficient. It chews up much more CPU time and it’s slower to respond.

That’s where Caddy comes in. Caddy has automatic TLS capabilities with LetsEncrypt and it can also serve static content. This takes the load off of Mastodon’s rails web server.

Start by adding a new service to your compose file:

 caddy:
 image: caddy:2-alpine
 restart: unless-stopped
 container_name: caddy
 ports:
 - "80:80"
 - "443:443"
 volumes:
 - ./caddy/etc-caddy:/etc/caddy:Z
 - ./caddy/logs:/logs:Z
 - mastodon-public:/srv/mastodon/public:ro
 hostname: "tootchute.com"
 networks:
 - internal_network
 - external_network

Change the hostname to fit your server. The mastodon-public volume is the one that Mastodon uses for its public content and mounting it inside the Caddy container allows Caddy to serve those assets.

In my case, I created a caddy directory in my home directory to hold the configuration and log files:

$ mkdir caddy/{etc-caddy,logs}

🤓 NERD ALERT. The :Z on the volumes for configuration and logs ensures that these directories have the right SELinux contexts so that the container can access the files in these directories. If your system does not use SELinux, you can omit the :Z.

I wrote a caddy configuration in ./caddy/etc-caddy/Caddyfile that is a slight tweak of Robert Riemann’s version:

{
 # Global options block. Entirely optional, https is on by default
 # Optional email key for lets encrypt
 email major@mhtx.net
 # Optional staging lets encrypt for testing. Comment out for production.
 # acme_ca https://acme-staging-v02.api.letsencrypt.org/directory

 # admin off
}

tootchute.com {
 log {
 # format single_field common_log
 output file /logs/access.log
 }

 root * /srv/mastodon/public

 encode gzip

 @static file

 handle @static {
 file_server
 }

 handle /api/v1/streaming* {
 reverse_proxy streaming:4000
 }

 handle {
 reverse_proxy web:3000
 }

 #header {
 # Strict-Transport-Security "max-age=31536000;"
 #}

 header /sw.js Cache-Control "public, max-age=0";
 header /emoji* Cache-Control "public, max-age=31536000, immutable"
 header /packs* Cache-Control "public, max-age=31536000, immutable"
 header /system/accounts/avatars* Cache-Control "public, max-age=31536000, immutable"
 header /system/media_attachments/files* Cache-Control "public, max-age=31536000, immutable"

 handle_errors {
 @5xx expression `{http.error.status_code} >= 500 && {http.error.status_code} < 600`
 rewrite @5xx /500.html
 file_server
 }
}

Be sure to change tootchute.com to your Mastodon server’s domain as well as email to your email. In addition, you may want to uncomment the acme_ca option shown there to avoid hitting LetsEncrypt’s production API limits while you are testing your deployment. (Comment out the staging server later to ensure you get a valid, trusted certificate.)

Let’s bring up our new Caddy container!

$ docker-compose up -d

Initialize Mastodon

At this point, we have Caddy serving content and all of our Mastodon containers are running. However, the Mastodon database isn’t populated at all. Let’s do that now:

docker-compose run --rm web bundle exec rake db:setup

This step uses your environments file to run all of Mastodon’s database migrations and perform some initial setup steps. It might take about 30 seconds to run.

Create our first user once the setup process finishes:

$ docker-compose run --rm web bin/tootctl accounts create USERNAME --email YOUR_EMAIL --confirmed --role Owner

This command creates a new administrative user, sets the email address for that user, and confirms the account. The confirmation part allows you to skip the email confirmation process for that first account. Your initial password prints out as soon as the command finishes.

You should be able to access your Mastodon deployment on the domain you chose (mine is tootchute.com and log in as the user you just created. If something doesn’t look right, examine the container logs to see if it’s something obvious:

$ docker-compose logs -f --since 5m

If a container is in a restart loop, you should catch it fairly quickly in the logs.

Next steps

First, turn off new registrations if you plan to run a single user instance like I do. Click the preferences gear/cog on the main page, click Administration, *Server Settings, and Registrations.

Next, enable two-factor authentication for your account. Click the preferences gear/cog on the main page, click Account, and then Two-factor Auth.

Finally, back up your environments file (.env.production) and your docker-compose.yaml. This will make it much easier to recover from a failure or migrate to a new server.

If you’re using remote assets in S3 or Backblaze, you don’t need to back up that content. Focus on backing up postgres and redis on a regular basis:

# Dump postgres data
$ docker-compose exec postgres pg_dump -d mastodon -U postgres --no-owner > backups/pgdump-$(date +%F_%H-%M-%S).sql

# Copy redis data
$ docker-compose cp redis:/data/dump.rdb backups/

Let me know if you run into problems with the steps described in this post. I assembled them from my shell history and some notes I took along the way. There’s always a chance I missed something.

Posts on Mastodon were called “toots” for ages since that’s the supposed sound that an elephant trunk makes. Many people want to switch that to “posts” and the latest version of Mastodon changed the “toot” button to “publish.” I’ll call them toots forever. Heck, I’m the owner of tootchute.com. 😉 ↩︎

docker-compose on Fedora CoreOS

major@mhtx.net (Major Hayden) — Sat, 17 Dec 2022 00:00:00 +0000

Deploying applications in containers provides lots of flexibility and compatibility benefits.

Once you package your application and its dependencies in a container, that container runs almost anywhere without issues. Very few of the old “it worked on my machine!” problems remain. However, the challenge of running a container and linking it up with other helpful pieces of software still remains.

Web applications need something to serve HTTP requests and handle TLS. They also need databases, and those databases must be online and available first. All of these need reliable storage that is easily managed.

In my personal infrastructure, I keep coming back to docker-compose.

🐇 In a hurry? Skip to the last section of this post if you want to skip my reasons for using docker-compose and you just want to see the steps.

Why docker-compose?

The tried-and-true docker-compose is one of the original “set your desired state” systems for managing containers. You specify what your container deployment should look like and docker-compose finds a way to get your containers in order. Sometimes that’s a fresh start without any existing containers. Sometimes it involves managing an existing fleet of containers and adjusting their configuration.

As an example, deploying your first container with docker-compose is easy. Assemble a basic YAML file, point docker-compose at it, and your container is running!

Need to change the configuration? Just make your changes in the YAML file, re-run docker-compose, and it knows enough to make the right changes. If containers need to be restarted, it takes care of that.

Why not kubernetes?

I’ve run my own kubernetes deployment and I’ve also been a consumer of large kubernetes and OpenShift deployments. All of this experience taught me two things:

Kubernetes and OpenShift are great. Like really great. Once you learn them, they are incredibly powerful tools that make a developer’s life much easier.
Running my own kubernetes or OpenShift deployment on my own time (and own dime) is not enjoyable.

Deploying, maintaining, and troubleshooting kubernetes infrastructure on your own is time consuming. Shared storage and networking caused me the most headaches in the past.

What about k3s?

I love k3s. However, that still means I have to figure out networking for inter-container communication and load balancing. Shared storage is also needed.

I could argue that running my own k3s deployment is easier than full kubernetes, but in the end, there’s more extra stuff around it that I don’t want to maintain.

Why don’t you use managed kubernetes?

Great question! Several providers have some excellent kubernetes offerings out there. Smaller providers, such as Digital Ocean and VULTR, have affordable offerings that are packed with features.

The challenge is that kubernetes deployments have overhead for the control plane, so you can’t use all of the virtual machines that you rent. For example, you may get three virtual machines in your cluster, each with 2GB RAM, but you can really only use about 1GB of RAM from each instance for your containers.

In addition, you will eventually need shared storage and some type of load balancer. The costs add up quickly.

It’s easy to start with a $50-$70 kubernetes offering and later find yourself cracking $100 per month after adding on storage and load balancers.

What about podman?

Podman is a delight to use. You can toss some kubernetes YAML at it for deployments or pods and it will start them up for you. It also has a handy feature for exporting a container to a systemd unit file so you can manage it like any other systemd unit.

However, when I want to make quick adjustments to a container configuration, it can be frustrating to get that done. I also like to make adjustments to all of my containers in one place since some applications depend on multiple containers running in tandem.

The podman-compose project helps quite a bit, but it still lacks some of docker-compose’s features.

I use podman constantly on my laptop and desktop for development, testing, and toolbox containers.

What’s the big deal about CoreOS?

Fedora CoreOS provides the foundation for my container infrastructure. I love Fedora already, but here’s what makes CoreOS special to me:

Automatic updates arrive via ostree as a fully tested minimal unit
My system reboots automatically to apply the updates and it reverts back to the previous working update if the update fails
It comes with all of the container tools and configurations that I need
Whenever I need development or troubleshooting tools, toolbox containers are one step away

It’s a constantly-updated OS that is designed for containers. What could be better than that?

Adding docker-compose to CoreOS

Deploy CoreOS in your favorite cloud or on your favorite piece of hardware (I have a post about deploying it on Hetzner Cloud). Login as the core user via ssh.

Go to the releases page for docker-compose and get the latest release for your architecture. Move it into place once you download it:

$ curl -sLO https://github.com/docker/compose/releases/download/v2.14.1/docker-compose-linux-x86_64
$ sudo mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

Let’s add a really basic deployment of traefik’s whoami container. Start a new file called docker-compose.yaml:

services:
 whoami:
 container_name: whoami
 image: docker.io/traefik/whoami
 ports:
 - 8080:80
 restart: unless-stopped

If we try to bring up our containers now, we get an error:

$ docker-compose up -d
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3Dcore%22%3Atrue%7D%7D": dial unix /var/run/docker.sock: connect: permission denied

The core user is not in the docker group and does not have permissions to talk to the docker socket.

$ id
uid=1000(core) gid=1000(core) groups=1000(core),4(adm),10(wheel),16(sudo),190(systemd-journal) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Add the docker group to the core user as a supplementary group:

$ sudo usermod -a -G docker core

Log out of your ssh session and log in again to pick up the new group. Start the containers once more:

$ id
uid=1000(core) gid=1000(core) groups=1000(core),4(adm),10(wheel),16(sudo),190(systemd-journal),982(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ docker-compose up -d
[+] Running 4/4
 ⠿ whoami Pulled
 ⠿ 029cd1bf7e7c Pull complete
 ⠿ e73b694ead4f Pull complete
 ⠿ 99df6e9e9886 Pull complete
[+] Running 2/2
 ⠿ Network core_default Created
 ⠿ Container whoami Started

Success!

🤔 BUT WAIT! If we reboot our system, the containers won’t start up at boot time!

On CoreOS, there’s a docker.socket file that starts the actual docker service (which is actually moby-engine on Fedora and not full-fledged docker). As soon as something touches the socket, systemd starts the docker service. That keeps it out of the way until something actually asks to use it.

$ systemctl status docker.socket
● docker.socket - Docker Socket for the API
 Loaded: loaded (/usr/lib/systemd/system/docker.socket; enabled; preset: enabled)
 Active: active (running) since Sat 2022-12-17 22:48:42 UTC; 4min 47s ago
 Until: Sat 2022-12-17 22:48:42 UTC; 4min 47s ago
 Triggers: ● docker.service
 Listen: /run/docker.sock (Stream)
 Tasks: 0 (limit: 2207)
 Memory: 0B
 CPU: 596us
 CGroup: /system.slice/docker.socket

After a reboot, nothing pokes the socket and the associated service never starts. You can try it yourself! Reboot and the whoami container will be down. Run docker-compose ps one time and suddenly your containers are running!

Let’s fix this so containers come up on boot without any extra work (or socket poking):

$ sudo systemctl enable --now docker.service

The socket will still be handled by systemd, but now the docker service itself will always start at boot whether someone touched it or not.

Enjoy your new lightweight container infrastructure. ✨

Adventures with the mastodon herd

major@mhtx.net (Major Hayden) — Fri, 11 Nov 2022 00:00:00 +0000

All of the recent changes at Twitter inspired me to take a second look at mastodon. In short, mastodon is a federated social network that feels a bit like someone took Twitter and split it up into a vast network of independent servers.

Why mastodon?

It feels a lot like Twitter, but better.

You can search for people, follow them, and publish messages (called toots). They can also follow you and see the messages you publish.

The big difference is that you don’t join a central server with mastodon. There’s a massive network of servers to choose from and you can create accounts on one or more of those servers to get started. You can even run your own!

Mastodon reminds me of email for many reasons:

There’s no central server. You join a server (from the massive, growing list) and start publishing messages.
Everything is on an eventual consistency model. If a mastodon server goes offline for a bit or has network issues, messages and other data will synchronize when it’s back online.
You can follow people on your server or on other servers. You choose who to mute or block and you can create lists that help you group certain contacts.

After joining the fosstodon.org server, I noticed that it was really easy to begin following people and get messages. I reconnected with people that I had not heard from in a very long time!

Migrating from Twitter

One of my first questions after joining mastodon was: “How do I find the people I follow on Twitter?”

Many Twitter users are adding their mastodon accounts to their bio to make them easier to find on mastodon. For example, I added my mastodon account, @major@fosstodon.org, to my twitter bio:

Adding this to your bio makes it easier for people to find you via some helpful tools. I used debirdify to look through my Twitter account for mastodon handles of the people I follow. Within seconds, it provided links to about 15 mastodon accounts and offered me a CSV that I could directly import into my mastodon server. 🎉 (Mastodon servers have some awesome import and export capabilities.)

I’ve also heard good things about Fedifinder and Twitodon for helping you find Twitter friends on mastodon. There’s a helpful article on Wired with more suggestions.

Apps

I’m neck-deep in the Android ecosystem, so most of my suggestions here are for Android devices. I tried the main mastodon app first. It looks great, updates quickly, and is very easy to use. However, inserting GIFs into toots became really frustrating (although I hear that’s being fixed).

I moved to Tusky and it’s my go-to mastodon app. You can add multiple accounts, posting media is incredibly easy, and it has tons of configuration knobs everywhere.

There are various desktop applications for mastodon, but the web interface is good enough for me! The default web interface looks a lot like Twitter with a big timeline running down the middle and section links on the right.

However, I use Tweetdeck with Twitter and I wanted something similar on mastodon. Go into the settings for the web application, and choose Appearance. Click Enable advanced web interface, save the changes, and click Back to Mastodon. Enjoy your Tweetdeck-like multi-column interface! ✨

Run your own instance

The federated nature of mastodon means you can run your own single user instance if you want! Buy a domain you like (or use a subdomain off an existing domain) and deploy!

The upstream repository has a helm chart which works well with kubernetes. Also, there’s a docker-compose file which works well for smaller deployments.

I went the docker-compose route on a small cloud instance at Hetzner, but I modified the upstream docker-compose.yml:

version: '3'
volumes:
 certs:
 postgres:
 redis:
 mastodon:
services:
 traefik:
 image: docker.io/library/traefik:latest
 container_name: traefik
 restart: unless-stopped
 command:
 # Tell Traefik to discover containers using the Docker API
 - --providers.docker=true
 - --providers.docker.exposedbydefault=false
 # Enable the Trafik dashboard
 - --api.dashboard=true
 # Set up LetsEncrypt
 #- --certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
 - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
 - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=porkbun
 - --certificatesresolvers.letsencrypt.acme.email=major@mhtx.net
 - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
 # Set up an insecure listener that redirects all traffic to TLS
 - --entrypoints.web.address=:80
 - --entrypoints.web.http.redirections.entrypoint.to=websecure
 - --entrypoints.web.http.redirections.entrypoint.scheme=https
 - --entrypoints.websecure.address=:443
 # Set up the TLS configuration for our websecure listener
 - --entrypoints.websecure.http.tls=true
 - --entrypoints.websecure.http.tls.certResolver=letsencrypt
 - --entrypoints.websecure.http.tls.domains[0].main=toots.cloud
 - --entrypoints.websecure.http.tls.domains[0].sans=*.toots.cloud
 environment:
 - PORKBUN_SECRET_API_KEY=*****
 - PORKBUN_API_KEY=*****
 ports:
 - '80:80'
 - '443:443'
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock:ro
 - certs:/letsencrypt
 labels:
 - "traefik.enable=true"
 - 'traefik.http.routers.traefik.rule=Host(`traefik.toots.cloud`)'
 - "traefik.http.routers.traefik.entrypoints=websecure"
 - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
 - "traefik.http.routers.traefik.service=api@internal"
 - 'traefik.http.routers.traefik.middlewares=strip'
 - 'traefik.http.middlewares.strip.stripprefix.prefixes=/traefik'

 postgres:
 container_name: postgres
 restart: always
 image: docker.io/library/postgres:14-alpine
 shm_size: 256mb
 env_file: .env.production
 healthcheck:
 test: ['CMD', 'pg_isready', '-U', 'postgres']
 volumes:
 - postgres:/var/lib/postgresql/data
 - ./postgres-setup.sh:/docker-entrypoint-initdb.d/init-user-db.sh:Z
 environment:
 - 'POSTGRES_HOST_AUTH_METHOD=trust'

 redis:
 container_name: redis
 restart: always
 image: docker.io/library/redis:7-alpine
 healthcheck:
 test: ['CMD', 'redis-cli', 'ping']
 volumes:
 - redis:/data

 web:
 container_name: web
 image: docker.io/tootsuite/mastodon:v3.5.3
 restart: always
 env_file: .env.production
 command: bash -c "rm -f /mastodon/tmp/pids/server.pid; bundle exec rails s -p 3000"
 healthcheck:
 # prettier-ignore
 test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:3000/health || exit 1']
 ports:
 - '3000'
 depends_on:
 - postgres
 - redis
 volumes:
 - mastodon:/mastodon/public/system
 labels:
 - "traefik.enable=true"
 - "traefik.http.routers.web.rule=Host(`toots.cloud`)"
 - "traefik.http.routers.web.entrypoints=websecure"
 - "traefik.http.routers.web.tls.certresolver=letsencrypt"

 streaming:
 container_name: streaming
 image: docker.io/tootsuite/mastodon:v3.5.3
 restart: always
 env_file: .env.production
 command: node ./streaming
 healthcheck:
 # prettier-ignore
 test: ['CMD-SHELL', 'wget -q --spider --proxy=off localhost:4000/api/v1/streaming/health || exit 1']
 ports:
 - '4000'
 depends_on:
 - postgres
 - redis

 sidekiq:
 container_name: sidekiq
 image: docker.io/tootsuite/mastodon:v3.5.3
 restart: always
 env_file: .env.production
 command: bundle exec sidekiq
 depends_on:
 - postgres
 - redis
 volumes:
 - mastodon:/mastodon/public/system
 healthcheck:
 test: ['CMD-SHELL', "ps aux | grep '[s]idekiq\ 6' || false"]

Here are the main changes I made:

Specified the exact URL/tag for each container
Added traefik to handle TLS
Used named volumes instead of filesystem directories (made SELinux much happier)
Added a provisioning script for postgres

The provisioning script for postgres allows me to bring up postgres without needing to run any extra commands:

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username postgres <<-EOSQL
	CREATE USER mastodon WITH PASSWORD 'super-secret-password' CREATEDB;
EOSQL

Once you have all of this in place, run the usual docker-compose up -d and wait for everything to start. Then you can run through the initial mastodon setup:

$ docker-compose run --rm web bundle exec rake mastodon:setup

You will need to answer lots of questions, including your domain name, postgres/redis details, email configuration, and object storage configuration. I use Mailgun for mastodon’s email since it makes the setup much easier and has a very low cost. For object storage, I went with a public Backblaze B2 bucket since it’s Amazon S3 compatible but very inexpensive¹.

When the setup finishes, it will dump an environments file to the screen. Be sure to save that file. This will allow you to start up all of the containers again with the same configuration later. A copy of the environments file will be kept inside the container storage as well.

Self-hosted instance takeaways

I’ve been running my own mastodon instance for a few days and I’m not sure if I will keep it. Sure, I love having an instance on a hilarious domain like toots.cloud and having full control over my mastodon experience.

But it’s one more thing to manage, patch, and back up.

The fosstodon.org community has been excellent so far and I’m contributing to their costs each month via their Patreon page. Every mastodon instance is going through growing pains recently due to really high demand.

The last count from the Mastodon Users bot shows massive interest:

If you’re on a server that isn’t performing well: be patient.

Ask for ways that you can help technically or financially. One of the biggest reminders that I get from mastodon is that every server is a community. The community must come together to make each server successful as a part of the big fediverse.

Be sure to note the endpoint for your Backblaze bucket when you create it. You will need to specify that endpoint when you set up mastodon. As an example, my endpoint is https://s3.us-west-001.backblazeb2.com and my region is us-west-001. ↩︎

Mount NFS shares in kubernetes

major@mhtx.net (Major Hayden) — Fri, 08 Apr 2022 00:00:00 +0000

Kubernetes offers a plethora of storage options for mounting volumes in pods, and NFS is included. I have a Synology NAS at home and some of my pods in my home kubernetes deployment need access to files via NFS.

Although the Kubernetes documentation has a bunch of examples about setting up NFS mounts, I ended up being more confused than when I started. This post covers a simple example that you can copy, adapt, and paste as needed.

Verify that NFS is working

NFS can be tricky to get right and it’s important to verify that it’s working outside of kubernetes before you try mounting it in a pod. Trust me – NFS looks quite simple at first glance but you can get confused quickly. Test out the easiest stuff first.

As an example: Synology made some recent changes to their NFS configuration where you must specify shares with a netmask (192.168.10.0/255.255.255.0) or in CIDR notation (192.168.10.0/24). That took me a while to figure out going back and forth from server to client and back again.

If you’re making your shares from a regular Linux server, refer to the Arch Linux NFS documentation. It’s one of the best write-ups on NFS around!

First, I verified that the mount is showing up via showmount:

$ showmount -e 192.168.10.60
Export list for 192.168.10.60:
/volume1/media 192.168.10.50/32

My NFS server is on 192.168.10.60 and my NFS client (running kubernetes) is on 192.168.10.50.

🤔 Got an error or can’t see any exports? Double check the IP addresses allowed to access the share on the server side and verify your client machine’s IP address.

☝🏻 Remember to re-export your shares on the server with exportfs -arv if you made changes! The NFS server won’t pick them up automatically. Display your currently running exports with exportfs -v.

Let’s try mounting the share next:

$ sudo mount -t nfs 192.168.10.60:/volume1/media /tmp/test
$ df -hT /tmp/test
Filesystem Type Size Used Avail Use% Mounted on
192.168.10.60:/volume1/media nfs4 16T 7.5T 8.3T 48% /tmp/test

Awesome! 🎉

Unmount the test:

$ sudo umount /tmp/test

Mount NFS in a pod

First off, we need a deployment where we can mount up an NFS share. I decided to take a Fedora 35 container and create a really basic deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: fedoratest
 name: fedoratest
 namespace: fedoratest
spec:
 template:
 spec:
 containers:
 - image: registry.fedoraproject.org/fedora:35
 name: fedora
 command: ["/bin/bash", "-c", "--"]
 args: ["while true; do sleep 30; done;"]

This is a really silly deployment that causes Fedora to sleep forever until someone stops the pod. I mainly want something that I can shell into and ensure NFS is working.

Now we need to add two pieces to the deployment:

An NFS volume:

volumes:
 - name: nfs-vol
 nfs:
 server: 192.168.10.60
 path: /volume1/media

A path to mount the volume

volumeMounts:
 - name: nfs-vol
 mountPath: /opt/nfs

When we add in those NFS pieces, we get the following deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: fedoratest
 name: fedoratest
 namespace: fedoratest
spec:
 template:
 spec:
 containers:
 - image: registry.fedoraproject.org/fedora:35
 name: fedora
 command: ["/bin/bash", "-c", "--"]
 args: ["while true; do sleep 30; done;"]
 volumeMounts:
 - name: nfs-vol
 mountPath: /opt/nfs
 volumes:
 - name: nfs-vol
 nfs:
 server: 192.168.10.60
 path: /volume1/media

Save that as fedoratest.yaml and apply it:

$ kubectl apply -f fedoratest.yaml

Let’s see if the volume worked:

$ kubectl describe -n fedoratest deployment/fedoratest
✂
 Volumes:
 nfs-vol:
 Type: NFS (an NFS mount that lasts the lifetime of a pod)
 Server: 192.168.10.60
 Path: /volume1/media
 ReadOnly: false
✂

Now let’s have a look at it inside the container itself:

$ kubectl -n fedoratest exec -it deployment/fedoratest -- sh
sh-5.1$ cd /opt/nfs
sh-5.1$ ls
dir1 dir2 dir3

Let’s ensure we can write files:

sh-5.1$ touch doot
sh-5.1$ ls -al doot
-rwxrwxrwx. 1 root root 0 Apr 8 21:03 doot
sh-5.1$ rm doot

I can write files, but writing them as root causes problems for other applications. In this case, my NFS server uses UID 1035 for my user and GID 100 for my group. Lucky for us, we can set this up within our deployment configuration using securityContext:

apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: fedoratest
 name: fedoratest
 namespace: fedoratest
spec:
 template:
 spec:
 securityContext:
 runAsUser: 1035 # Use my UID on the NFS server
 runAsGroup: 100 # Use my GID on the NFS server
 containers:
 - image: registry.fedoraproject.org/fedora:35
 name: fedora
 command: ["/bin/bash", "-c", "--"]
 args: ["while true; do sleep 30; done;"]
 volumeMounts:
 - name: nfs-vol
 mountPath: /opt/nfs
 volumes:
 - name: nfs-vol
 nfs:
 server: 192.168.10.60
 path: /volume1/media

Apply this change:

$ kubectl apply -f fedoratest.yaml

Now try to write a file again:

$ kubectl -n fedoratest exec -it deployment/fedoratest -- sh
sh-5.1$ touch doot
sh-5.1$ ls -al doot
-rwxrwxrwx. 1 1035 100 0 Apr 8 21:08 doot
sh-5.1$ rm doot

Perfect! Now files are owned by the correct UID and GID.

Extra credit

If you plan to have plenty of pods mounting storage from the same NFS server, you might want to consider building out a persistent volume first and then making claims from it. The kubernetes examples repository has a good example of a persistent NFS volume and a persistent volume claim made from that volume.

Run Xorg applications with podman

major@mhtx.net (Major Hayden) — Sun, 17 Oct 2021 00:00:00 +0000

Containers are a great way to deliver and run all kinds of applications. Although many people build containers for server applications, you can also use them for client applications on your local workstation. This helps when you want to test new applications without disrupting your existing system or when you use an immutable system such as Fedora Silverblue.

Podman takes this further by allowing you to run a client application without root access or daemons. This post covers how to build a container with an Xorg application and run it on a Fedora system.

Building the container

Let’s start with a simple container that contains xeyes. This simple application simply puts a pair of eyes on your screen that follow your mouse movements around the desktop. It has very few dependencies and it’s a great way to test several capabilities on the desktop.

Here’s a very simple container build file:

# xeyes-container
FROM registry.fedoraproject.org/fedora:latest
RUN dnf -y install xeyes
CMD xeyes

Let’s install podman and build the container:

$ sudo dnf -y install podman
$ podman build -t xeyes -f xeyes-container .

Run the container

Now that we have our xeyes container, let’s run it.

$ podman run --rm xeyes
Error: Can't open display:

We’re missing the DISPLAY variable inside the container. Let’s add it:

$ echo $DISPLAY
:0
$ podman run --rm -e DISPLAY xeyes
Error: Can't open display: :0

Well, we have the display variable inside now, but there’s another problem. Inside the container, xeyes can’t make a connection to our X daemon. This socket normally appears in /tmp/.X11-unix, but the container doesn’t have it. Let’s try adding this inside the container:

$ podman run --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix xeyes
Error: Can't open display: :0

Darn! This should be working. Let’s check the system journal:

AVC avc: denied { write } for pid=10817 comm="xeyes" name="X0" dev="tmpfs"
ino=42 scontext=system_u:system_r:container_t:s0:c143,c574
tcontext=system_u:object_r:user_tmp_t:s0 tclass=sock_file permissive=0

Uh oh. SELinux is upset that a container is trying to mess with the X0 socket for our Xorg server that sits in /tmp/.X11-unix. You may be tempted to run setenforce 0, but wait. We can fix this with podman!

Podman allows you to set security options for a particular container with --security-opt. We need to run this container with an SELinux context that allows it to talk to something in tmpfs. Examining the container-selinux project shows that container_runtime_t can work with tmpfs:

type container_runtime_tmp_t alias docker_tmp_t;
files_tmp_file(container_runtime_tmp_t)

Let’s try adding this to our podman command now:

$ podman run --rm -e DISPLAY \
 -v /tmp/.X11-unix:/tmp/.X11-unix \
 --security-opt label=type:container_runtime_t xeyes

I now have a set of eyeballs on my desktop! 👀

Extra credit

The xeyes application is extremely simple, but you can run much more complex applications using this same method. Keep in mind that certain applications might require extra packages inside the container, such as fonts or GTK themes. Jessie Frazelle has a great repository full of containers that she uses regularly and this might give you inspiration to create some of your own! 🤓

Photo credit: Jonny Gios on Unsplash

Wildcard LetsEncrypt certificates with Traefik and Cloudflare

major@mhtx.net (Major Hayden) — Mon, 16 Aug 2021 00:00:00 +0000

Wildcard certificates make it easy to secure lots of subdomains under a single domain. For example, you can secure web.example.com and mail.example.com with a single certificate for *.example.com. Fortunately, LetsEncrypt allows you to get wildcard certificates via a DNS ownership check (often called a DNS-01 challenge).

Fortunately, Traefik can request a certificate from LetsEncrypt automatically and complete the challenge for you. It can publish DNS records to multiple providers, but my favorite is Cloudflare. They will host your DNS zones and records for free. They also have a robust API for managing DNS records (also free).

In this post, we will cover the basics of getting TLS working with Traefik. We can add a wildcard certificate on top and then re-use that same certificate for other containers running behind Traefik.

Basic setup

First, we need a running instance of Traefik. The Traefik documentation explains this entire process in detail and I highly recommend reading the basics on configuration discovery, routers, and TLS settings.

We will use docker-compose to make this easier to manage. If you’re on Fedora, install docker-compose:

dnf install docker-compose

Now we need a docker-compose.yml file:

---
version: "3"
services:
 traefik:
 image: traefik:latest
 container_name: traefik
 restart: unless-stopped
 command:
 # Tell Traefik to discover containers using the Docker API
 - --providers.docker=true
 # Enable the Trafik dashboard
 - --api.dashboard=true
 # Set up LetsEncrypt
 - --certificatesresolvers.letsencrypt.acme.dnschallenge=true
 - --certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare
 - --certificatesresolvers.letsencrypt.acme.email=EMAIL_ADDRESS
 - --certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json
 # Set up an insecure listener that redirects all traffic to TLS
 - --entrypoints.web.address=:80
 - --entrypoints.web.http.redirections.entrypoint.to=websecure
 - --entrypoints.web.http.redirections.entrypoint.scheme=https
 - --entrypoints.websecure.address=:443
 # Set up the TLS configuration for our websecure listener
 - --entrypoints.websecure.http.tls=true
 - --entrypoints.websecure.http.tls.certResolver=letsencrypt
 - --entrypoints.websecure.http.tls.domains[0].main=home.example.com
 - --entrypoints.websecure.http.tls.domains[0].sans=*.home.example.com
 environment:
 - CLOUDFLARE_EMAIL=CLOUDFLARE_ACCOUNT_EMAIL_ADDRESS
 - CLOUDFLARE_DNS_API_TOKEN=CLOUDFLARE_TOKEN_GOES_HERE
 ports:
 - 80:80
 - 443:443
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock:ro
 - certs:/letsencrypt
 labels:
 - "traefik.enable=true"
 - 'traefik.http.routers.traefik.rule=Host(`home.example.com`)'
 - "traefik.http.routers.traefik.entrypoints=websecure"
 - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
 - "traefik.http.routers.traefik.service=api@internal"
 - 'traefik.http.routers.traefik.middlewares=strip'
 - 'traefik.http.middlewares.strip.stripprefix.prefixes=/traefik'

In this example, we tell Traefik about our desired setup in the command section, including our listeners. Our insecure listener on port 80 redirects to secure connections on port 443 and we tell Traefik that we plan to use LetsEncrypt to get the certificates.

We provide the username and Cloudflare API key in the environment section. Follow Cloudflare’s guides for managing API tokens and keys carefully to generate a token.

The labels section sets up a rule where traffic destined for home.example.com goes to the Traefik dashboard. this is helpful in case you make mistakes or you can’t figure out why something is working. You can go to the dashboard to show all of the existing services, listeners, and other configurations.

☝🏻 Before applying this docker-compose file, change a few things:

Set your LetsEncrypt email address in the line with --certificatesresolvers.letsencrypt.acme.email
Set your Cloudflare account email address for the CLOUDFLARE_EMAIL environment variable
Set your Cloudflare DNS API token for the CLOUDFLARE_DNS_API_TOKEN environment variable
Change the Host() rules from example.com to match your domain name

Run docker-compose up -d and then docker-compose logs -f traefik to see if Traefik came up successfully with certificates. If you run into any problems, double check that your Cloudflare email and token are accurate. Also verify that your Cloudflare token has the correct permissions to adjust the dns zone.

Adding a container

At this point, we can add another container and it can use the same TLS certificate we requested from LetsEncrypt already!

The librespeed project provides a self-hosted network speed test that you can run on any network. It also runs perfectly inside a container. The linuxserver.io librespeed container is well maintained and easy to run.

Add this to your docker-compose.yml right under the Traefik configuration:

 librespeed:
 image: ghcr.io/linuxserver/librespeed
 container_name: librespeed
 restart: unless-stopped
 ports:
 - 80
 labels:
 - "traefik.enable=true"
 - "traefik.http.routers.librespeed.rule=Host(`librespeed.home.example.com`)"
 - "traefik.http.routers.librespeed.entrypoints=websecure"
 - "traefik.http.routers.librespeed.tls.certresolver=letsencrypt"

Check the labels section. We first enable Traefik so it will route requests to the container. Then we set a host rule so that traffic for librespeed.home.example.com comes to this container. We only listen for TLS traffic (remember our redirect for insecure traffic earlier).

Finally, we tell Traefik to use the same certresolver as before. Traefik is smart enough to know that *.home.example.com covers the librespeed.home.example.com subdomain just fine.

Run docker-compose up -d once more and now librespeed has a secure connection using the original wildcard certificate.

Renewals

LetsEncrypt certificates are valid for only 90 days. That’s why automation plays such an important role in handling renewals. You certainly don’t want to set calendar reminders to log into your server and run a script every 90 days. 😱

Traefik automatically knows when the expiration date approaches. When the certificate has less than 30 days left until the expiration date, Traefik automatically renews the certificate.

💣 Be careful with your DNS zone and with your DNS API keys! If you accidentally delete the API key or make big changes to your DNS zone, there’s a chance that Traefik may not be able to renew the certificate.

Photo credit: Veron Wessels on Unsplash

Rootless container management with docker-compose and podman

major@mhtx.net (Major Hayden) — Fri, 09 Jul 2021 00:00:00 +0000

Everyone has an opinion for the best way to manage containers, and there are many contenders depending on how much complexity you can handle and how much automation you require. One of my favorite ways to manage containers is docker-compose.

Overview of docker-compose

docker-compose uses a simple YAML syntax to explain what your desired end state should look like. The compose specification covers all of the relevant configurations for containers, volumes, networks, and more. After each change, docker-compose compares your configuration to the running containers and makes all of the required changes.

This provides some advantages over using docker run ... or podman run ... since you can put the YAML into version control and track your configuration changes all in one place. I was tracking the configuration in shell scripts that ran docker with lots of parameters and that became difficult to manage.

What about Podman?

Podman is a tool for managing containers, much like Docker, but it has some distinct advantages:

No daemons are needed
You can run containers as your user, or as root
The commands and arguments are nearly identical to docker (no swarm support)
Podman 3 added a complete Docker-compatible API

This last part, the Docker-compatible API is quite interesting and this allows docker-compose to work with podman as well as it does with docker.

Let’s try it out!

Getting everything ready

Start with a working Fedora 34 system and install some packages:

💣 HEADS UP: The podman-docker package brings in podman, an alias for the docker command that actually runs podman, and the docker-compatible API via a socket. If you want to run podman and docker side by side on the same machine, install podman instead of podman-docker here. If you had docker installed already, you may need to remove it with dnf remove docker-ce docker-ce-cli.

dnf install docker-compose podman-docker

We’re going to do something different here. Intead of starting the podman socket or docker daemon as root, we’re going to start the podman socket as a regular user. Switch to a regular user and start the socket:

$ systemctl enable --now --user podman.socket
Created symlink /home/major/.config/systemd/user/sockets.target.wants/podman.socket → /usr/lib/systemd/user/podman.socket.

But wait, where’s the socket?

$ ls -al $XDG_RUNTIME_DIR/podman/podman.sock
srw-rw----. 1 major major 0 Jul 9 16:49 /run/user/1000/podman/podman.sock

That’s a podman socket running as my user and exposing a docker-compatible API. 🎉

Time for docker-compose

Now it’s time to use docker-compose with podman as a regular user and run a container as our regular user.

We can use librespeed for this example, and the LinuxServer librespeed container is a great way to deploy it. It’s a self-hosted speed test application that works well with desktops and mobile devices.

First, we begin with the suggested docker-compose configuration:

---
version: "2.1"
services:
 librespeed:
 image: ghcr.io/linuxserver/librespeed
 container_name: librespeed
 environment:
 - PUID=1000
 - PGID=1000
 - TZ=Etc/UTC
 volumes:
 - librespeed:/config
 ports:
 - 8080:80
 restart: unless-stopped

volumes:
 librespeed: {}

Save that as docker-compose.yml in your current directory.

Keep in mind that docker-compose is expecting to find our docker socket in /var/run/docker.sock, but we’re running the podman socket as our regular user. Let’s export the DOCKER_HOST variable and run docker-compose to bring up our new container:

$ export DOCKER_HOST="unix:$XDG_RUNTIME_DIR/podman/podman.sock"
$ docker-compose up -d
Pulling librespeed (ghcr.io/linuxserver/librespeed:)...
10f45b17b9ab: Download complete
f23b92877416: Download complete
a5bf9c523af4: Download complete
00fe9b963179: Download complete
bfafa0ba1dc9: Download complete
c583b34264f1: Download complete
9d26cce56b8d: Download complete
70de87880afd: Download complete
0ad6c2578069: Download complete
a8792749de3b: Download complete
2d31530d2d8b: Download complete
Creating librespeed ... done
$ docker-compose ps
 Name Command State Ports
--------------------------------------------
librespeed /init Up () :8080->80/tcp

The container is up and running as our user. Let’s check the nginx process inside the container to be sure:

$ ps -xu |grep "nginx: master"
major 3805 0.0 0.4 5860 4692 ? Ss 16:53 0:00 nginx: master process /usr/sbin/nginx -c /config/nginx/nginx.conf

Sweet! 🥳

Time for a speed test

If we’ve come this far, we might as well test our internet speed to ensure the container works!

Remember that we used port 8080 as a replacement for 80 in our docker-compose file to avoid issues with regular users being denied access to create a listener on ports under 1024.

Let’s see how fast my connection is today:

Photo credit: Michael D Beckwith on Unsplash

Get faster GitLab runners with a ramdisk

major@mhtx.net (Major Hayden) — Fri, 16 Aug 2019 00:00:00 +0000

When you build tons of kernels every day like my team does, you look for speed improvements anywhere you can. Caching repositories, artifacts, and compiled objects makes kernel builds faster and it reduces infrastructure costs.

Need for speed

We use GitLab CI in plenty of places, and that means we have a lot of gitlab-runner configurations for OpenShift (using the kubernetes executor) and AWS (using the docker-machine executor). The runner’s built-in caching makes it easy to upload and download cached items from object storage repositories like Google Cloud Storage or Amazon S3.

However, there’s an often overlooked feature hiding in the configuration for the docker executor that provides a great performance boost: mounting tmpfs inside your container. Not familiar with tmpfs? Arch Linux has a great wiki page for tmpfs and James Coyle has a well-written blog post about what makes it unique from the older ramfs.

RAM is much faster than your average cloud provider’s block storage. It also has incredibly low latency relative to most storage media. There’s a great interactive latency page that allows you to use a slider to travel back in time to 1990 and compare all kinds of storage performance numbers. (It’s really fun! Go drag the slider and be amazed.)

Better yet, many cloud providers give you lots of RAM per CPU on their instances, so if your work isn’t terribly memory intensive, you can use a lot of this RAM for faster storage.

Enabling tmpfs in Docker containers

⚠️ Beware of the dangers of tmpfs before adjusting your runner configuration! See the warnings at the end of this post.

This configuration is buried in the middle of the docker executor documentation. You will need to add some extra configuration to your [runners.docker] section to make it work:

[runners.docker]
 [runners.docker.tmpfs]
 "/ramdisk" = "rw,noexec"

This configuration mounts a tmpfs volume underneath /ramdisk inside the container. By default, this directory will be mounted with noexec, but if you need to execute scripts from that directory, change noexec to exec:

[runners.docker]
 [runners.docker.tmpfs]
 "/ramdisk" = "rw,exec"

In our case, compiling kernels requires executing scripts, so we use exec for our tmpfs mounts.

You must be specific for exec! As an example, this tmpfs volume will be mounted with noexec since that is the default:

[runners.docker]
 [runners.docker.tmpfs]
 "/ramdisk" = "rw"

Extra speed

For even more speed, we moved the objects generated by ccache to the ramdisk. The seek times are much lower and this allows ccache to look for its cached objects much more quickly.

Git repositories are also great things to stash on tmpfs. Big kernel repositories are usually 1.5GB to 2GB in size with tons of files. Checkouts are really fast when they’re done in tmpfs.

Dangers are lurking

⚠️ As mentioned earlier, beware of the dangers of tmpfs.

All of the containers on the machine will share the same amount of RAM for their tmpfs volumes. Be sure to account for how much each container will use and how many containers could be present on the same machine.
Be aware of how much memory your tests will use when they run. In our case, kernel compiles can consume 2-4GB of RAM, depending on configuration, so we try our best to leave some memory free.
These volumes also have no limits on how much data can go into the volume. However, if you put too much data into the tmpfs volume and your system runs critically low on available RAM, you could see a huge drop in performance, system instability, or even a crash. 🔥

buildah error: vfs driver does not support overlay.mountopt options

major@mhtx.net (Major Hayden) — Tue, 13 Aug 2019 00:00:00 +0000

Buildah and podman make a great pair for building, managing and running containers on a Linux system. You can even use them with GitLab CI with a few small adjustments, namely the switch from the overlayfs to vfs storage driver.

I have some regularly scheduled GitLab CI jobs that attempt to build fresh containers each morning and I use these to get the latest packages and find out early when something is broken in the build process. A failed build appeared in my inbox earlier this week with the following error:

+ buildah bud -f builds/builder-fedora30 -t builder-fedora30 .
vfs driver does not support overlay.mountopt options

My container build script¹ is fairly basic, but it does include a change to use the vfs storage driver:

# Use vfs with buildah. Docker offers overlayfs as a default, but buildah
# cannot stack overlayfs on top of another overlayfs filesystem.
export STORAGE_DRIVER=vfs

The script doesn’t change any mount options during the build process. A quick glance at the /etc/containers/storage.conf revealed a possible problem:

[storage.options]
# Storage options to be passed to underlying storage drivers

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

These mount options make sense when used with an overlayfs filesystem, but they are not used with vfs. I commented out the mountopt option, saved the file, and ran a test build locally. Success!

Fixing the build script involved a small change to the storage.conf just before building the container:

# Use vfs with buildah. Docker offers overlayfs as a default, but buildah
# cannot stack overlayfs on top of another overlayfs filesystem.
export STORAGE_DRIVER=vfs

# Newer versions of podman/buildah try to set overlayfs mount options when
# using the vfs driver, and this causes errors.
sed -i '/^mountopt =.*/d' /etc/containers/storage.conf

My containers are happily building again in GitLab.

The original build script is no longer available, but the remainder of the repository still exists. ↩︎

Build containers in GitLab CI with buildah

major@mhtx.net (Major Hayden) — Fri, 24 May 2019 00:00:00 +0000

My team at Red Hat depends heavily on GitLab CI and we build containers often to run all kinds of tests. Fortunately, GitLab offers up CI to build containers and a container registry in every repository to hold the containers we build.

This is really handy because it keeps everything together in one place: your container build scripts, your container build infrastructure, and the registry that holds your containers. Better yet, you can put multiple types of containers underneath a single git repository if you need to build containers based on different Linux distributions.

Building with Docker in GitLab CI

By default, GitLab offers up a Docker builder that works just fine. The CI system clones your repository, builds your containers and pushes them wherever you want. There’s even a simple CI YAML file that does everything end-to-end for you.

However, I have two issues with the Docker builder:

Larger images: The Docker image layering is handy, but the images end up being a bit larger, especially if you don’t do a little cleanup in each stage.
Additional service: It requires an additional service inside the CI runner for the dind (“Docker in Docker”) builder. This has caused some CI delays for me several times.

Building with buildah in GitLab CI

On my local workstation, I use podman and buildah all the time to build, run, and test containers. These tools are handy because I don’t need to remember to start the Docker daemon each time I want to mess with a container. I also don’t need sudo.

All of my containers are stored beneath my home directory. That’s good for keeping disk space in check, but it’s especially helpful on shared servers since each user has their own unique storage. My container pulls and builds won’t disrupt anyone else’s work on the server and their work won’t disrupt mine.

Finally, buildah offers some nice options out of the box. First, when you build a container with buildah bud, you end up with only three layers by default:

Original OS layer (example: fedora:30)
Everything you added on top of the OS layer
Tiny bit of metadata

This is incredibly helpful if you use package managers like dnf, apt, and yum that download a bunch of metadata before installing packages. You would normally have to clear the metadata carefully for the package manager so that your container wouldn’t grow in size. Buildah takes care of that by squashing all the stuff you add into one layer.

Of course, if you want to be more aggressive, buildah offers the --squash option which squashes the whole image down into one layer. This can be helpful if disk space is at a premium and you change the layers often.

Getting started

I have a repository called os-containers in GitLab that maintains fully updated containers for Fedora 29 and 30. The .gitlab-ci.yml file calls build.sh for two containers: fedora29 and fedora30. Open the build.sh file and follow along here:

# Use vfs with buildah. Docker offers overlayfs as a default, but buildah
# cannot stack overlayfs on top of another overlayfs filesystem.
export STORAGE_DRIVER=vfs

First off, we need to tell buildah to use the vfs storage driver. Docker uses overlayfs by default and stacking overlay filesystems will definitely lead to problems. Buildah won’t let you try it.

# Write all image metadata in the docker format, not the standard OCI format.
# Newer versions of docker can handle the OCI format, but older versions, like
# the one shipped with Fedora 30, cannot handle the format.
export BUILDAH_FORMAT=docker

By default, buildah uses the oci container format. This sometimes causes issues with older versions of Docker that don’t understand how to parse that type of metadata. By setting the format to docker, we’re using a format that almost all container runtimes can understand.

# Log into GitLab's container repository.
export REGISTRY_AUTH_FILE=${HOME}/auth.json
echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

Here we set a path for the auth.json that contains the credentials for talking to the container repository. We also use buildah to authenticate to GitLab’s built-in container repository. GitLab automatically exports these variables for us (and hides them in the job output), so we can use them here.

buildah bud -f builds/${IMAGE_NAME} -t ${IMAGE_NAME} .

We’re now building the container and storing it temporarily as the bare image name, such as fedora30. This is roughly equivalent to docker build.

CONTAINER_ID=$(buildah from ${IMAGE_NAME})
buildah commit --squash $CONTAINER_ID $FQ_IMAGE_NAME

Now we are making a reference to our container with buildah from and using that reference to squash that container down into a single layer. This keeps the container as small as possible.

The commit step also tags the resulting image using our fully qualified image name (in this case, it’s registry.gitlab.com/majorhayden/os-containers/fedora30:latest)

buildah push ${FQ_IMAGE_NAME}

This is the same as docker push. There’s not much special to see here.

Maintaining containers

GitLab allows you to take things to the next level with CI schedules. In my repository, there is a schedule to build my containers once a day to catch the latest updates. I use these containers a lot and they need to be up to date before I can run tests.

If the container build fails for some reason, GitLab will send me an email to let me know.

Use a secret as an environment variable in OpenShift deployments

major@mhtx.net (Major Hayden) — Thu, 06 Dec 2018 00:00:00 +0000

OpenShift deployments allow you to take a container image and run it within a cluster. You can easily add extra items to the deployment, such as environment variables or volumes.

The best practice for sensitive environment variables is to place them into a secret object rather than directly in the deployment configuration itself. Although this keeps the secret data out of the deployment, the environment variable is still exposed to the running application inside the container.

Creating a secret

Let’s start with a snippet of a deploymentConfig that has a sensitive environment variable in plain text:

spec:
 containers:
 - env:
 - name: MYAPP_SECRET_TOKEN
 value: vPWps5E7KO8KPlljaD3eXED3f6jmLsV5mQ
 image: "fedora:29"
 name: my_app

The first step is to create a secret object that contains our sensitive environment variable:

apiVersion: v1
kind: Secret
metadata:
 name: secret-token-for-my-app
stringData:
 SECRET_TOKEN: vPWps5E7KO8KPlljaD3eXED3f6jmLsV5mQ

Save this file as secret-token.yml and deploy it to OpenShift:

oc apply -f secret-token.yml

Query OpenShift to ensure the secret is ready to use:

$ oc get secret/secret-token-for-my-app
NAME TYPE DATA AGE
secret-token-for-my-app Opaque 1 1h

Using the secret

We can adjust the deployment configuration to use this new secret:

spec:
 containers:
 - env:
 - name: MYAPP_SECRET_TOKEN
 valueFrom:
 secretKeyRef:
 key: SECRET_TOKEN
 name: secret-token-for-my-app
 image: "fedora:29"
 name: my_app

This configuration tells OpenShift to look inside the secret object called secret-token-for-my-app for a key matching SECRET_TOKEN. The value will be passed into the MYAPP_SECRET_TOKEN environment variable and it will be available to the application running in the container.

Security note: If someone has access to change the deployment configuration object, they could get access to the value of the secret without having direct access to the secret object itself. It would be trivial to change the startup command in the container to print all of the environment variables in the container (using the env command) and view them in the container logs.

Research Paper: Securing Linux Containers

major@mhtx.net (Major Hayden) — Fri, 14 Aug 2015 20:45:50 +0000

It seems like there’s a new way to run containers every week. The advantages and drawbacks of each approach are argued about on mailing lists, in IRC channels, and in person, around the world. However, the largest amount of confusion seems to be around security.

Launching secure containers

I’ve written about launching secure containers on this blog many times before:

However, my goal this time around was to do something more comprehensive and slightly more formal. After getting my GSEC and GCUX certifications from SANS/GIAC, there was an option to enhance the certification to a gold status by writing a peer-reviewed research paper on a topic related to the exam. It was a great experience to go through the review process and get feedback on the technical material as well as the structure of the paper itself.

The paper

Without further ado, here are links to the Securing Linux Containers paper:

PDF version without watermarks
PDF version from SANS (has some watermarks and SANS/GIAC extra pages)

The paper is written for readers who have some level of familiarity with Linux and some virtualization technologies. It’s a useful paper even for people who haven’t worked with containers.

It starts with an overview of Linux containers and how they differ from other types of virtualization, such as KVM or Xen. From there, it covers how to secure the host system underneath the containers and how to provide security within the containers themselves. There’s also a section on how to start a simple container on CentOS 7 and inspect the security controls inside and outside the container.

Licensing

I’m also very proud to announce that the paper is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC-BY-SA). You are free to quote it as much as you like (even for commercial purposes), but I’d ask that you maintain the same license and attribute me as the author.

Thank you

This paper wouldn’t have been possible without some serious help from these awesome people:

Richard Carbone was my advisor from SANS and he helped tremendously
Dan Walsh reviewed the content and gave me several pointers on topics to add and adjust
Paul Voccio, Antony Messerli, and Brad McConnell from Rackspace also provided feedback
My mother, Neta Greene, is the best educator I know and she fueled my interest in writing and sharing with others

Feedback

Please let me know if you spot any errors or areas that need clarification. This is one of my favorite topics and I enjoy talking about it. Find me on Freenode IRC as mhayden and I’ll be glad to talk more there.

Improving LXC template security

major@mhtx.net (Major Hayden) — Thu, 18 Jun 2015 19:52:11 +0000

I’ve been getting involved with the Fedora Security Team lately and we’re working as a group to crush security bugs that affect Fedora, CentOS (via EPEL) and Red Hat Enterprise Linux (via EPEL). During some of this work, I stumbled upon a group of Red Hat Bugzilla tickets talking about LXC template security.

The gist of the problem is that there’s a wide variance in how users and user credentials are handled by the different LXC templates. An inventory of the current situation revealed some horrifying problems with many OS templates.

Many of the templates set an awful default root password, like rooter, toor, or root. Some of the others create a regular user with sudo privileges and give it a default, predictable password unless the user specifies otherwise.

There are some bright spots, though. Fedora and CentOS templates will accept a root password from the user during the build and set a randomized password for the root user if a password isn’t specified. Ubuntu Cloud takes another approach by locking out the root user and requiring cloud-init configuration data to configure the root account.

I kicked off a mailing list thread and wrote a terrible pull request to get things underway. Stéphane Graber requested that all templates use a shared script to handle users and credentials via standardized environment variables and command line arguments. In addition, all passwords for users (regular or root) should be empty with password-less logins disabled. Those are some reasonable requests and I’m working on a shell script that’s easy to import into LXC templates.

There’s also a push to remove sshd from all LXC templates by default, but I’m hoping to keep that one tabled until the credentials issue is solved.

If you’d like to help out with the effort, let me know! I’ll probably get some code up onto Github soon and as for comments.

Try out LXC with an Ansible playbook

major@mhtx.net (Major Hayden) — Wed, 17 Dec 2014 13:50:26 +0000

The world of containers is constantly evolving lately. The latest turn of events involves the CoreOS developers when they announced Rocket as an alternative to Docker. However, LXC still lingers as a very simple path to begin using containers.

When I talk to people about LXC, I often hear people talk about how difficult it is to get started with LXC. After all, Docker provides an easy-to-use image downloading function that allows you to spin up multiple different operating systems in Docker containers within a few minutes. It also comes with a daemon to help you manage your images and your containers.

Managing LXC containers using the basic LXC tools isn’t terribly easy - I’ll give you that. However, managing LXC through libvirt makes the process much easier. I wrote a little about this earlier in the year.

I decided to turn the LXC container deployment process into an Ansible playbook that you can use to automatically spawn an LXC container on any server or virtual machine. At the moment, only Fedora 20 and 21 are supported. I plan to add CentOS 7 and Debian support soon.

Clone the repository to get started:

git clone https://github.com/major/ansible-lxc.git
cd ansible-lxc
ansible-playbook -i hosts playbook.yml

If you’re running the playbook on the actual server or virtual machine where you want to run LXC, there’s no need to alter the hosts file. You will need to adjust it if you’re running your playbook from a remote machine.

As the playbook runs, it will install all of the necessary packages and begin assembling a Fedora 21 chroot. It will register the container with libvirt and do some basic configuration of the chroot so that it will work as a container. You’ll end up with a running Fedora 21 LXC container that is using the built-in default NAT network created by libvirt. The playbook will print out the IP address of the container at the end. The default password for root is fedora. I wouldn’t recommend leaving that for a production use container. ;)

All of the normal virsh commands should work on the container. For example:

# Stop the container gracefully
virsh shutdown fedora21
# Start the container
virsh start fedora21

Feel free to install the virt-manager tool and manage everything via a GUI locally or via X forwarding:

yum -y install virt-manager dejavu* xorg-x11-xauth
# OPTIONAL: For a better looking virt-manager interface, install these, too
yum -y install gnome-icon-theme gnome-themes-standard

Launch secure LXC containers on Fedora 20 using SELinux and sVirt

major@mhtx.net (Major Hayden) — Tue, 22 Apr 2014 04:11:00 +0000

Getting started with LXC is a bit awkward and I’ve assembled this guide for anyone who wants to begin experimenting with LXC containers in Fedora 20. As an added benefit, you can follow almost every step shown here when creating LXC containers on Red Hat Enterprise Linux 7 Beta (which is based on Fedora 19).

You’ll need a physical machine or a VM running Fedora 20 to get started. (You could put a container in a container, but things get a little dicey with that setup. Let’s just avoid talking about nested containers for now. No, really, I shouldn’t have even brought it up. Sorry about that.)

Prep Work

Start by updating all packages to the latest versions available:

yum -y upgrade

Verify that SELinux is in enforcing mode by running getenforce. If you see Disabled or Permissive, get SELinux into enforcing mode with a quick configuration change:

sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config

I recommend installing setroubleshoot-server to make it easier to find the root cause of AVC denials:

Reboot now. This will ensure that SELinux comes up in enforcing mode (verify that with getenforce after reboot) and it ensures that auditd starts up sedispatch (for setroubleshoot).

Install management libraries and utilities

Let’s grab libvirt along with LXC support and a basic NAT networking configuration.

yum -y install libvirt-daemon-lxc libvirt-daemon-config-network

Launch libvirtd via systemd and ensure that it always comes up on boot. This step will also adjust firewalld for your containers and ensure that dnsmasq is serving up IP addresses via DHCP on your default NAT network.

systemctl start libvirtd.service
systemctl enable libvirtd.service

Bootstrap our container

Installing packages into the container’s filesystem will take some time.

yum -y --installroot=/var/lib/libvirt/filesystems/fedora20 --releasever=20 --nogpg install systemd passwd yum fedora-release vim-minimal openssh-server procps-ng iproute net-tools dhclient

This step fills in the filesystem with the necessary packages to run a Fedora 20 container. We now need to tell libvirt about the container we’ve just created.

virt-install --connect lxc:// --name fedora20 --ram 512 --filesystem /var/lib/libvirt/filesystems/fedora20/,/

At this point, libvirt will know enough about the container to start it and you’ll be connected to the console of the container! We need to adjust some configuration files within the container to use it properly. Detach from the console with CTRL-].

Let’s stop the container so we can make some adjustments.

virsh -c lxc:// shutdown fedora20

Get the container ready for production

Hop into your container and set a root password.

chroot /var/lib/libvirt/filesystems/fedora20 /bin/passwd root

We will be logging in as root via the console occasionally and we need to allow that access.

echo "pts/0" >> /var/lib/libvirt/filesystems/fedora20/etc/securetty

Since we will be using our NAT network with our auto-configured dnsmasq server (thanks to libvirt), we can configure a simple DHCP setup for eth0:

cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network
NETWORKING=yes
EOF
cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
ONBOOT=yes
DEVICE=eth0
EOF

Using ssh makes the container a lot easier to manage, so let’s ensure that it starts when the container boots. (You could do this via systemctl after logging in at the console, but I’m lazy.)

chroot /var/lib/libvirt/filesystems/fedora20/
ln -s /usr/lib/systemd/system/sshd.service /etc/systemd/system/multi-user.target.wants/
exit

Launch!

Cross your fingers and launch the container.

virsh -c lxc:// start --console fedora20

You’ll be attached to the console during boot but don’t worry, hold down CTRL-] to get back to your host prompt. Check the dnsmasq leases to find your container’s IP address and you can login as root over ssh.

cat /var/lib/libvirt/dnsmasq/default.leases

Security

After logging into your container via ssh, check the process labels within the container:

# ps aufxZ
LABEL USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 1 0.0 1.3 47444 3444 ? Ss 03:18 0:00 /sbin/init
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 18 0.0 2.0 43016 5368 ? Ss 03:18 0:00 /usr/lib/systemd/systemd-journald
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 38 0.4 7.8 223456 20680 ? Ssl 03:18 0:00 /usr/bin/python -Es /usr/sbin/firewalld -
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 40 0.0 0.7 26504 2084 ? Ss 03:18 0:00 /usr/sbin/smartd -n -q never
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 41 0.0 0.4 19268 1252 ? Ss 03:18 0:00 /usr/sbin/irqbalance --foreground
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 44 0.0 0.6 34696 1636 ? Ss 03:18 0:00 /usr/lib/systemd/systemd-logind
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 46 0.0 1.8 267500 4832 ? Ssl 03:18 0:00 /sbin/rsyslogd -n
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 dbus 47 0.0 0.6 26708 1680 ? Ss 03:18 0:00 /bin/dbus-daemon --system --address=syste
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 rpc 54 0.0 0.5 41992 1344 ? Ss 03:18 0:00 /sbin/rpcbind -w
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 55 0.0 0.3 25936 924 ? Ss 03:18 0:00 /usr/sbin/atd -f
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 56 0.0 0.5 22728 1488 ? Ss 03:18 0:00 /usr/sbin/crond -n
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 60 0.0 0.2 6412 784 pts/0 Ss+ 03:18 0:00 /sbin/agetty --noclear -s console 115200
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 74 0.0 3.2 339808 8456 ? Ssl 03:18 0:00 /usr/sbin/NetworkManager --no-daemon
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 394 0.0 5.9 102356 15708 ? S 03:18 0:00 \_ /sbin/dhclient -d -sf /usr/libexec/nm
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 polkitd 83 0.0 4.4 514792 11548 ? Ssl 03:18 0:00 /usr/lib/polkit-1/polkitd --no-debug
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 rpcuser 110 0.0 0.6 46564 1824 ? Ss 03:18 0:00 /sbin/rpc.statd
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 111 0.0 1.3 82980 3620 ? Ss 03:18 0:00 /usr/sbin/sshd -D
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 409 0.0 1.9 131576 5084 ? Ss 03:18 0:00 \_ sshd: root@pts/1
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 413 0.0 0.9 115872 2592 pts/1 Ss 03:18 0:00 \_ -bash
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 438 0.0 0.5 123352 1344 pts/1 R+ 03:19 0:00 \_ ps aufxZ
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 411 0.0 0.8 44376 2252 ? Ss 03:18 0:00 /usr/lib/systemd/systemd --user
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 412 0.0 0.5 66828 1328 ? S 03:18 0:00 \_ (sd-pam)
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 436 0.0 0.4 21980 1144 ? Ss 03:19 0:00 /usr/lib/systemd/systemd-hostnamed

You’ll notice something interesting if you run getenforce now within the container — SELinux is disabled. Actually, it’s not really disabled. The processing of SELinux policy is done on the host. The container isn’t able to see what’s going on outside of its own files and processes. The libvirt documentation for LXC hints at the importance of this isolation:

A suitably configured UID/GID mapping is a pre-requisite to making containers secure, in the absence of sVirt confinement.

In the absence of the “user” namespace being used, containers cannot be considered secure against exploits of the host OS. The sVirt SELinux driver provides a way to secure containers even when the “user” namespace is not used. The cost is that writing a policy to allow execution of arbitrary OS is not practical. The SELinux sVirt policy is typically tailored to work with an simpler application confinement use case, as provided by the “libvirt-sandbox” project.

This leads to something really critical to understand:

Containers don’t contain

Dan Walsh has a great post that goes into the need for sVirt and the protections it can provide when you need to be insulated from potentially dangerous virtual machines or containers. If a user is root inside a container, they’re root on the host as well. (There’s an exception: UID namespaces. But let’s not talk about that now. Oh great, first it was nested containers and now I brought up UID namespaces. Sorry again.)

Dan’s talk about securing containers hasn’t popped up on the Red Hat Summit presentations page quite yet but here are some notes that I took and then highlighted:

Containers don’t contain. The kernel doesn’t know about containers. Containers simply use kernel subsystems to carve up namespaces for applications.
Containers on Linux aren’t complete. Don’t compare directly to Solaris zones yet.
Running containers without Mandatory Access Control (MAC) systems like SELinux or AppArmor opens the door for full system compromise via untrusted applications and users within containers.

Using MAC gives you one extra barrier to keep a malicious container from getting higher levels of access to the underlying host. There’s always a chance that a kernel exploit could bypass MAC but it certainly raises the level of difficulty for an attacker and allows server operators extra time to react to alerts.

Docker, trusted builds, and Fedora 20

major@mhtx.net (Major Hayden) — Wed, 26 Mar 2014 05:17:58 +0000

Docker is a hot topic in the Linux world at the moment and I decided to try out the new trusted build process. Long story short, you put your Dockerfile along with any additional content into your GitHub repository, link your GitHub account with Docker, and then fire off a build. The Docker index labels it as “trusted” since it was build from source files in your repository.

I set off to build a Dockerfile to provision a container that would run all of the icanhazip services. Getting httpd running was a little tricky, but I soon had a working Dockerfile that built and ran successfully on Fedora 20.

The trusted build process kicked off without much fuss and I found myself waiting for a couple of hours for my job to start. I was sad to see an error after waiting so long:

Installing : httpd-2.4.7-3.fc20.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file

Well, that’s weird. It turns out that cap_set_file is part of libcap that sets filesystem capabilities based on the POSIX.1e standards. You can read up on capabilities in the Linux kernel capabilities FAQ. (Special thanks to Andrew Clayton getting me pointed in the right direction there.)

Marek Goldmann ran into this problem back in September 2013 and opened a bug report. Marek proposed a change to the Docker codebase that would remove setfcap from the list of banned capabilities in the LXC template used by docker. Another workaround would be to use the -privileged option to perform a build in privileged mode (available in docker 0.6+).

Both of those workarounds are unavailable when doing trusted builds with docker’s index. Sigh.

I fired off an email to Docker’s support staff and received a quick reply:

Major,

We are aware of this issue, and we are currently working on a fix, and we hope to have something we can start testing this week. I’m not sure when we will be able to roll out the fix, but we are hoping soon. Until then, there isn’t anything you can do to work around it. Sorry for the inconvenience.

If anything changes, we will be sure to let you know.

Ken

It wasn’t the answer I wanted but it’s good to know that the issue is being worked. In the meantime, I’ll push an untrusted build of the icanhazip Docker container up to the index for everyone to enjoy.

Stay tuned for updates.

UPDATED 2014-08-08: Per Thomas’ comment below, this has been fixed upstream.

Containers on Major Hayden

Automatic container updates with Podman quadlets

Setting up a basic quadlet

Enabling automatic updates

Customizing update behavior

Monitoring updates

Further reading

Connect Caddy to Porkbun

DNS validation

Adding Porkbun support to Caddy

Automated Caddy builds with updates

Connecting to Porkbun

Renewals

Further reading

How I learned to stop worrying and love the CoreOS

Updates

Toolbox

Layering

Declarative provisioning

Pets and cattle

More questions?

Quadlets might make me finally stop using docker-compose

What’s a quadlet?

So why not stick with docker-compose?

Quaint quadlets quickly!

Launch the quadlets

Managing containers

Fixing a ghost database migration failure

Why is the site down?

Why did the database migration fail?

Going upstream

More searching

How I fixed it

Deploy a containerized Ghost blog 👻

Launch CoreOS

Preparing Caddy

Deploying containers

Ghost initial setup

Customizing Ghost

Switching to production domain

Launch a watchtower container via podman quadlets

Quadlets

Configure a quadlet on boot

Let’s go 🚀

CoreOS as a pet

What’s this pet stuff about?

How does CoreOS fit in?

My development tools are missing!

My tool won’t work in the toolbox.

How do updates work?

I have more questions.

Automatic container updates with watchtower

Enter watchtower

Deploy

Further considerations

Second try at self-hosting Mastodon

Early start

Self-hosted adventure

Official guide

docker-compose

kubernetes

Deployment

Generate the environment file

Delete the deployment (for real)

Add Caddy

Initialize Mastodon

Next steps

docker-compose on Fedora CoreOS

Why docker-compose?

Why not kubernetes?

What about k3s?

Why don’t you use managed kubernetes?

What about podman?

What’s the big deal about CoreOS?

Adding docker-compose to CoreOS

Adventures with the mastodon herd

Why mastodon?

Migrating from Twitter

Apps

Run your own instance