Imagebuilder on Major Hayden

Build a custom CentOS Stream 9 cloud image

major@mhtx.net (Major Hayden) — Fri, 06 May 2022 00:00:00 +0000

This is my third post about Image Builder, so I guess you could say that I enjoy using it¹. It’s a great way to define a custom cloud image, build it, and (optionally) ship it to a supported cloud provider.

This post covers how to build a customized CentOS Stream 9 image along with a custom repository for additional packages. In this case, that’s Extra Packages for Enterprise Linux (EPEL).

Why do I need a custom image anyway?

Building your own image empowers you to choose which packages you want, which services run at boot time, and where you deploy your image. Some cloud providers may not have an image from the Linux distribution you enjoy most, or they might have an image with the wrong package set.

Some cloud providers build images with too many packages or too few. Sometimes they add configuration that doesn’t exist in the OS itself. I’ve even found some that alter cloud-init and force you to log in directly as the root user. 😱

I enjoy building my own images so I know exactly what it contains and I know that the configuration came from the OS itself.

First steps

This post uses CentOS Stream 9 as an example. You will need a physical host, virtual machine, or cloud instance running CentOS Stream 9 first. We start by installing some packages:

$ sudo dnf install osbuild-composer weldr-client

What do these packages contain?

osbuild-composer ensures you have osbuild, the low-level image build component, along with configuration and an osbuild-composer² worker that builds the image.
weldr-client contains the composer-cli command line tool that makes it easy to interact with osbuild-composer

One nice thing about this stack is that it starts via systemd’s socket activation and it only runs when you query it. Let’s start the socket now and ensure it comes up on a reboot:

$ sudo systemctl enable --now osbuild-composer.socket

Verify that the API is responding:

$ composer-cli status show
API server status:
 Database version: 0
 Database supported: true
 Schema version: 0
 API version: 1
 Backend: osbuild-composer
 Build: NEVRA:osbuild-composer-46-1.el9.x86_64

Adding EPEL

CentOS Stream 9 has most of the packages I want, but I really love this program called htop that displays resource usage and allows you to introspect certain processes or namespaces easily. This package is only available in the EPEL repository, so we need to add that one to our list of enabled repositories for image builds.

osbuild-composer comes with its own set of repositories in the package and does not use the system’s repositories. You can list all the enabled repositories that it knows about:

# composer-cli sources list
AppStream
BaseOS
RT

If we want to add EPEL, we can dump the configuration from one of these to a file and edit it:

# composer-cli sources info AppStream | tee epel.ini
check_gpg = true
check_ssl = true
id = "AppStream"
name = "AppStream"
rhsm = false
system = true
type = "yum-baseurl"
url = "https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/"

After editing the EPEL repository file, it should look like this:

check_gpg = true
check_ssl = true
id = "EPEL9"
name = "EPEL9"
rhsm = false
system = false
type = "yum-baseurl"
url = "https://mirrors.kernel.org/fedora-epel/9/Everything/x86_64/"

You can use any mirror you prefer, but the kernel mirrors are very fast for me from most locations. Now we need to add this repository:

# composer-cli sources add epel.ini
# composer-cli sources list
AppStream
BaseOS
EPEL9
RT

We now have EPEL9 in our list. 🎉

Define our image

All image definitions, or blueprints, are in TOML format. Here’s my simple one for this post:

# Save this file as image.toml
name = "centos9"
description = "Major's awesome CentOS 9 image"
version = "0.0.1"

[[packages]]
name = "tmux"

[[packages]]
name = "vim"

# This is the one that comes from EPEL.
[[packages]]
name = "htop"

Now we push our blueprint and solve the dependencies to ensure we added our EPEL repository properly:

# composer-cli blueprints push image.toml
# composer-cli blueprints depsolve centos9
 -- SNIP --
 2:vim-filesystem-8.2.2637-16.el9.noarch
 which-2.21-27.el9.x86_64
 xz-5.2.5-7.el9.x86_64
 xz-libs-5.2.5-7.el9.x86_64
 zlib-1.2.11-33.el9.x86_64
 htop-3.1.2-3.el9.x86_64

And there’s htop at the end of the list! 🎉

Make the image

The fun part has arrived! Let’s build an image:

# composer-cli compose start centos9 ami --size=4096
Compose ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e added to the queue

This command does a few things:

Starts an image build with our centos9 blueprint (from the name section of my image.toml file)
Outputs an image type that works well on AWS (Amazon Machine Image, or AMI)
Limits the image size to 4GB (be sure this is not too large for your preferred instance size)

🤔 Note that you do not need to set the size explicitly here, but I do it as a good measure. When your instance boots, cloud-init runs growpart to expand the storage to fit the disk size in your cloud instance.

💣 However, growpart will not shrink the disk at boot time. If you choose a size that is larger than the disk space in your cloud instance, you will likely see an error at provisioning time.

Let’s check the status after a few minutes:

# composer-cli compose status
ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e FINISHED Fri May 6 16:16:08 2022 centos9 0.0.1 ami 4096

If you want to get a copy of the image and import it yourself into your favorite cloud, you can do that now:

# composer-cli compose image ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e
ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e-image.raw
# ls -alh ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e-image.raw
-rw-------. 1 root root 2.7G May 6 16:20 ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e-image.raw

You can also let osbuild-composer do this for you! I have one post on this blog about automatically uploading to AWS and another post on the Red Hat blog about doing the same with Azure.

I once worked on the team that makes Image Builder happen, so I may be a little bit biased. Enjoy the post anyway. 😉 ↩︎
Most people at Red Hat just call it composer, but I’ll use the full name here to avoid confusion. ↩︎

Deploy a custom Fedora 35 AMI to AWS with Image Builder

major@mhtx.net (Major Hayden) — Tue, 16 Nov 2021 00:00:00 +0000

Fedora reigns supreme as my Linux distribution of choice when I deploy new workloads to public clouds. It gives me a well-tested, modern Linux system with tons of helpful tools.

Fedora’s cloud images provide a great base to begin building a cloud deployment, but sometimes I find myself wanting a highly customized image with some features I care about. For example, I may want some packages pre-installed that aren’t included with the default cloud image, or I may want certain services stopped or started at boot time.

Fortunately, Fedora includes Image Builder. Image Builder does the hard work of building your image, uploading it to a cloud provider, and then registering that image. It has built-in support for handing AMIs (Amazon Machine Images) at AWS and you can use it as a one-stop-shop for customizing an AMI for your use case.

Configure the AWS cli

Start by installing the awscli package:

$ sudo dnf install awscli

The cli will need some way to access your AWS account. I usually create an administrative user and attach an administrative policy to the user. This allows me to control my entire account via the cli tool.

💣 (Note: Although this works well for my simple uses here, be careful using administrative users on large accounts. You may want to be more restrictive with your IAM policies, but that’s a topic for a different post.)

We start by accessing the IAM dashboard in your AWS account. Follow these steps:

Click Users on the left and then Add users on the right
Enter a name for your user (I used desktop-cli)
Click the Access key - Programmatic access checkbox
Click Next: Permissions
Click Attach existing policies directly and then tick the box next to Administrator Access
Click Next: Tags and add any tags if your organization requires them
Click Next: Review and then Create User

The next screen shows your Access key ID and your Secret access key. Click Show to display your secret access key. Go back to your cli tool and run the configuration command:

$ aws configure --profile personal
AWS Access Key ID [None]: << enter your access key id here >>
AWS Secret Access Key [None]:<< enter your secret access key here >>
Default region name [None]: us-east-1
Default output format [None]:

Let’s verify that the credentials work:

$ aws --profile personal sts get-caller-identity
{
 "UserId": "xxx",
 "Account": "xxx",
 "Arn": "arn:aws:iam::xxx:user/desktop-cli"
}

To avoid typing your profile over and over, just export an environment variable:

$ export AWS_PROFILE=personal

Preparing permissions for image imports

When you import images into AWS, some services must take action on your behalf to copy your image from S3 into an EBS snapshot, and then register that snapshot as an AMI. This requires an S3 bucket and some extra permissions for some AWS services.

First off, let’s create an S3 bucket to hold our image:

$ aws s3 mb s3://image-upload-bucket-blog-post
make_bucket: image-upload-bucket-blog-post

From here, we follow the AWS documentation for importing images with some slight modifications for Image Builder. Save this file as trust-policy.json:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": { "Service": "vmie.amazonaws.com" },
 "Action": "sts:AssumeRole",
 "Condition": {
 "StringEquals":{
 "sts:Externalid": "vmimport"
 }
 }
 }
 ]
}

Now we create the vmimport role with the policy that allows AWS to assume this role and import an image:

$ aws iam create-role --role-name vmimport \
 --assume-role-policy-document "file://trust-policy.json"

We need to set some policy for our new vmimport role now to limit what AWS is allowed to do in our account. Save the following as role-policy.json:

{
 "Version":"2012-10-17",
 "Statement":[
 {
 "Effect": "Allow",
 "Action": [
 "s3:GetBucketLocation",
 "s3:GetObject",
 "s3:ListBucket"
 ],
 "Resource": [
 "arn:aws:s3:::image-upload-bucket-blog-post",
 "arn:aws:s3:::image-upload-bucket-blog-post/*"
 ]
 },
 {
 "Effect": "Allow",
 "Action": [
 "ec2:ModifySnapshotAttribute",
 "ec2:CopySnapshot",
 "ec2:RegisterImage",
 "ec2:Describe*"
 ],
 "Resource": "*"
 }
 ]
}

🛑 Stop here and change image-upload-bucket-blog-post to the S3 bucket name that you used in the first step of this section.

$ aws iam put-role-policy --role-name vmimport --policy-name vmimport \
 --policy-document "file://role-policy.json"

Image Builder also needs a user that can perform some functions inside AWS to upload and import the image. Let’s create a policy for a new user and save it as image-builder-policy.json:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "VisualEditor0",
 "Effect": "Allow",
 "Action": [
 "ec2:CreateTags",
 "ec2:RegisterImage",
 "ec2:ImportSnapshot",
 "ec2:DescribeImportSnapshotTasks"
 ],
 "Resource": "*"
 },
 {
 "Sid": "VisualEditor1",
 "Effect": "Allow",
 "Action": [
 "s3:PutObject",
 "s3:GetObject",
 "s3:DeleteObject"
 ],
 "Resource": "arn:aws:s3:::image-upload-bucket-blog-post/*"
 }
 ]
}

Add the policy, create a user, and attach the policy to your user:

$ aws iam create-policy --policy-name imagebuilder \
 --policy-document "file://image-builder-policy.json"
$ aws iam create-user --user-name imagebuilder
$ aws iam attach-user-policy --user-name imagebuilder \
 --policy-arn arn:aws:iam::YOUR_ACCOUNT_NUMBER:policy/imagebuilder

The ARN for your IAM policy includes your account number and the ARN should appear after running the create-policy command.

Install Image Builder

It’s no secret that I love good command line tools over graphical interfaces, so we will follow the cli steps for Image Builder in the remainder of this post. Let’s start by installing everything we need for Image Builder and starting the socket activation unit:

$ sudo dnf install composer-cli osbuild-composer
$ sudo systemctl enable --now osbuild-composer.socket

Verify that osbuild-composer is listening:

$ composer-cli status show
API server status:
 Database version: 0
 Database supported: true
 Schema version: 0
 API version: 1
 Backend: osbuild-composer
 Build: NEVRA:osbuild-composer-%{epoch}:37-1.fc35.x86_64

That was easy!

Build and deploy our AMI

Image Builder uses specifications called blueprints. These are TOML files that tell Image Builder how to configure your image. You can configure these in many different ways, but here’s the one I’m using for this post:

name = "major-perfect-f35"
description = "Major's perfect Fedora 35 cloud image"
version = "0.0.1"

[[packages]]
name = "firewalld"

[[packages]]
name = "tmux"

[[packages]]
name = "vim"

[[packages]]
name = "zsh"

[[customizations.user]]
name = "major"
key = "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIcfW3YMH2Z6NpRnmy+hPnYVkOcxNWLdn9VmrIEq3H0Ei0qWA8RL6Bw6kBfuxW+UGYn1rrDBjz2BoOunWPP0VRM= major@amdbox"
shell = "/usr/bin/zsh"
groups = ["wheel"]

[customizations.timezone]
timezone = "America/Chicago"

[customizations.firewall.services]
enabled = ["ssh", "dhcpv6-client"]

[customizations.services]
enabled = ["sshd", "firewalld"]

I saved this file as image.toml. The next step involves pushing the blueprint and solving the dependencies in the blueprint:

$ composer-cli blueprints push image.toml
$ composer-cli blueprints depsolve major-perfect-f35

We need some extra configuration to tell Image Builder how to authenticate with AWS. Save this file as aws.toml:

provider = "aws"

[settings]
accessKeyID = "YOUR_ACCESS_KEY_ID"
secretAccessKey = "YOUR_SECRET_ACCESS_KEY"
bucket = "image-upload-bucket-blog-post"
region = "us-east-1"
key = "major-perfect-f35"

Replace the bucket name with your S3 bucket and set an S3 object name in key. To get your access key and secret access key, run this command:

$ aws iam create-access-key --user imagebuilder

Finally, we’re ready to tell Image Builder to deploy our image! Run this last command to start the compose:

composer-cli compose start major-perfect-f35 ami major-perfect-f35 aws.toml

Replace major-perfect-35 with the name in your blueprint. Now, follow along in the system journal as your image is deployed:

[AWS] 🚀 Uploading image to S3: image-upload-bucket-blog-post/major-perfect-f35
[AWS] 📥 Importing snapshot from image: image-upload-bucket-blog-post/major-perfect-f35
[AWS] 🚚 Waiting for snapshot to finish importing: import-snap-06bc48cb9779f98d8
[AWS] 🧹 Deleting image from S3: image-upload-bucket-blog-post/major-perfect-f35
[AWS] 📋 Registering AMI from imported snapshot: snap-02ed2710572b7b94b
[AWS] 🎉 AMI registered: ami-0964ea222b6a6711e

(Don’t ask me who put all of the emojis in the logging code. 🤭)

Let’s verify that our image is fully imported:

$ aws ec2 describe-images --filters "Name=tag:Name,Values=major-perfect-f35"
{
 "Images": [
 {
 "Architecture": "x86_64",
 "CreationDate": "2021-11-16T18:12:33.000Z",
 "ImageId": "ami-0964ea222b6a6711e",
 "ImageLocation": "xxx/major-perfect-f35",
 "ImageType": "machine",
 "Public": false,
 "OwnerId": "xxx",
 "PlatformDetails": "Linux/UNIX",
 "UsageOperation": "RunInstances",
 "State": "available",
 "BlockDeviceMappings": [
 {
 "DeviceName": "/dev/sda1",
 "Ebs": {
 "DeleteOnTermination": true,
 "SnapshotId": "snap-02ed2710572b7b94b",
 "VolumeSize": 6,
 "VolumeType": "gp2",
 "Encrypted": false
 }
 }
 ],
 "EnaSupport": true,
 "Hypervisor": "xen",
 "Name": "major-perfect-f35",
 "RootDeviceName": "/dev/sda1",
 "RootDeviceType": "ebs",
 "Tags": [
 {
 "Key": "Name",
 "Value": "major-perfect-f35"
 }
 ],
 "VirtualizationType": "hvm"
 }
 ]
}

Go forth and build instances with your new, customized Fedora 35 AMI! 🎉

Photo credit: Svetlozar Apostolov

Build Fedora AWS images in GitHub Actions with Image Builder

major@mhtx.net (Major Hayden) — Fri, 06 Aug 2021 00:00:00 +0000

GitHub Actions provides infrastructure for all kinds of amazing automation. Anyone can test software, build packages, deploy applications, or even publish a blog (like this one!) with a few snippets of YAML. I often use it to bundle my software in a container after testing it. 🤖

One day, as I was working through another Packer configuration, I wondered if there was a way to build cloud images directly in GitHub Actions without building an instance in the cloud, making tons of changes, and snapshotting that image. Building a cloud image without booting it first seems like a cleaner way to work and it seems like it could be an easier workflow.

I worked on the Image Builder team at Red Hat last year and really enjoyed the way we could build an image anywhere and then ship that image anywhere. That gave me an idea: What if I could use Image Builder in GitHub Actions to ship images to AWS with all the customizations I want, perhaps even on a schedule? 🤔

What is Image Builder?

The idea behind Image Builder is that anyone should be able to create images for various clouds and virtualization platforms with some simple software. Nobody should worry about how a particular cloud sets up cloud-init or what kernel configuration might be required to run in a particular cloud. Someone should do that for you and you should focus on what you need in your images to be successful at your task.

Image Builder has two parts:

osbuild: It drives the low-level image building processes. All of the loopback setup, image packaging, and per-cloud configuration adjustments all happen here.
osbuild-composer: It exposes different APIs which allow you to specify how the image should be built in a brief TOML configuration. You pass the TOML blueprint to osbuild-composer, tell it what type of image you want, and where you want the image delivered. It takes care of all of that.

You can get Image Builder on all current versions of Fedora, CentOS Stream, and Red Hat Enterprise Linux 8.3 or later. Install it via dnf:

$ sudo dnf install osbuild-composer
$ sudo systemctl enable --now osbuild-composer.socket

If you love to DIY (do it yourself), read my Build AWS images with Image Builder blog post from last summer. From here on out, I’ll only talk about consuming Image Builder via GitHub Actions.

Challenges

Building images in GitHub Actions comes with some challenges. For example, the only Linux choice is Ubuntu but Image Builder is not supported on Ubuntu currently. Luckily, we have containers!

I started with a repository to build containers with Image Builder included. The repository builds containers for Fedora 34, Fedora rawhide (the next Fedora release), and CentOS Stream 8. You can download these containers locally and run them, too:

# With podman
$ podman pull ghcr.io/major/imagebuilder:centos-stream8
$ podman pull ghcr.io/major/imagebuilder:fedora-34
$ podman pull ghcr.io/major/imagebuilder:fedora-rawhide
# With docker
$ docker pull ghcr.io/major/imagebuilder:centos-stream8
$ docker pull ghcr.io/major/imagebuilder:fedora-34
$ docker pull ghcr.io/major/imagebuilder:fedora-rawhide

Image Builder relies on systemd socket activation and that means systemd must be running inside the container. For nearly all systems, that requires adding the --privileged argument when you run the container and running the container as root. It’s not ideal, but it works fine in GitHub Actions since the instance is thrown away immediately after the image build is done.

To run these containers on your local system, you may need something like this:

$ sudo podman run --rm --detach --privileged --name imagebuilder \
 ghcr.io/major/imagebuilder:fedora-34

Initially, I thought the single CPU on the Actions runner would make the build process too slow, but I was pleasantly surprised to see that most builds finished in 4-7 minutes. The network throughput from the runner to AWS was also quite fast. 👏🏻

Building images

I often need a Fedora container or VM for doing packaging work and testing other contributors’ packages, so I set out to make a proof of concept for Fedora. My proof of concept is over in GitHub at major/imagebuilder-fedora. You can fork my PoC and customize everything as you wish!

The workflow follows a set of steps that I’ll explain below.

First, we need AWS credentials so we can drop off the image at AWS. There is a basic template in TOML format that contains placeholders for account credentials and other data:

provider = "aws"

[settings]
accessKeyID = "$AWS_ACCESS_KEY_ID"
secretAccessKey = "$AWS_SECRET_ACCESS_KEY"
bucket = "$AWS_S3_BUCKET"
region = "$AWS_DEFAULT_REGION"
key = "$IMAGE_KEY"

The actions workflow fills in that TOML file with information from GitHub Actions secrets and environment variables:

- name: Fill in the AWS template
 run: |
 cat shared/aws-template.toml | envsubst > shared/aws-config.toml
 env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
 AWS_S3_BUCKET: major-aws-image-import
 AWS_DEFAULT_REGION: us-east-1
 IMAGE_KEY: "${{ matrix.blueprint }}-${{ github.run_id }}"

From there, the workflow runs the build-image.sh script and here’s where the fun starts. The container starts up and we wait for the osbuild-composer API to respond:

# Start the container.
echo "🚀 Launching the container"
sudo podman run --rm --detach --privileged \
 -v $(pwd)/shared:/repo \
 --name $CONTAINER_NAME \
 $CONTAINER

# Wait for composer to be fully running.
echo "⏱ Waiting for composer to start"
for i in `seq 1 10`; do
 sleep 1
 composer-cli status show && break
done

Once the API is up, we push the blueprint into osbuild-composer and tell it to solve the dependencies. The depsolve step is optional, but it can find problems with your package set fairly quickly so you can make adjustments.

echo "📥 Pushing the blueprint"
composer-cli blueprints push /repo/${BLUEPRINT_NAME}.toml

echo "🔎 Solving dependencies in the blueprint"
composer-cli blueprints depsolve ${BLUEPRINT_NAME} > /dev/null

The blueprints are in the shared directory in the repository. For example, there’s a fedora-imagebuilder blueprint that builds an image with Image Builder inside it so you can build an image with Image Builder with Image Builder. (This reminds me of a meme. 🤭)

name = "fedora-imagebuilder"
description = "Image Builder - rawhide"
version = "0.0.2"
modules = []
groups = []

[[packages]]
name = "cockpit-composer"
version = "*"

[[packages]]
name = "osbuild"
version = "*"

[[packages]]
name = "osbuild-composer"
version = "*"

[customizations.services]
enabled = ["cockpit.socket", "osbuild-composer.socket"]

Now we’re ready to build the image (or in Image Builder terms, start the compose). After starting it, we extract the ID of the compose so we can monitor it while it runs.

if [[ $SHIP_TO_AWS == "yes" ]]; then
 echo "🛠 Build the image and ship to AWS"
 composer-cli --json \
 compose start $BLUEPRINT_NAME ami $IMAGE_KEY /repo/aws-config.toml \
 | tee compose_start.json > /dev/null
else
 echo "🛠 Build the image"
 composer-cli --json compose start ${BLUEPRINT_NAME} ami | tee compose_start.json
fi

COMPOSE_ID=$(jq -r '.body.build_id' compose_start.json)

I get a little nervous when I can’t see any status updates, so I follow the systemd journal while the build runs. The script checks on the build frequently to see if it has finished. This process takes about 4-7 minutes in GitHub Actions for most of the images I’ve built.

# Watch the logs while the build runs.
podman-exec journalctl -af &

COUNTER=0
while true; do
 composer-cli --json compose info "${COMPOSE_ID}" | tee compose_info.json > /dev/null
 COMPOSE_STATUS=$(jq -r '.body.queue_status' compose_info.json)

 # Print a status line once per minute.
 if [ $((COUNTER%60)) -eq 0 ]; then
 echo "💤 Waiting for the compose to finish at $(date +%H:%M:%S)"
 fi

 # Is the compose finished?
 if [[ $COMPOSE_STATUS != RUNNING ]] && [[ $COMPOSE_STATUS != WAITING ]]; then
 echo "🎉 Compose finished."
 break
 fi
 sleep 1

 let COUNTER=COUNTER+1
done

Once the images finish building and they deploy to AWS (usally less than 15 minutes altogether), you should be able to see them inside your AWS account:

Extra credit

Your automation doesn’t have to end here! 🤖

You can also add extra repositories to your compose with composer-cli sources add... if you have custom repositories with your software or if you need packages from RPMFusion.

GitHub Actions could boot an instance from your image, run some basic tests, and apply tags to the AMI to make provisioning easier. I always like having an AMI with a latest tag so I can deploy from the most recently built image whenever I need to test something.

Photo credit: Cameron Venti on Unsplash