Container-based Server Deployments: Worth it er Nah?

This document both explores and demonstrates the process of using composable bootc containers (RHEL Flavor) as the basis for your organizations server deployment strategy. The desired outcome is to produce an anaconda-iso for a scenario in which VMs can be provisioned at various specifications, but use the same underlying ISO for a boot-to-install installation of RHEL.

Background and setup

This Article Expects the following:

1. Per this article, we’re going to work from a RHEL 9 Workstation sub’d to a developer account. This article was authored on an M3 Max MacBook Pro and as such will use aarch64 releases and packages.
2. You are using podman

the link above navigates to the installation instructions, however, podman CLI is present by default on RHEL Server 9.x

1. From inside podman, you have signed into redhat’s container registry
   1. from within podman desktop
      1. Settings gear, bottom left.
      2. Registries Tab
      3. RedHat Container Registry – Use your developer account to sign in.
   2. from within Podman CLIpodman login registry.redhat.io

This content is available in Anytype.

Follow link to ask a permission to get the content

It might be worth noting that while things are pretty straightforward with docker and their container registry, RedHat has users connect to registry.redhat.io, but you browse the registry from catalog.redhat.com.

Why bootc Containers in the first place?

Bootc containers are actually radically disruptive and could potentially change the nature of what it is to deploy and patch systems in the future.

In the Current Paradigm

In the current paradigm, in the best-case scenario, server operating systems are deployed using various methods and tools (VMWare/AHV templates, Windows Deployment Services, Ignition Files, Bootstrap Scripts in a number of languages, anaconda-isos). Once the best-case scenario operating system is deployed you have to worry about patching (WSUS, Public Repos, Satellite, 3rd Party ITSM Packages). Packages, once applied, become part of the “monolithic” operating system and if something breaks you’re likely/typically fricked and might have to restore the VM from a back up prior to the patch OR you would have to re-provision the VM altogether all this probably after you’ve done some troubleshooting the server in the first place to see if it was salvage able.

In the Bootc Paradigm

ENTER BOOTC CONTAINERS!

• Bootc containers Provide an In-place operating system, As opposed to installing patches on a cadence to an existing operating system (Operating System + Updates) your operating system is now released as a wholistic component where there is no concept of updates, no individual package updates, just an operating system image that’s been move forward so much and released at a new tag.
• Bootc containers provide “transact-ability”: Bootc container opearting systems are transactable. Pull the latest version of the operating system into your stack and if it breaks anything you can simply re-define the previous version in and you’re back in business.
• Bootc Containers can talk to hardware: Up to this point, we’ve used containers for user-space applications. However, bootc containers contain the straight-up kernel, meaning that thanks to ( ( ( tooling ) ) ) these containers with kernel space can be applied directly to bare metal. So instead of templates, convoluted deployment tools and various other what-nots you literally compose your server image with the same Dockerfile/Containerfile as a traditional user-space container and its good to deploy.

It’s worth noting, probably, that this actually means hardware selection is GREATLY simplified as you can track hardware support in kernel version shipped with each image to know how your hardware will behave, for example, ARC GPUs

• We can now apply container security scans on the kernel, drivers, the bootloadeder and more. This basically means that all the tools we’ve come up with to rapidly scanning, validation, and signing on the Kernel, Drivers and Bootlader

Dockerfile vs Containerfile, Considerations

• TLDR, For now stick with a Dockerfile to compose your images. certain syntax is ignored in Containerfile files when using RHEL 9 bootc images.

Cover Your Computes

Its worth explicitly pointing out that this means you can run the same image both in kubernetes clusters and servers a like. Adopting this approach as a best practice might result in:

1. Improve organizational agility by making multi-paradigm pipelines a part of a standard build processes in product and cloud. Workloads would become homogeneous within a couple sprint cycles bringing unprecedented opportunities for administration, observability, right-sizing footprint, and SAVE MONEY
2. Improve organizational cohesion thought a shared set of standard tools, minimize technical debit, “it’s familiar to everyone”
3. Be Multi-cloud Ready – Deploy where ever VMs and Containers are sold. Though a impact study on licensing and opensource alternatives is a good idea. (Windows containers have licenses/restrictions :))

Getting Started

To get started, we want to get access to the container, poke around, and maybe try a rudimentary experiment to “see something work” in the container. So below we will focus on pulling different versions of a Red Hat Enterprise Linux, Poke around the container metadata and functionality, and then wrap up by deploying a simple web app and python script

Pulling the container image

podman pull registry.redhat.io/rhel9/rhel-bootc:latest

Something to look at later: looks like there’s a candidate in rhel8/rhel-bootc:latest so might be worth seeing if we

Composing our First Container OS

In this section, we’re going to go line-by-line down an example server template in a versioned directory name according to the server template, which in this case is: COMPANYNAME-RHEL9BC-SERVER. You’ll notice below that I’m providing the equivalent Ansible syntax to illustrate how blisteringly agile bootc containers are against traditional deployment tools

Install a Package

Installing packages happen just how you think

Bootc paradigm (Dockerfile)	Ansible (InstallNGINX.yaml)
RUN dnf install nginx	—- – name: Install NGINX hosts: localhost become: true tasks: – name: Install Nginx ansible.builtin.dnf: name: nginx state: present

Configure The Firewall

It’s Triple membrained – you can thinking about container networking like the image below. The container has an OS firewall like normal that must be configured, for better or worse, that’s not enough if the container

Theoretically** needs to be tested. You still need to publish ports in your container solution and it hasn’t been observed if the build process counts as the first run.

Bootc paradigm (Dockerfile)	Ansible (InstallNGINX.yaml)
RUN firewall-cmd —add-service=http —permanent EXPOSE 80/tcp RUN firewall-cmd —add-service=https —permanent EXPOSE 443/tcp RUN firewall-cmd reload	—- – name: Open Port 80 and 443 hosts: localhost become: true tasks: – name: Open Port 80 ansible.posix.firewalld: service: http state: enabled permanent: true immediate: true – name: Open Port 443 ansible.posix.firewalld: service: https state: enabled permanent: true immediate: true

Configure secrets into Env

Please don’t argue abstraction. This provides a system level secret like an API key for use by your software so you don’t have to ship secrets with your code.

Bootc paradigm (Dockerfile)	Ansible (InstallNGINX.yaml)
RUN “ECHO secret_key=!(unf9u47nfqwofn20u4fb20ugfb20 > /etc/profile”	—- – name: Open Port 80 and 443 hosts: localhost become: true tasks: – name: Add Environmental variablemn ansible.builtin.lineinfile: path: /etc/profile line: secret_key=!(unf9u47nfqwofn20u4fb20ugfb20

Deploy Custom Content

[TODO]

Create ISO Image Part One: Create Bootc container, Upload to registry

Customizing the image happens both in the Dockerfile and in the config.json or config.tool file you supply when you execute boot-image-builder. So for the purposes of this very limited demdemo we’re going to evaluate a couple functionalities that are essential for securely deploying and troubleshooting applications:

1. Can I Install a package from the RH’s repos?
   1. In this demo that will be the nginx package
2. Can I deploy my application to the container?
   1. In this demo, it’s an index.html file to NGINXs web root. A beefer example of this might be deploying a Flutter application compiled for web
3. Can I add an environmental variables at the system level for application to consume? (like hiding a api key. This is “secrets management”)
   1. We’re going to add an environmental variable: SEEKRTS=’ThatsWhyItsASecret’

Bootc paradigm (Dockerfile)	Ansible (InstallNGINX.yaml)
# INSTALL THE OS – AKA Pull The Latest bootc OS Image FROM: registry.redhat.io/rhel9/rhel-bootc:latest # Install and enable HTTPD server RUN dnf install httpd && systemctl enable httpd # “deploy application” COPY index.html /usr/share/httpd/html/ # Create a secret in the container for our application to use RUN “ECHO SEEKRTS=’ThatsWhyItsASecret’ > /etc/profile”	(… Install and configure a whole Operating System …) —- – name: Install NGINX hosts: localhost become: true tasks: – name: Install Nginx ansible.builtin.dnf: name: nginx state: present – name: Deploy application ansible.builtin.copy: src: ./index.html dest: /usr/share/nginx/html/ – name: Add Environmental variablemn ansible.builtin.lineinfile: path: /etc/profile line: SEEKRTS=’ThatsWhyItsASecret’

Let’s start by creating our Dockerfile for this custom Image

FROM registry.redhat.io/rhel9/rhel-bootc:latest
RUN dnf -y install httpd && systemctl enable httpd
#Optional Cloud-init
#dnf -y install cloud-init &&ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && dnf clean all
COPY index.html /usr/share/httpd/html
RUN echo seekrts='thatswhyitssecret' > /etc/profile

And now we’ll build the custom bootc container image that will server as the template of our custom ISO

#: podman build . -t customwebapp-bootc:0.2
STEP 1/5: FROM registry.redhat.io/rhel9/rhel-bootc:latest
STEP 2/5: RUN dnf -y install httpd && systemctl enable httpd && dnf clean all
--> Using cache 1a7aa7a5d42553ad624a6d97e243edc92b7aab33f39219fff47c85cf8ce1b977
--> 1a7aa7a5d425
STEP 3/5: RUN dnf -y install cloud-init &&     ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants &&     dnf clean all
--> Using cache 7bbce93ea3ac68bc9558599fd3e1562e31b953d6c53f2a9027ff85d25d785fe6
--> 7bbce93ea3ac
STEP 4/5: COPY index.html /usr/share/httpd/html/
--> Using cache fe9ad86098bf817ac7a1ab60b41b7b43acf693dbb477ca6836ea29f1f4afcf7d
--> fe9ad86098bf
STEP 5/5: RUN echo seekrts=thatswhyitsseekrt > /etc/profile
--> Using cache dbc87b7a950a44270457c5af0d3f47767459bdbe1c772790ab775f2fa7db7f90
COMMIT customwebapp-bootc:0.2
--> dbc87b7a950a
Successfully tagged localhost/customwebapp-bootc:0.2
dbc87b7a950a44270457c5af0d3f47767459bdbe1c772790ab775f2fa7db7f90

Finally, let’s push it to a registry so we can use it with bootc-image-builder

List our Images

[vmadmin@RHEL-BOOTC Company-cvmt-auctiontemplate]$ podman image ls
REPOSITORY                           TAG         IMAGE ID      CREATED      SIZE
localhost/customwebapp-bootc         0.2         dbc87b7a950a  2 hours ago  1.74 GB
localhost/sample-build               0.2         7fd558d797e8  3 hours ago  1.73 GB
localhost/sample-build               0.1         eda7be9f7804  3 hours ago  1.55 GB
registry.redhat.io/rhel9/rhel-bootc  latest      6b73e1d4ff64  2 days ago   1.48 GB

Sign in To Our Registry

[vmadmin@RHEL-BOOTC Company-cvmt-auctiontemplate]$ podman login quay.io
Username: kemmetdaniel
Password: 
Login Succeeded!

Make sure you’ve made a registry at quay.io for your project, for me, that quay.io/<username>/customwebapp-bootc

now lets push our latest image to quay (replay command for illustration)

You don’t seem to be able to push these containers to external repos. I got a permission denied error for a specific FS layer and I’m wondering if that’s red hat denying redistribution? but then there’s this: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/building_running_and_managing_containers/assembly_working-with-container-images_building-running-and-managing-containers#proc_redistributing-ubi-images_assembly_working-with-container-images

[vmadmin@RHEL-BOOTC Company-cvmt-auctiontemplate]$ podman push dbc87b7a950a quay.io/kemmetdaniel/customwebapp-bootc
Getting image source signatures
Copying blob 73299d564c44 skipped: already exists  
Copying blob 73299d564c44 skipped: already exists  
Copying blob 73299d564c44 skipped: already exists  
Copying blob 73299d564c44 skipped: already exists  
Copying blob 73299d564c44 skipped: already exists  
Copying blob 8bfc0e089ba9 skipped: already exists  
Copying blob 8bfc0e089ba9 skipped: already exists  
Copying blob 023ad0147de4 skipped: already exists  
Copying blob 023ad0147de4 skipped: already exists  
Copying blob 023ad0147de4 skipped: already exists  
Copying blob 902c65b86464 skipped: already exists  
Copying blob 902c65b86464 skipped: already exists  
Copying blob 902c65b86464 skipped: already exists  
Copying blob 6d0697e1538f skipped: already exists  
Copying blob 6d0697e1538f skipped: already exists  
Copying blob e23ce3682769 skipped: already exists  
Copying blob e23ce3682769 skipped: already exists  
Copying blob e23ce3682769 skipped: already exists  
Copying blob e23ce3682769 skipped: already exists  
Copying blob 6a30f81fe9bd skipped: already exists  
Copying blob 6a30f81fe9bd skipped: already exists  
Copying blob 85878db74f63 skipped: already exists  
Copying blob 85878db74f63 skipped: already exists  
Copying blob 0d225f690d87 skipped: already exists  
Copying blob 0d225f690d87 skipped: already exists  
Copying blob 0d225f690d87 skipped: already exists  
Copying blob 0d225f690d87 skipped: already exists  
Copying blob b801484b4058 skipped: already exists  
Copying blob b801484b4058 skipped: already exists  
Copying blob b801484b4058 skipped: already exists  
Copying blob c80444e8da1e skipped: already exists  
Copying blob c80444e8da1e skipped: already exists  
Copying blob d3019f07d560 skipped: already exists  
Copying blob d3019f07d560 skipped: already exists  
Copying blob d3019f07d560 skipped: already exists  
Copying blob 253e5f6c1818 skipped: already exists  
Copying blob 253e5f6c1818 skipped: already exists  
Copying blob ea63be1073b5 skipped: already exists  
Copying blob ea63be1073b5 skipped: already exists  
Copying blob ea63be1073b5 skipped: already exists  
Copying blob ea63be1073b5 skipped: already exists  
Copying blob 86db9a7d11a5 skipped: already exists  
Copying blob 86db9a7d11a5 skipped: already exists  
Copying blob e718848b472a skipped: already exists  
Copying blob e718848b472a skipped: already exists  
Copying blob e718848b472a skipped: already exists  
Copying blob 6436b45d7584 skipped: already exists  
Copying blob 6436b45d7584 skipped: already exists  
Copying blob 6436b45d7584 skipped: already exists  
Copying blob b72788d3bc17 skipped: already exists  
Copying blob b72788d3bc17 skipped: already exists  
Copying blob b72788d3bc17 skipped: already exists  
Copying blob a5f4b0fa0ba7 skipped: already exists  
Copying blob a5f4b0fa0ba7 skipped: already exists  
Copying blob a5f4b0fa0ba7 skipped: already exists  
Copying blob 0643e31e10c3 skipped: already exists  
Copying blob 0643e31e10c3 skipped: already exists  
Copying blob 21b080004e82 skipped: already exists  
Copying blob 21b080004e82 skipped: already exists  
Copying blob 123a23fee02f skipped: already exists  
Copying blob 123a23fee02f skipped: already exists  
Copying blob 123a23fee02f skipped: already exists  
Copying blob 2e76eb312a23 skipped: already exists  
Copying blob 2e76eb312a23 skipped: already exists  
Copying blob 2e76eb312a23 skipped: already exists  
Copying blob 3a29477447e8 skipped: already exists  
Copying blob 3a29477447e8 skipped: already exists  
Copying blob d9494cc0368c skipped: already exists  
Copying blob d9494cc0368c skipped: already exists  
Copying blob d9494cc0368c skipped: already exists  
Copying blob 9f86bf49766d skipped: already exists  
Copying blob 9f86bf49766d skipped: already exists  
Copying blob f543bc96340b skipped: already exists  
Copying blob f543bc96340b skipped: already exists  
Copying blob f543bc96340b skipped: already exists  
Copying blob f543bc96340b skipped: already exists  
Copying blob 4070d2ab9780 skipped: already exists  
Copying blob 4070d2ab9780 skipped: already exists  
Copying blob 4070d2ab9780 skipped: already exists  
Copying blob 356559c8327a skipped: already exists  
Copying blob 356559c8327a skipped: already exists  
Copying blob 490246637928 skipped: already exists  
Copying blob 490246637928 skipped: already exists  
Copying blob 034a5f021d05 skipped: already exists  
Copying blob 034a5f021d05 skipped: already exists  
Copying blob 034a5f021d05 skipped: already exists  
Copying blob 034a5f021d05 skipped: already exists  
Copying blob 034a5f021d05 skipped: already exists  
Copying blob ebdfea18c39b skipped: already exists  
Copying blob ebdfea18c39b skipped: already exists  
Copying blob da954960b83f skipped: already exists  
Copying blob ffddc307e1e3 skipped: already exists  
Copying blob f3b3631bd2ae skipped: already exists  
Copying blob fa726281edb3 skipped: already exists  
Copying blob b59d3cdd9df2 skipped: already exists  
Copying blob 618380a55c16 skipped: already exists  
Copying blob 56ae520906ff skipped: already exists  
Copying blob 7ef8812459d2 skipped: already exists  
Copying blob 4a1b9b497f96 skipped: already exists  
Copying blob b05a7094d63e skipped: already exists  
Copying blob f013018cca8d skipped: already exists  
Copying blob 650df335ecf3 skipped: already exists  
Copying blob 946a25fb0c0f skipped: already exists  
Copying blob c3cda956eee9 skipped: already exists  
Copying blob 9c30fcf60cfb skipped: already exists  
Copying blob 24255c22dc7a skipped: already exists  
Copying blob 7463ebbdaf9c skipped: already exists  
Copying blob 4006ed542b3a skipped: already exists  
Copying blob 18565b0d3c9c skipped: already exists  
Copying blob 223cda199c6a skipped: already exists  
Copying blob 8c1ca12adb00 skipped: already exists  
Copying blob 10dc7ee51efa skipped: already exists  
Copying blob ce8e6a0addbb skipped: already exists  
Copying blob 8fb4963935b3 skipped: already exists  
Copying blob 58c400c02302 skipped: already exists  
Copying blob 4a378d2224bd skipped: already exists  
Copying blob ff89bfb07d32 skipped: already exists  
Copying blob 487a8f8e79ed skipped: already exists  
Copying blob 2c777a89e1fd skipped: already exists  
Copying blob 7bef611c89b5 skipped: already exists  
Copying blob a915cf96ae67 skipped: already exists  
Copying blob bc0d7c1a77ee skipped: already exists  
Copying blob ad312c5c40cc skipped: already exists  
Copying blob 9d430ec6fa16 skipped: already exists  
Copying blob 9110bb23e07f skipped: already exists  
Copying blob d14913e2aab5 skipped: already exists  
Copying blob bd9ddc54bea9 skipped: already exists  
Copying blob 845179fef43d skipped: already exists  
Copying config dbc87b7a95 done   | 
Writing manifest to image destination

and verify it’s in the registry

Create ISO Image Part Two: Use bootc-image-builder to produce a bootable ISO

Okay great so now we have a custom bootc image definition we expect this image to have httpd installed and running, and to have a new index file placed. Now, we want to take that Bootc container and make it into an ISO so we can boot and install that image on to bare metal or in a VM

Creating a config.toml file to configure out “base image”

Documentation here, it looks like there are a lot of different things you can do so this probably warrants a deeper exploration. Otherwise, below I’ve created an example config.toml file that adds a user to the system, vmadmin who has both a password and a public key for access to the VM/server once it’s deployed.

[[blueprint.customizations.user]]
name = "vmadmin"
password = "Seekrtz123!"
key = "ssh-ed25519 jfnAIPJDOFAIDSFJOASIHBFNJOPSABINOA[SPBNUO[ASBOUDF COMPUTERNAME"
groups = ["wheel"]

Identifying the ISO build syntax

syntax is:

sudo podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v $(pwd)/config.toml:/config.toml:ro -v $(pwd)/output:/output registry.redhat.io/rhel9/bootc-image-builder:latest --type iso quay.io/kemmetdaniel/customwebapp-bootc

explaination

Command Component	Switch
sudo podman run —rm	run the container, removing it as soon as execution completes
—it	run the container interactively allocating a TTY console
—privileged	A “privileged” container is given the same access to devices as the user launching the container, with the exception of virtual consoles (/dev/tty\d+) when running in systemd mode
—pull=newer	pull image policy, never pull on run, or newer pull if there’s a newer image. basically this is like saying run the latest version
—security-opt label=type:unconfined_t	Security options, here I believe we using to for something SELinux releated, but I’m not sure what other than “unconfined_t”
-v	Volumes, we’re making 2 different volumes and binding them to the bootc-image-builder container. 1 provides our config.toml file the other uses a output/ as storage for our finished ISO
registry.redhat.io/rhel9/bootc-image-builder:latest	path to the latest version of bootc-image-builder from Redhat
—type iso	we’re telling bootc-image-builder we want an ISO as opposed to QCOW wmdk or whatever else.
quay.io/kemmetdaniel/customwebapp-bootc	registry path to the bootc container re created above.

Calamity Ensues:

and you come out the otherside with a bootable ISO

The ISO goes through the normal set up process but touchless. It’s quick

#modal-sidebar

promising to see HTTPD in the Kernel Messages! :O

Conclusion

So it looks like everything is configured as expected minus the index.html file being in the wrong place (that’s on me), to be picked up by httpd. So all in all, great success. I have a container and a deployment ISO for an identical release of my workload. I’d imagine container builds would go into a CD pipeline to Kubernetes or CoreOS and the ISOs to your hypervisor for use in VM Build config templates.

The Open Source version of bootc-image-builder Appears to be: quay.io/centos-bootc/bootc-image-builder:latest The RedHat version of bootc-image-builder is: registry.redhat.io/rhel9/bootc-image-builder:9.4-1730814784

Additional Reading

• What is a Bootable Container?, fedoraproject.org
• How to use the —privileged flat with container engines, redhat.com