In past few weeks I was asked on multiple occasions about how is Fedora Linux release composed. As a part of learning topic for Fedora Infrastructure Weekly meeting I already presented some snippets of how we compose artifacts. Since the meeting is in text and has fairly limitted time, I have decided to create this post.

Prerequirements

You should be familliar with package build process in fedora the tooling used to achieve reproducible builds. As well as diferencies between types of artifacts that are delivered by Fedora Release Engineering

Pungi basics

Pungi consists of multiple separate executables backed by a common library. The main entry-point is the pungi-koji script. It loads the compose configuration and kicks off the process. Composing itself is done in phases. Each phase is responsible for generating some artifacts on disk and updating the compose object that is threaded through all the phases. Pungi itself does not actually do that much. Most of the actual work is delegated to separate executables. Pungi just makes sure that all the commands are invoked in the appropriate order and with correct arguments. It also moves the artifacts to correct locations.

The executable name pungi-koji comes from the fact that most of those separate executables submit tasks to Koji that does the actual work in an auditable way. However unlike doing everything manually in Koji, Pungi will make sure you are building all images from the same package set, and will produce even deliverables that Koji can not create like DNF repos and installer ISOs.

Pungi config

The configuration file parser is provided by (kobo)[https://github.com/release-engineering/kobo] The file follows a Python-like format. It consists of a sequence of variables that have a value assigned to them.

Pungi phases

Phases are executed in sequential order (left-to-right in the diagram), but there are use cases where multiple phases run in parallel. This happens for phases that are waiting for a Koji task to finish.

Init

This is the first phase that is running in each compose, it prepares comp files for different variants. The comps file that Pungi takes as input is not really pure comps as used by tools like DNF. There are extensions used to customize how the file is processed.

The first step of Pungi processing is to retrieve the actual file. This can use anything that Exporting files from SCM supports.

Pungi extensions are arch attribute on packageref, group and environment tags. The value of this attribute is a comma separated list of architectures.

Second step Pungi performs is creating a file for each architecture. This is done by removing all elements with incompatible arch attribute. No additional clean up is performed on this file. The resulting file is only used internally for the rest of the compose process.

Third and final step is to create comps file for each Variant.Arch combination. This is the actual file that will be included in the compose. The start file is the original input file, from which all elements with incompatible architecture are removed. Then clean up is performed by removing all empty groups, removing non-existing groups from environments and categories and finally removing empty environments and categories. As a last step groups not listed in the variants file are removed.

Pkgset

Multiple places in Pungi can use files from external storage. The configuration is similar independently of the backend that is used, although some features may be different.

The so-called scm_dict is always put into configuration as a dictionary, which can contain following keys.

scm – indicates which SCM system is used. This is always required. Allowed values are:

file – copies files from local filesystem git – copies files from a Git repository cvs – copies files from a CVS repository rpm – copies files from a package in the compose koji – downloads archives from a given build in Koji build system repo

for Git and CVS backends this should be URL to the repository for RPM backend this should be a shell style glob matching package names (or a list of such globs) for file backend this should be empty for Koji backend this should be an NVR or package name branch

branch name for Git and CVS backends, with master and HEAD as defaults Koji tag for koji backend if only package name is given otherwise should not be specified file – a list of files that should be exported.

dir – a directory that should be exported. All its contents will be exported. This option is mutually exclusive with file.

command – defines a shell command to run after Git clone to generate the needed file (for example to run make). Only supported in Git backend.

options – a dictionary of additional configuration options. These are specific to different backends.

Currently supported values for Git:

credential_helper – path to a credential helper used to supply username/password for remotes that require authentication.

Buildinstall

Spawns a bunch of threads, each of which runs either lorax or buildinstall command (the latter coming from anaconda package). The commands create boot.iso and other boot configuration files. The image is finally linked into the compose/ directory as netinstall media.

The created images are also needed for creating live media or other images in later phases.

With lorax this phase runs one task per variant.arch combination. For buildinstall command there is only one task per architecture and product.img should be used to customize the results.

Gather

This phase uses data collected by pkgset phase and figures out what packages should be in each variant. The basic mapping can come from comps file, a JSON mapping or additional_packages config option. This inputs can then be enriched by adding all dependencies. See Gathering packages for details.

Once the mapping is finalized, the packages are linked to appropriate places and the rpms.json manifest is created.

ExtraFiles

This phase collects extra files from the configuration and copies them to the compose directory. The files are described by a JSON file in the compose subtree where the files are copied. This metadata is meant to be distributed with the data (on ISO images).

Createrepo

This phase creates RPM repositories for each variant.arch tree. It is actually reading the rpms.json manifest to figure out which packages should be included.

OSTree

Updates an ostree repository with a new commit with packages from the compose. The repository lives outside of the compose and is updated immediately. If the compose fails in a later stage, the commit will not be reverted.

Implementation wise, this phase runs rpm-ostree command in Koji runroot (to allow running on different arches).

Createiso

Generates ISO files and accumulates enough metadata to be able to create image.json manifest. The file is however not created in this phase, instead it is dumped in the pungi-koji script itself.

The files include a repository with all RPMs from the variant. There will be multiple images if the packages do not fit on a single image.

The image will be bootable if buildinstall phase is enabled and the packages fit on a single image.

There can also be images with source repositories. These are never bootable.

ExtraIsos

This phase is very similar to createiso, except it combines content from multiple variants onto a single image. Packages, repodata and extra files from each configured variant are put into a subdirectory. Additional extra files can be put into top level of the image. The image will be bootable if the main variant is bootable.

LiveImages, LiveMedia

Creates media in Koji with koji spin-livecd, koji spin-appliance or koji spin-livemedia command. When the media are finished, the images are copied into the compose/ directory and metadata for images is updated.

ImageBuild

This phase wraps up koji image-build. It also updates the metadata ultimately responsible for images.json manifest.

OSBuild

Similarly to image build, this phases creates a koji osbuild task. In the background it uses OSBuild Composer to create images.

OSBS

This phase builds container base images in OSBS.

The finished images are available in registry provided by OSBS, but not downloaded directly into the compose. The is metadata about the created image in compose/metadata/osbs.json.

ImageContainer

This phase builds a container image in OSBS, and stores the metadata in the same file as OSBS phase. The container produced here wraps a different image, created it ImageBuild or OSBuild phase. It can be useful to deliver a VM image to containerized environments.

OSTreeInstaller

Creates bootable media that carry an ostree repository as a payload. These images are created by running lorax with special templates. Again it runs in Koji runroot.

Repoclosure

Run repoclosure on each repository. By default errors are only reported in the log, the compose will still be considered a success. The actual error has to be looked up in the compose logs directory. Configuration allows customizing this.

ImageChecksum

Responsible for generating checksums for the images. The checksums are stored in image manifest as well as files on disk. The list of images to be processed is obtained from the image manifest. This way all images will get the same checksums irrespective of the phase that created them.

Test

This phase is supposed to run some sanity checks on the finished compose.

The only test is to check all images listed the metadata and verify that they look sane. For ISO files headers are checked to verify the format is correct, and for bootable media a check is run to verify they have properties that allow booting.

Fedora and Pungi

All composes are defined by configuration files kept in the pungi-fedora repository. Composes fall into two categories. They may be release candidates created on demand or nightly composes set to run at a scheduled time each day. Main branch is defining current Fedora Rawhide. There is also file with part of the pungi configuration file that is shared between different composes.

Once the release cycle steps into the phase of branching new release new branch for that release is created with all the configuration from rahide and beta and final configs are created/updated to reflect the current release cycle.

[Error - typeError]

Building server respin ISO

[Error - fatalError]

[Error - typeError]