Fedora CoreOS Stable Release Guide

Hey folks! Ready to dive into the world of Fedora CoreOS? This guide is your ultimate companion for navigating the release process, ensuring a smooth and successful update. Whether you're a seasoned pro or just getting started, we'll break down each step, making it easy to understand and execute. Let's get this show on the road! This article provides a comprehensive guide to the release process for Fedora CoreOS's stable stream. From prerequisites to post-release checks, it covers all the necessary steps to ensure a smooth and successful update. It's designed to be a useful resource for anyone involved in the Fedora CoreOS release process, providing clarity and guidance throughout. The guide aims to provide a clear and concise roadmap for the release, ensuring a systematic approach to each stage.

Prerequisites: Setting the Stage for Success

Before we jump into the nitty-gritty, let's make sure we've covered the basics. Check out the prerequisites to ensure you're all set. Think of this as your pre-flight checklist. Making sure you've got all the tools and access you need will save you headaches later. This is where you'll find the essential requirements, ensuring a smooth and efficient release process. It's like preparing your workspace before starting a big project. Make sure you have the necessary tools, permissions, and access to all the systems involved.

Issue Title Update

Update the issue title with today's date. Once the pipeline spits out the new version ID, append it to the title e.g. (31.20191117.3.0). This helps keep things organized and makes it easier to track the release. By updating the issue title, you keep all stakeholders informed about the current release status.

Pre-release: Promoting Changes to Stable

This is where we prep the changes for the stable branch. It's a critical step that ensures all the latest improvements and fixes are included. Here's what you need to do:

• Add the ok-to-promote label to the issue. This signals that the changes are ready for promotion. This is a crucial step to let the system know that the changes are ready to be moved to the stable branch.
• Review the promotion PR. Check the promotion PR against the stable branch on https://github.com/coreos/fedora-coreos-config. Ensure everything looks good before merging.
• Merge the promotion PR. Once CI has passed, merge it. This integrates the changes into the stable branch. This is the moment when the changes are incorporated into the stable branch. Make sure the CI has passed and all checks are green before merging to avoid any potential issues.

Manual Alternative

Sometimes, you might need to manually run the process. This can be the case if you need to add an extra commit or change something in manifest.yaml. Here's how:

• git fetch upstream
• git checkout stable
• git reset --hard upstream/stable
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing
• Open PR against the stable branch on https://github.com/coreos/fedora-coreos-config

Build: Crafting the New Release

Time to get those builds rolling! This is where we create the actual release artifacts. This step is about generating the builds for different architectures. Here's how:

• Start a build job. Start a build job (select stable and enable EARLY_ARCH_JOBS, leave all other defaults). This will automatically run multi-arch builds.
• Post links to the jobs. Post links to the jobs as a comment to this issue:
  • x86_64
  • aarch64 (multi-arch build job)
  • ppc64le (multi-arch build job)
  • s390x (multi-arch build job)
• Wait for the jobs to finish. Wait for the jobs to finish and succeed:
  • x86_64
  • aarch64
  • ppc64le
  • s390x
• Edit the issue title. Edit the issue title to append the new version ID e.g. (31.20191117.3.0). This helps in keeping the tracking and identification of the builds.

This build phase is crucial; it's where the new versions of Fedora CoreOS are created for all supported architectures. The build jobs generate the necessary artifacts that will be used for the update. Ensuring the jobs succeed is paramount for a successful release.

Sanity-Check the Build: Ensuring Quality

Before we unleash the release, let's make sure it's up to snuff. This involves verifying the build and running several tests to ensure everything is working as expected. Let's make sure things are working right before we go live! This is where we verify the integrity and functionality of the build. Use the the build browser for the stable stream:

• Verify parent commit and version. Verify that the parent commit and version match the previous stable release (in the future, we'll want to integrate this check in the release job):
  • x86_64
  • aarch64
  • ppc64le
  • s390x
• Check kola extended upgrade runs. Check kola extended upgrade runs to make sure they didn't fail:
  • x86_64
  • aarch64
  • ppc64le
  • s390x
• Check kola AWS runs. Check kola AWS runs to make sure they didn't fail:
  • x86_64
  • aarch64
• Check kola OpenStack runs. Check kola OpenStack runs to make sure they didn't fail:
  • x86_64
  • aarch64
• Check kola Azure runs. Check kola Azure runs to make sure they didn't fail:
  • x86_64
  • aarch64
• Check kola GCP runs. Check kola GCP runs to make sure they didn't fail:
  • x86_64
  • aarch64

This is the sanity check stage. This thorough inspection helps catch potential problems early, ensuring the new release is stable and reliable. By verifying the parent commit, running upgrade tests, and checking cloud-specific runs, we minimize the risk of issues during the rollout.

⚠️ Release: The Point of No Return ⚠️

IMPORTANT: This is the critical moment. Once the OSTree commit is imported into the unified repo, any machine that manually runs rpm-ostree upgrade will get the update. This step involves running the release job and refreshing metadata.

Run the Release Job

• Run the release job. Run the release job, filling in for parameters stable and the new version ID.
• Post a link to the job. Post a link to the job as a comment to this issue.
• Wait for job to finish. Wait for the job to finish.

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh Metadata (Stream and Updates)

• Wait for all releases. Wait for all releases that will be released simultaneously to reach this step in the process.
• Run the workflow. Go to the rollout workflow, click "Run workflow", and fill out the form.

Rollout General Guidelines

When setting a rollout start time ask "when would be the best time to react to any errors or regressions from updates?". Commonly we select 2PM UTC so that the rollout's start at 10 am EST(±1 for daylight savings), but these can be fluid and adjust after talking with the fedora-coreos IRC. Note, this is impacted by the day of the week and holidays. The later in the week the release gets held up due to unforeseen issues the more likely the rollout time allocation will need to shrink or the release will need to be deferred.

Manual Alternative

• Make sure fedora-coreos-stream-generator is up-to-date. Make sure your fedora-coreos-stream-generator binary is up-to-date.

From a checkout of this repo:

• Update stream metadata. Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/stable/releases.json  -output-file=streams/stable.json -pretty-print

• Add a rollout. Add a rollout. For example, for a 48-hour rollout starting at 10 AM ET the same day, run:

./rollout.py add stable <version> "10 am ET today" 48

• Commit the changes and open a PR. Commit the changes and open a PR against the repo.

• Verify the PR. Verify that the expected OS versions appear in the PR on https://github.com/coreos/fedora-coreos-streams.

• Post a link to the PR. Post a link to the resulting PR as a comment to this issue.

• Review and approve the PR. Review and approve the PR, then wait for someone else to approve it also.

• Merge the PR. Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3.

• Verify the new version. Verify the new version shows up on the download page.

• Verify the update graph. Verify the incoming edges are showing up in the update graph:

  • x86_64
  • aarch64
  • ppc64le
  • s390x

Update Graph Manual Check

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=aarch64&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=ppc64le&stream=stable&rollout_wariness=0'
curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=s390x&stream=stable&rollout_wariness=0'

This is where the magic happens! Running the release job and refreshing the metadata makes the new version available to users. The rollout workflow determines how and when the update is pushed out. Following the guidelines ensures a controlled and safe release.

Housekeeping: Wrapping Up

Alright, almost there! Let's take care of a few final details. These are the finishing touches to ensure everything is set up for the next release.

• Open an issue for the next release. If one doesn't already exist, open an issue in this repo for the next release in this stream. Use the approximate date of the release in the title.
• Assign the issue and card. Issues opened via the previous link will automatically create a linked Jira card. Assign the GitHub issue and Jira card to the next person in the rotation.

And that's a wrap! Following these housekeeping steps ensures the smooth continuation of the Fedora CoreOS release process. It's about keeping things organized and preparing for the next update. Great job, everyone! By following these steps, you've successfully navigated the Fedora CoreOS stable release process. Congrats! You've successfully released a new version of Fedora CoreOS! Now you're ready to tackle the next one. Keep up the great work! If you have any questions or run into any snags along the way, don't hesitate to reach out. Thanks for being a part of the Fedora CoreOS community! Now go forth and spread the word about the latest and greatest version of Fedora CoreOS. Cheers!