Rolling Updates for Cloud Mac CI Runners with Safe Rollback

Remote teams often treat CI runners on cloud Macs as always-on infrastructure, but the runners themselves still need to be upgraded. The riskiest approach is to overwrite the application directory and restart while a job is still running: build processes may continue using old files while the scheduler has already loaded new components, leaving behind a mixed state that is difficult to reproduce. A safer approach is to drain jobs first, then switch releases using immutable version directories and an atomic symbolic-link update.

Define the Safe Update Boundary First

A controlled update must satisfy at least four conditions: no new jobs are accepted, the current job finishes naturally, the old and new versions can coexist, and the previous version can be restored immediately if the switch fails. “Draining” does not mean clearing the workspace or terminating a build. It means pausing the runner at a job boundary.

During an update, the most important state is not whether the service is running, but whether the old job has finished and no new job has started.

Before starting, record the current version, the runner process ID, the identifier of the job in progress, and its working directory. If the job includes an Xcode build, also check for xcodebuild, test processes, or archive-export subprocesses. Do not rely solely on whether the top-level runner appears idle. Even after it exits, a faulty script may leave child processes running in the background.

Define explicit upgrade criteria in advance: the maximum time the current job may run, how long to wait before escalating to manual inspection, and which smoke tests must pass before job intake resumes. The team should set these rules based on actual project duration rather than improvising after a failure occurs.

Separate Binaries, Configuration, and Workspaces

Runner binaries should be stored in separate versioned directories, while persistent configuration and build workspaces should remain outside those directories. A simple layout looks like this:

/opt/ci-runner/
├── current -> releases/runner-current
├── previous-target
├── drain
├── active.pid
├── config/
├── work/
└── releases/
    ├── runner-old/
    └── runner-current/

config stores registration data and private configuration, with access restricted to the runner user. work contains checkouts and temporary build data. releases contains only the runner binaries. Do not copy tokens into every version directory, and do not allow an update package to overwrite the workspace.

With this separation in place, an upgrade changes only the target of current. The previous version remains intact, so rollback does not require another download or installation. Old releases should be removed only after the observation period ends, and at least the most recent verified version should always be retained.

Drain at Job Boundaries with a Wrapper

The easiest model to control is one in which the runner executes a single job, exits, and then lets a wrapper decide whether to accept another. Because the option for single-job execution varies between runners, first encapsulate the actual launch command in $RUNNER_CMD, then use a common loop:

#!/bin/zsh
set -u

root=/opt/ci-runner
lock="$root/wrapper.lock"

mkdir "$lock" 2>/dev/null || exit 1

cleanup() {
  rm -f "$root/active.pid"
  rmdir "$lock" 2>/dev/null
}

trap cleanup EXIT INT TERM

while [[ ! -e "$root/drain" ]]; do
  "$root/current/bin/$RUNNER_CMD" run --once &
  child=$!
  print -r -- "$child" > "$root/active.pid"

  if wait "$child"; then
    status=0
  else
    status=$?
  fi

  rm -f "$root/active.pid"

  if (( status != 0 )); then
    sleep 10
  fi
done

When preparing an update, run touch /opt/ci-runner/drain. The wrapper allows the current child process to finish but does not start another iteration. Then read active.pid and use kill -0 PID to determine whether the process still exists. An empty PID file does not prove that the machine is idle; also check for build and test subprocesses associated with the job.

If the runner does not support exiting after a single job, use its native mechanism for pausing job intake and verify that it does not prefetch work while paused. If that behavior cannot be confirmed, simulate a long-running job in an isolated environment and observe the draining process first.

Verify the Update Package and Switch Atomically

Extract the new version into a temporary directory. After verification, rename it into releases. The checksum for the downloaded package must come from a trusted team release record; it must not be generated ad hoc next to the same unverified archive.

root=/opt/ci-runner
release="$root/releases/runner-next"
archive=/tmp/runner-next.tar.gz
expected="$EXPECTED_SHA256"

actual="$(shasum -a 256 "$archive" | awk '{print $1}')"
[[ "$actual" = "$expected" ]] || exit 2

mkdir -p "$release"
tar -xzf "$archive" -C "$release"
"$release/bin/$RUNNER_CMD" --version || exit 3

old="$(readlink "$root/current")"
print -r -- "$old" > "$root/previous-target"

ln -sfn "$release" "$root/current.next"
mv -fh "$root/current.next" "$root/current"

Before switching the symbolic link, confirm that the binaries in the new directory are executable, built for the correct architecture, and able to read the external configuration. Keep the switch itself brief. Do not include extraction, dependency installation, or network downloads in the critical section. If the update also requires configuration migration, copy the configuration to a temporary file first, validate its format, and then replace it atomically.

Prevent Permission Drift During Extraction

Archives may carry different ownership or permission settings. Before switching, verify that users other than the runner user cannot modify the program files. Sensitive configuration should generally use 600, while the configuration directory should generally use 700. Also confirm that the workspace is still owned by the original runner user so that write failures are not discovered only when the new version runs for the first time.

Smoke Testing and Fast Rollback

Do not remove drain immediately after the switch. Manually launch one controlled smoke-test job and verify at minimum that the runner can read its configuration, create a temporary workspace, invoke development tools, complete a minimal build, save artifacts, and clean up and exit correctly.

Pay close attention to paths in the logs. The new process should start from the new directory referenced by current, while caches, configuration, and workspaces should remain at their fixed paths. If the logs mix paths from the old and new versions, an old process is still running and job intake must not resume.

If validation fails, read the old link target from previous-target, switch back using the same temporary-link method, and rerun the minimal job. Do not modify and retest files in place inside the new release directory, because that violates the immutability of the release. After confirming that the old version is restored, remove the failed version and rebuild the release package.

Once validation passes, run rm -f /opt/ci-runner/drain and let the process manager restart the wrapper. Finally, observe at least one real job from acceptance through compilation, artifact archiving, and process exit. Record the version number, switch result, and rollback target in the update log. Only then is the update complete; a runner showing as online is not sufficient reason to stop early.

Frequently asked questions