Upgrade Orchard
To upgrade Orchard, use the command that matches how you installed it. An upgrade rolls out new control-plane pods and runs database migrations. It does not restart your apps.
Before a major upgrade, take a backup of the control-plane database. This database holds every organization, project, and app definition on the instance. Its continuous backups are off by default, so check that you turned them on.
Upgrade an installer host
Section titled “Upgrade an installer host”Run the following command on the host:
sudo orchardctl updateorchard-update is the same command. It upgrades the chart to the newest release with /etc/orchard/values.yaml, restarts deployments that use a moving image tag, and refreshes the host commands. For the full sequence, refer to orchardctl update.
If you chose automatic updates during installation (--updates automatic), a nightly systemd timer runs the same command. To stop it, run sudo systemctl disable --now orchard-update.timer.
Pin a version on an installer host
Section titled “Pin a version on an installer host”An installer host runs the server, frontend, and MCP images of the chart it installed. A pod restart does not change their version. orchardctl update installs the newest chart, which moves them together. To stay on one release until you choose to move:
-
Turn off nightly updates, because the timer always installs the newest chart:
sudo systemctl disable --now orchard-update.timer -
Run the installer with
--chart-version. On an existing host, the re-run keeps your answers and settings. If the values file setslatestimage tags, the re-run removes them, so the server and frontend run the images of that chart version.sudo ./install.sh --chart-version <VERSION> -
To move to another release later, pass the version to
orchardctl update. Any extra arguments go tohelm upgrade.sudo orchardctl update --version <VERSION>Without
--version,orchardctl updateinstalls the newest chart.
orchardctl update also applies k3s patches and newer third-party images, independent of the Orchard version. To hold those too, add these lines to /etc/orchard/env:
ORCHARD_K3S_UPDATES=offORCHARD_ADDON_UPDATES=offIf you turn them off, apply their security fixes yourself. For what each one covers, refer to Security updates.
Find release notes
Section titled “Find release notes”Release notes are on GitHub Releases. Each service has its own release, tagged <service>/v<version>, such as server/v2.95.4 or frontend/v1.87.0. Each release lists its changes, with a link to each commit.
The chart has no release page of its own. Its version is the highest version among the services it ships, so the release with that number is the newest change in that chart. To see the chart version you run, use the following command:
helm list -n orchardAfter a dashboard release, the dashboard shows its release notes to each person once. The notes come from assets.orchard.my. To turn them off, set UPDATES_URL=off.
Upgrade k3s
Section titled “Upgrade k3s”orchardctl update also applies the latest k3s patch, from the k3s release channels. To see your version, or to move to a newer minor version, use the following commands:
sudo orchardctl k3ssudo orchardctl k3s upgradeFor what each step does, refer to orchardctl k3s.
Security updates
Section titled “Security updates”Most parts of an instance get security fixes without an Orchard release:
| Part | How it gets security fixes |
|---|---|
| k3s, Kubernetes, and containerd | orchardctl update applies patches from the k3s release channels. orchardctl k3s upgrade moves to a new minor version. |
| Postgres minor versions | Orchard applies them from the CloudNativePG image catalog. |
| The host operating system | Your distribution’s updates. Orchard does not change them. |
| Traefik, CloudNativePG, and cert-manager | orchardctl update upgrades them from their own charts. Refer to Third-party update policy. |
| Redis, pgbouncer, BuildKit, and the metrics and log stores | orchardctl update installs newer versions from their own registries. Refer to Third-party update policy. To see them, run orchardctl images. |
| Railpack | A new Orchard chart, because a version mismatch breaks builds. orchardctl images tells you when one is out. |
| Orchard | A new Orchard chart. orchardctl update installs it. |
Security fixes in third-party components do not wait for an Orchard release. A new major version of a third-party component needs a person, because it can change behavior. orchardctl update tells you when one is out. To turn off these updates, add ORCHARD_ADDON_UPDATES=off to /etc/orchard/env.
Upgrading CloudNativePG does not restart your databases, because the installer turns on in-place updates of the instance manager.
An upgrade turns on, once, features that newer releases run by default. Orchard prints each change. If you turn a feature off afterward, it stays off. For the list, refer to New defaults on an existing host.
orchardctl update also sets kernel.apparmor_restrict_unprivileged_userns to 0 where it is 1, such as on Ubuntu 24.04, because rootless builds need user namespaces. It skips this when builderSandbox.enabled or builders.allowPrivilegedOnHost is true.
Third-party update policy
Section titled “Third-party update policy”orchardctl update installs new versions of Traefik, CloudNativePG, cert-manager, and the third-party images from their own sources. It does not wait for an Orchard release. Each new version goes through the following rules:
- Version class: A new major version never installs on its own.
orchardctl updateprints a note, so that you can read its upgrade notes first. Below version 1.0, a new minor version counts as a new major version. For example,0.61.0to0.61.3installs, and0.61.0to0.62.0needs a person. Redis, pgbouncer, and BuildKit take patch versions only. cloudflared uses calendar versions and takes any newer version. - Soak period: A version installs only after it has been public for 7 days. If a release turns out to be broken or compromised during that time, and its publisher replaces it, your instance never installs it. A version without a publication date waits.
- Security fast track: If OSV.dev lists a vulnerability in the running version, the update installs a fixed version at once, without the soak period. It picks the version that fixes the most of those advisories and adds no new advisory. If more than one version qualifies, it prefers the newest one that has soaked, then the oldest one that has not. The command prints
security update: <advisory> affects <running version>, moving to <new version>.
The fast track follows the version class rules, with one exception: a standalone tool below version 1.0 can move to a new minor version to fix a vulnerability. Refer to Security fixes in a new minor version. If no version that the rules allow fixes a vulnerability, orchardctl update names the version that fixes it and leaves the upgrade to you.
If OSV.dev does not answer, every version waits for the soak period. Missing advisory data never blocks an update, and never skips the soak period. When OSV.dev answers, the update also skips a version that has an advisory that the running version does not have.
orchardctl update respects versions in /etc/orchard/images-held, including on the fast track. It sends OSV.dev package names and version numbers only, and nothing about your instance.
Security fixes in a new minor version
Section titled “Security fixes in a new minor version”Below version 1.0, a project often fixes a vulnerability only in a new minor version. A standalone tool keeps no data that it cannot rebuild, and no other component depends on its stored format. If no release in its running minor version fixes a vulnerability, orchardctl update moves a standalone tool to the nearest minor version that does. The new version fixes the advisories, adds no advisory, and follows the fast track preference for a version that has soaked. The move never crosses a major version, and it never happens without an advisory. A new minor version without a security fix still needs a person.
Operators and components that hold data stay in their minor version, because a new minor version can change their CRDs, their APIs, or the data that they store. For those components, orchardctl update prints the version that fixes the vulnerability, and you upgrade.
| Component | Crosses a minor version for a security fix | Reason |
|---|---|---|
| Trivy | Yes | Scans an image and exits. It downloads the vulnerability database that its version reads. |
| Vector | Yes | Ships logs and keeps only its read positions. A version that rejects its configuration does not come up, and the update goes back. |
| skopeo and aws-cli | Yes | Run one task and exit. |
| vmagent, kube-state-metrics, node-exporter | Yes | Collect and serve metrics, and keep nothing that they cannot rebuild. |
| cloudflared | Yes | Forwards traffic and keeps nothing. It takes any newer version in any case. |
| Traefik, CloudNativePG, cert-manager | No | Operators. A new minor version can change their CRDs. Traefik routes every request, CloudNativePG runs every database, and cert-manager issues every certificate. |
| Redis, pgbouncer, BuildKit | No | Take patch versions only. Redis holds data, pgbouncer sits in front of every database connection, and a new BuildKit minor version can change the build cache. |
| VictoriaMetrics, VictoriaLogs, versitygw, zot | No | Hold data: metrics, logs, objects, and the images that Orchard builds. |
Trivy and Vector are the standalone tools below version 1.0, so the exception applies to them. The CloudNativePG chart is below version 1.0, so a CloudNativePG fix that is only in a new chart minor version needs a person.
Settings
Section titled “Settings”Set these in /etc/orchard/env:
| Variable | Default | Effect |
|---|---|---|
ORCHARD_UPDATE_SOAK_DAYS |
7 |
The number of days that a version must be public before it installs. 0 turns off the wait. |
ORCHARD_ADDON_UPDATES |
on |
off turns off add-on and image updates, including security fixes. |
Publication dates
Section titled “Publication dates”| Source | Publication date |
|---|---|
| Traefik, CloudNativePG, and cert-manager charts | The created field of the version in the chart repository’s index.yaml. |
| Images on Docker Hub | The time that Docker Hub recorded for the last push of the tag. |
Images on quay.io |
The time that quay.io recorded for the tag. |
Images on other registries, such as ghcr.io and registry.k8s.io |
The created time in the image configuration. |
A new push to an existing Docker Hub tag starts its soak period again. The publisher’s build sets the created time in an image configuration and in a chart index, so treat those dates as the publisher’s claim.
Advisory sources
Section titled “Advisory sources”orchardctl update checks each component against a package that OSV.dev indexes. For a chart, it checks the version of the application that the chart installs.
| Component | OSV.dev package |
|---|---|
| Traefik | Go module github.com/traefik/traefik/v3, for the running major version |
| CloudNativePG | Go module github.com/cloudnative-pg/cloudnative-pg |
| cert-manager | Go module github.com/cert-manager/cert-manager |
| Redis | Git repository https://github.com/redis/redis |
| BuildKit | Go module github.com/moby/buildkit |
| skopeo | Go module github.com/containers/skopeo |
| VictoriaMetrics and vmagent | Go module github.com/VictoriaMetrics/VictoriaMetrics |
| kube-state-metrics | Go module k8s.io/kube-state-metrics/v2 |
| node-exporter | Go module github.com/prometheus/node_exporter |
| Vector | Git repository https://github.com/vectordotdev/vector |
| Trivy | Go module github.com/aquasecurity/trivy |
| versitygw | Go module github.com/versity/versitygw |
| aws-cli | Git repository https://github.com/aws/aws-cli |
| zot | Go module zotregistry.dev/zot/v2, for the running major version |
| cloudflared | Git repository https://github.com/cloudflare/cloudflared |
pgbouncer and VictoriaLogs have no OSV.dev package that matches their release versions, so only the soak period applies to them. OSV.dev lists Vector advisories without the affected release versions, so a Vector advisory does not start the fast track.
Re-run the installer
Section titled “Re-run the installer”You can also re-run the installer on an existing host. A re-run:
- Keeps your secrets, chart values, environment variables, and volume sizes.
- Changes a value only when you change an answer that it depends on, and prints each change.
- Saves a copy of the values file before it changes the file.
- Does not restart k3s unless the registry mirror configuration changed.
- Upgrades CloudNativePG to the chart version that the installer pins, one minor version at a time, and installs the Barman Cloud plugin. It never moves either chart to an older version.
- Keeps the backup method of your databases. To move them to the Barman Cloud plugin, refer to Move database backups to the Barman Cloud plugin.
Without --chart-version, a re-run installs the newest chart. For more information, refer to Run the installer again.
Upgrade a Helm install
Section titled “Upgrade a Helm install”Run the following command:
helm upgrade orchard oci://ghcr.io/hackclub/orchard/charts/orchard \ --version <VERSION> --namespace orchard -f my-values.yaml --server-side=falseOn Helm 3, leave out --server-side=false, because Helm 3 does not have this flag. For the reason Helm 4 needs it, refer to Upgrade the chart.
Pass --version on every upgrade. Without it, Helm installs the newest chart. A published chart sets the server, frontend, MCP, and backup proxy images to the versions in that release, so the chart version pins them too.
Database migrations
Section titled “Database migrations”Schema migrations run when the server starts. Upgrade forward. An older server with a newer schema is not supported.
Effect on running apps
Section titled “Effect on running apps”Upgrading the control plane does not restart tenant apps. The server, frontend, and MCP pods roll out, and apps keep serving during the rollout.
Roll back an upgrade
Section titled “Roll back an upgrade”You can roll back the release with helm rollback. If the new version migrated the schema, the old version may not run against it. If downtime matters, test upgrades on a non-production instance first.
Upgrade from an earlier version
Section titled “Upgrade from an earlier version”Move database backups to the Barman Cloud plugin
Section titled “Move database backups to the Barman Cloud plugin”CloudNativePG 1.31 removes its built-in Barman Cloud support (spec.backup.barmanObjectStore). A database that still archives that way stops taking backups after the operator moves to 1.31. The replacement is the Barman Cloud plugin, which needs CloudNativePG 1.30 or later.
An installer host set up with this version of the installer already uses the plugin, and the installer prints Database backups: Barman Cloud plugin in its summary. An older installer host, or a Helm install, uses the built-in support until you move it. Finish these steps before CloudNativePG 1.31 reaches the cluster.
-
Check the CloudNativePG chart version. Chart
0.29.xruns CloudNativePG 1.30.helm list -n cnpg-system -
Upgrade CloudNativePG to 1.30 and install the plugin.
-
On an installer host, run the installer again. The re-run upgrades CloudNativePG one minor version at a time, installs the plugin with its certificates, and keeps your backup method. Its summary shows how many databases still use the built-in support.
-
On a Helm install, upgrade the
cloudnative-pgchart one minor version at a time withconfig.data.ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES=true, so that the databases do not restart. By default, the plugin chart issues its certificates with cert-manager, so cert-manager must run in the cluster. Then install the plugin into the operator namespace:helm upgrade --install plugin-barman-cloud cnpg/plugin-barman-cloud \--namespace cnpg-system --version <PLUGIN_CHART_VERSION> --wait
-
-
Turn on the plugin for new databases. On an installer host, run the following command. On a Helm install, add
CNPG_BARMAN_PLUGIN=ontoserver.envand upgrade the chart.sudo orchardctl env set CNPG_BARMAN_PLUGIN on --apply -
In the dashboard, go to Instance admin > Configuration > Databases, and turn on Move backups to the Barman Cloud plugin.
The migration moves a few databases every 10 minutes. Each database keeps its archive path and server name, so its earlier backups stay restorable. The plugin runs a sidecar of up to 256 MiB of memory in each database instance, so the migration skips a namespace whose quota has no room for it. If a moved database stops archiving, the migration stops itself, and the setting offers to resume it. The Databases setting shows how many databases are left. Upgrade CloudNativePG to 1.31 only after none are left.
Turn off the Postgres superuser on older databases
Section titled “Turn off the Postgres superuser on older databases”Orchard creates dedicated databases with the CloudNativePG postgres superuser login turned off (enableSuperuserAccess: false). Orchard never uses that login. Databases created by earlier versions keep it until you run the script in apps/server/scripts/cnpg-disable-superuser.sh from a checkout of the repository, with kubectl access to the cluster.
The script patches one cluster at a time, and stops if any instance pod restarts:
apps/server/scripts/cnpg-disable-superuser.sh # dry run, lists clustersapps/server/scripts/cnpg-disable-superuser.sh --apply --limit 1 # one cluster firstapps/server/scripts/cnpg-disable-superuser.sh --applyRefresh BuildKit daemon policies
Section titled “Refresh BuildKit daemon policies”Persistent BuildKit daemons do not hold the registry login, and the chart’s buildkit-daemon-network-policy limits their egress. If your instance ran persistent daemons before the chart included that policy, upgrade the chart so that the policy exists, and check that builders.registryEgress lists your registry.
Related resources
Section titled “Related resources”- Back up and restore an instance - Learn what to back up before an upgrade and how to bring back the control plane.
- Host commands - Review every
orchardctlcommand, includingupdateandk3s. - Upgrade and roll back grafts - Upgrade extensions separately from Orchard.