Skip to content
Install

Troubleshooting Orchard

This page lists common failures on an Orchard instance by symptom, so you can match the error you see and go straight to the fix. Each entry gives the symptom, the cause, and the fix. If nothing here matches, collect logs and open an issue.

Image builds need a container registry, but DOCKER_REGISTRY_PUSH and DOCKER_REGISTRY_PULL are not set on the server. Point them at your registry and restart the server.

Cause: The instance has no registry configured. This occurs on a Helm install without registry values.

Fix: Set DOCKER_REGISTRY_PUSH and DOCKER_REGISTRY_PULL on the server to your own registry, and restart the server.

dial tcp 203.0.113.10:443: connect: connection refused

Cause: A network policy blocks the builder from reaching the registry. Builders can only reach DNS, public IP addresses on ports 80 and 443, and the destinations in builders.registryEgress.

Fix: Add the registry to builders.registryEgress, as described in Use your own registry.

orchard up fails at context-fetch with Connection refused

Section titled “orchard up fails at context-fetch with Connection refused”
wget: can't connect to remote host (203.0.113.10): Connection refused

Cause: Your S3 storage runs on the same server or cluster, behind its own ingress. A build fetches its files from that address, and a hostname that resolves to a node loops back to the ingress controller, which a network policy blocks. Backups still work because they do not run in the builder.

Fix: On an installer host, run the installer again. It adds the Traefik pods to builders.registryEgress, except in tunnel mode. With Helm, add the ingress controller’s pods to builders.registryEgress. Do not add the storage pods themselves, because a build should only reach storage the way the internet does.

Build fails with a missing module or command

Section titled “Build fails with a missing module or command”

Cause: The build targets the wrong stage. A multi-stage Dockerfile built at its first stage produces an image without the application.

Fix: Set the build stage to the final stage.

Cause: The builder has cache or disk pressure.

Fix: Check free disk on the build node and the size of the build cache volume.

A push fails quickly with an error that looks like a rate limit

Section titled “A push fails quickly with an error that looks like a rate limit”

Cause: The BuildKit daemon stopped, and the pusher reports the result.

Fix: Read the logs of the buildkit container instead of the pusher:

kubectl -n orchard-builders logs job/<BUILD_JOB_NAME> -c buildkit --tail=30
Build failed: could not schedule build pod after 300s - 0/1 nodes are available: 1 Insufficient memory.

Cause: No node has enough unreserved memory. The scheduler uses memory requests, not actual usage.

Fix: Reduce the memory that apps and databases request, or add a node.

Cause: The app exits soon after it starts.

Fix: Read the crash report on the app. It includes the logs of the container that stopped.

Cause: The container ran out of memory.

Fix: Raise the memory limit of the app, or fix the leak.

The app is live but its URL returns 502 Bad Gateway

Section titled “The app is live but its URL returns 502 Bad Gateway”

Cause: The pod runs, but nothing listens on the port Orchard routes to. Without a health check, a running container counts as ready. orchard up and other commands that wait for a deploy warn you with Nothing is listening on port.

Fix: Go to the app’s Network tab and set the port your process listens on. To catch this in a rollout, add a health check.

Cause: The health check uses the wrong path or port, or the app starts more slowly than the check allows.

Fix: Correct the health check, or give it a longer start time.

Cause: The request uses the public URL.

Fix: Use the internal cluster address shown under the port.

Cause: The tenant network policy blocks the CloudNativePG operator from reaching database pods. The server allows the namespace in CNPG_OPERATOR_NAMESPACE, which defaults to postgresql.

Fix: Set CNPG_OPERATOR_NAMESPACE in server.env to the namespace where the operator runs, such as cnpg-system. The installer sets this for you.

No backup storage. Set it up for this organization, or set a backup bucket on the instance.

Cause: The instance has no backup bucket. This occurs on a Helm install without object storage.

Fix: Refer to Configure database backups.

Cause: The connection uses the wrong host or endpoint.

Fix: Use the internal host and the -rw endpoint. The -ro endpoint only accepts reads.

Cause: Replicas are asynchronous.

Fix: Send reads that must see the latest writes to -rw.

Cause: No base backup exists. Archived write-ahead logs alone cannot restore a database.

Fix: Check how far back the database can be restored:

orchard db recovery <DATABASE_ID>

Cause: The container is out of memory. If the restarts line up with the backup window, the base backup uses memory from the same container.

Fix: Read the last termination reason. Raise the memory of the database. Do not set shared_buffers close to the memory limit. If you leave shared_buffers empty, Orchard sizes it.

Cause: The node’s kernel sets kernel.apparmor_restrict_unprivileged_unconfined=1, as Ubuntu 26.04 does. AppArmor then labels container processes cri-containerd.apparmor.d//&unconfined, and containerd’s default profile stops them signalling each other. Postgres logs could not signal for checkpoint: Permission denied, and the node’s kernel log shows apparmor="DENIED" operation="signal".

Fix: The installer and orchardctl update load a containerd profile that allows the stacked label. On a cluster you run yourself, load the same profile on every node before containerd starts. Copy it from containerd_apparmor_profile in install/install.sh to /etc/apparmor.d/cri-containerd.apparmor.d, then run apparmor_parser -r /etc/apparmor.d/cri-containerd.apparmor.d. The profile replaces containerd’s default, so running pods pick it up without a restart.

The database page shows one replica and no pods

Section titled “The database page shows one replica and no pods”

Cause: The instance is not running. If the page lists the storage volume as dangling, Kubernetes does not recreate the pod.

Fix: An operator must inspect the cluster:

kubectl -n <NAMESPACE> get pods,pvc

Cause: One of the following:

Fix: Correct the cause, then check the challenge:

kubectl get challenges -A

Cause: The app is protected and the webhook path is not a public path.

Fix: Add the path to the public paths list of the protected app.

Cause: The instance has no email configured, so Orchard hides email codes.

Fix: On the host, run sudo orchardctl login or sudo orchardctl password set. To add email, refer to Set up email.

On a Helm install, print a one-time sign-in link with the following command. To sign in as someone other than the first superadmin, add --email <ADDRESS>.

kubectl -n orchard exec deploy/server -- node dist/scripts/mint-login-link.js

The installer stops on an IPv6-only server

Section titled “The installer stops on an IPv6-only server”
IPv6-only servers aren't supported yet. This server has no IPv4 address.

Cause: The server has no IPv4 address. GitHub and ghcr.io have no IPv6 addresses, so the server cannot download k3s or the Orchard images, and Orchard cannot clone or build from GitHub. k3s runs with IPv4 inside the cluster, and Orchard listens on IPv4 only, so traffic to the IPv6 address does not reach it.

Fix: Use a server with an IPv4 address. A server with both IPv4 and IPv6 addresses works.

To try anyway, add --allow-ipv6-only when you run the installer. --yes does not imply this flag. NAT64 in front of an IPv4 server is unsupported, and Orchard has not tested it.

Cause: A step failed. The installer prints the step and the error.

Fix: Correct the problem, then run the installer again. It reads your current settings and continues from them. Do not delete /etc/orchard/values.yaml to get past a failure, because a new file means new secrets, and Orchard cannot read data encrypted with the old ones.

A tunnel install fails with a handshake error

Section titled “A tunnel install fails with a handshake error”
curl: (35) sslv3 alert handshake failure

Cause: The name is more than one level under your Cloudflare zone, such as orchard.dev.example.com. Cloudflare’s free certificate covers example.com and *.example.com only, so the connection fails at Cloudflare. The tunnel is healthy, and plain HTTP still works.

Fix: Use names directly under the zone, or add Advanced Certificate Manager in Cloudflare. To change a name, re-run the installer.

The dashboard returns a 404 at a name that resolves to the host

Section titled “The dashboard returns a 404 at a name that resolves to the host”

Cause: Traefik routes by Host header and does not know the name.

Fix: List the names, then add the missing one:

sudo orchardctl hostname
sudo orchardctl hostname add mango --apply

A LAN install does not respond at its sslip.io name

Section titled “A LAN install does not respond at its sslip.io name”

Cause: sslip.io is public DNS that maps a name to the address inside it. Some resolvers, such as the Tailscale resolver, do not return private addresses.

Fix: Use one of the machine’s addresses, or add a name that you control:

sudo orchardctl hostname add mango --apply
sudo orchardctl claim --url mango

Cause: The cluster rejects the write, and the apply queue retries it.

Fix: Read the events in the tenant namespace:

kubectl -n <TENANT_NAMESPACE> get events --sort-by=.lastTimestamp

An upgrade says a Certificate cannot be imported

Section titled “An upgrade says a Certificate cannot be imported”
Error: UPGRADE FAILED: Unable to continue with update: Certificate "orchard-tls" in namespace "orchard" exists and cannot be imported into the current release: invalid ownership metadata

Cause: You created the object by hand, and the chart now manages it.

Fix: Give the object to Helm, then run the upgrade again. This keeps the certificate that is already issued.

kubectl -n orchard label certificate orchard-tls app.kubernetes.io/managed-by=Helm
kubectl -n orchard annotate certificate orchard-tls \
meta.helm.sh/release-name=orchard \
meta.helm.sh/release-namespace=orchard

An upgrade says another operation is in progress

Section titled “An upgrade says another operation is in progress”
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Cause: An earlier Helm operation was interrupted, and the release is stuck in a pending state.

Fix: Find the last revision with the status deployed, and roll back to it. Then upgrade again.

sudo helm -n orchard history orchard
sudo helm -n orchard rollback orchard <REVISION>
sudo orchardctl update

If the stuck revision is the first install, there is nothing to roll back to. Uninstall the release, and install it again.

Collect the following before you ask for help:

kubectl -n orchard logs deploy/server --tail=200
kubectl -n <TENANT_NAMESPACE> get events --sort-by=.lastTimestamp

On an installer host, also include the output of orchardctl status. To report a problem, open an issue in the Orchard repository.

  • Troubleshooting Greenhouse - Fix Greenhouse machines, sessions, and pull requests by symptom.
  • Troubleshooting grafts - Fix failed downloads, signature errors, version conflicts, and failed migrations, and recover with safe mode.
  • Host commands - Review the orchardctl commands that this page uses on an installer host.