Check isolation with orchard doctor
Orchard keeps tenants apart only when your cluster is configured correctly, and many of the settings that matter live outside Orchard, in your CNI, runtime classes, and Helm values. The doctor reads your instance’s own configuration and cluster state, and reports every setting that weakens isolation, so you can find a gap before a tenant does. An optional self-test then proves enforcement by sending real traffic from a temporary tenant. Only superadmins can run it.
Each result has an id, a status, the evidence Orchard found, a fix, and a link to the matching section on this page.
| Status | Meaning |
|---|---|
pass |
The check found what it expects. |
warn |
Isolation holds, but a setting weakens it or leaves a decision to you. |
fail |
Isolation does not hold, or a configured resource is missing. |
unknown |
Orchard could not tell. Usually a missing permission, a managed cluster feature it cannot see, or a failed positive control. Treat it as not verified. |
Run it
Section titled “Run it”From the command line:
orchard doctororchard doctor --selftestorchard doctor prints a table and exits with code 11 when any check fails. --selftest also runs the isolation self-test after asking for confirmation (pass --yes in scripts). With -o json it prints the full report.
Over the API, GET /admin/doctor returns the report, POST /admin/doctor/selftest starts a self-test and returns its id, and GET /admin/doctor/selftest/{id} returns its progress and result.
The server needs read access to a few cluster objects. The chart grants it. If you manage the server’s RBAC yourself, allow get and list on daemonsets, runtimeclasses, priorityclasses and clusterissuers.
Checks
Section titled “Checks”Each section names the setting and the exact fix. For why the setting matters, refer to What you must do.
cni-enforcement
Section titled “cni-enforcement”A NetworkPolicy only works when the CNI enforces it. The doctor has two sources for this check: the name of the CNI and the result of the last isolation self-test. The self-test result wins, because it shows what the cluster does, and the name match is only a hint.
The name match looks for Cilium, Calico, Antrea, kube-router, Weave Net, Canal and Azure Network Policy Manager by their custom resources and daemonsets, and checks that the agent runs on every node. A CNI that is not on that list, or a managed CNI that does not run as a daemonset the server can see, such as GKE Dataplane V2, can enforce policy without matching. For the CNIs that Orchard supports, refer to Compatibility.
When the server has a self-test result, it reads the selftest-cross-tenant check from that run, provided the temporary namespaces were created correctly (selftest-namespace-shape passed):
pass: a pod in one temporary tenant namespace could not connect to a pod in the other, and the positive controls connected. The check passes whatever the CNI is called, unless the name match found a problem with a detected agent (thewarncases below about unready agents or two enforcers). The self-test probe runs on one node, so it cannot clear those.fail: a pod in one tenant namespace connected to a pod in the other. NetworkPolicy is not enforced, even if a known CNI is installed.- An inconclusive run falls back to the name match.
Without a usable self-test result:
pass: a known CNI was found. This shows the CNI is installed, not that it enforces policy.warn: the agent is not ready on every node, only Flannel was found, or kube-router runs next to another enforcer (another CNI from the list, or a k3s or RKE2 node).unknown: no known CNI was found by name. k3s and RKE2 embed a policy controller that is not visible as a daemonset, and managed clusters hide theirs.
Fix: run orchard doctor --selftest, and then run orchard doctor again. The report that --selftest prints is read before the self-test starts, so its cni-enforcement row reflects the previous run. If a tenant reached another tenant, install Cilium or Calico, or enable your provider’s network policy add-on. On Flannel, set networkPolicy.enforcer.enabled=true. For more information, refer to Enforce network policies on Flannel. With two enforcers, turn one off.
Self-test results are kept in memory on the server that ran the test. After a restart, or when the request reaches another server replica, the check falls back to the name match.
default-deny-coverage
Section titled “default-deny-coverage”Every namespace labelled orchard.my/managed must carry the default-deny-all NetworkPolicy. The doctor counts namespaces that do not. A namespace without it accepts and sends any traffic.
Fix: restart the server. It re-applies the policy to every managed namespace on boot. If a namespace keeps failing, read the server log for the error.
tenant-runtime-class
Section titled “tenant-runtime-class”TENANT_RUNTIME_CLASS puts tenant pods in a sandbox such as gVisor. Unset, tenants share the node kernel (warn). Set to a name that does not exist as a RuntimeClass, pods never schedule (fail).
Fix: set tenantSandbox.enabled=true on nodes that provide the runtime, or create the RuntimeClass.
builder-posture
Section titled “builder-posture”Shows how builds are isolated: vm-sandbox (pass), rootless (warn, shares the node kernel), or privileged-host (fail, a Dockerfile RUN step is root on the node). With a builder RuntimeClass configured, it must exist.
Fix: enable builderSandbox with Kata and KVM, and remove builders.allowPrivilegedOnHost.
pod-security-labels
Section titled “pod-security-labels”Managed namespaces must enforce Pod Security baseline or restricted. Orchard sets baseline on creation, so a failure means someone removed it.
Fix: kubectl label ns <name> pod-security.kubernetes.io/enforce=baseline --overwrite.
admission-policies
Section titled “admission-policies”The ten Orchard ValidatingAdmissionPolicies refuse object shapes Orchard never creates. None installed is a warn, a partial install is a fail, and policies that only Warn or Audit are a warn.
Fix: install the chart with admissionPolicies.enabled=true on Kubernetes 1.30 or later. Set admissionPolicies.validationActions to [Deny] (the default). Set hardeningActions to [Deny] once no pod reports a hardening warning.
redis-auth
Section titled “redis-auth”Redis holds sessions, throttle state and cached data. Without a password in REDIS_URL (or REDIS_PASSWORD), network policy is the only protection (warn). Without REDIS_URL, limits are per pod.
Fix: set a Redis password and include it in REDIS_URL.
secrets-encryption-key
Section titled “secrets-encryption-key”Without SECRETS_ENCRYPTION_KEY, secret environment variables, job secrets and database passwords are stored in plaintext (fail). Falling back to SSO_ENCRYPTION_KEY or a key under 32 characters is a warn.
Fix: set SECRETS_ENCRYPTION_KEY to openssl rand -hex 32 and restart. A boot backfill encrypts existing rows.
jwt-secret-strength
Section titled “jwt-secret-strength”JWT_SECRET signs every session and DNS proof. The doctor fails a missing, short (under 32 characters), example-like or low-variety value, and warns when it equals an encryption key. It never prints the value.
Fix: set JWT_SECRET to openssl rand -hex 32 or longer. Rotating it signs everyone out.
client-ip-trust
Section titled “client-ip-trust”Rate limits and abuse controls use the client address. Behind a proxy, the server sees only the proxy unless TRUST_PROXY says how many proxies to trust. The doctor judges the request you made: a private peer address carrying X-Forwarded-For that resolved to a private client means every user shares one address (warn).
Fix: set TRUST_PROXY to the number of proxies in front of the server, or a comma-separated list of their CIDRs. TRUST_PROXY=true is not valid and trusts nothing. Run the check from the dashboard host, not a port-forward, so it sees the real path.
signup-and-domain-verification
Section titled “signup-and-domain-verification”Open signup means anyone can create an organization. With it, custom-domain verification must be on, or a tenant can claim hostnames they do not own (fail).
Fix: turn on Require proof of domain ownership in Instance admin > Networking, or set CUSTOM_DOMAIN_VERIFICATION=true. Or set SIGNUP_MODE=invite_only.
registry-image-policy
Section titled “registry-image-policy”The orchard-tenant-images policy stops a pod using another project’s repository on the platform registry. off is a fail. A policy that only warns, or is not installed yet, is a warn. In auto mode the server falls back to warn when an existing image would break.
Fix: leave REGISTRY_IMAGE_POLICY unset or set it to enforce, and make sure DOCKER_REGISTRY_PULL and DOCKER_REGISTRY_PUSH are set.
nodeport-exposure
Section titled “nodeport-exposure”While TENANT_NODEPORT_EXPOSURE is not false, tenants can publish TCP ports on node addresses (warn). Nothing inside the cluster stops them being reachable on every node.
Fix: set tenant.nodePortExposure=false, and firewall node ports at your cloud or host.
hostname-style
Section titled “hostname-style”HOSTNAME_STYLE=readable puts organization and project names in public hostnames, which certificate transparency logs expose (warn).
Fix: unset HOSTNAME_STYLE. The default is opaque.
namespace-quotas
Section titled “namespace-quotas”Each managed namespace needs orchard-org-quota (CPU, memory, pods) and orchard-abuse-quota (load balancers, node ports, ephemeral storage). No quota at all is a fail, a missing one is a warn.
Fix: restart the server to re-apply the abuse quota, and set organization CPU and memory caps so the organization quota is created.
priority-class
Section titled “priority-class”The orchard-platform PriorityClass schedules platform pods before tenants and evicts them last. Missing is a warn.
Fix: set priorityClasses.enabled=true.
cert-issuer
Section titled “cert-issuer”The ClusterIssuer named by CERT_CLUSTER_ISSUER (default letsencrypt-prod) must exist and be Ready, or tenant ingresses never get certificates.
Fix: create the ClusterIssuer in cert-manager, or point CERT_CLUSTER_ISSUER at one that exists.
backup-proxy-buckets
Section titled “backup-proxy-buckets”With the backup proxy off (warn), backups and the build cache use instance-wide object-store credentials. With it on, the allowed bucket list lives in the r2-proxy-upstream Secret, which the server does not read, so the result is unknown.
Fix: enable r2Proxy and list only the buckets you intend in r2Proxy.allowedBuckets. Confirm ALLOWED_BUCKETS in the Secret yourself.
metrics-stack
Section titled “metrics-stack”The server calls the /health endpoint of VictoriaMetrics (VICTORIAMETRICS_URL). No answer is a warn.
Fix: check the VictoriaMetrics deployment in the orchard-metrics namespace and the URL.
logging-stack
Section titled “logging-stack”The server calls the /health endpoint of VictoriaLogs (VICTORIALOGS_URL). No answer is a warn.
Fix: check the VictoriaLogs deployment in the orchard-metrics namespace and the URL.
The isolation self-test
Section titled “The isolation self-test”Configuration checks show intent. The self-test shows what the cluster does. It runs only when a superadmin asks for it.
What it does:
- Creates two temporary namespaces named
orchard-selftest-<id>-aand-bthrough the server’s normal namespace creation path, so they get the realdefault-deny-allpolicy, quotas and Pod Security labels. It checks they did. - Starts a listener pod in each, then a probe pod in each, under your tenant RuntimeClass with the same hardened security context as tenant pods.
- From the probe in namespace A, tries a TCP connection to the kube-apiserver (the Service address and every endpoint), every node InternalIP and ExternalIP on ports 10250, 22, 6443 and 50000, the cloud metadata address, every Service in the Orchard, r2-proxy and metrics namespaces, the ingress controller pods’ non-serving ports (1024, 1042, 6060, any in
DOCTOR_INGRESS_DEBUG_PORTS, and any other port the pod declares), and the listener in namespace B. - Deletes both namespaces, waits for them to disappear and reports any that remain.
A dropped connection times out. A refused connection means the packet arrived, so it counts as reached. A target is pass only if every attempt was dropped.
Positive controls run in the same probe: cluster DNS, the public internet on 443 (DOCTOR_EXTERNAL_TARGET), a listener in the same namespace, an ingress controller serving port, and the listener in namespace B from inside namespace B. If any control fails, the probe cannot be trusted, so every result becomes unknown and the controls check says why. A failed control never produces a pass.
The result checks are selftest-namespace-shape, selftest-controls, selftest-apiserver, selftest-nodes, selftest-metadata, selftest-platform, selftest-ingress-ports and selftest-cross-tenant.
selftest-namespace-shape
Section titled “selftest-namespace-shape”The temporary namespace must carry the managed label, default-deny-all, both quotas and a Pod Security enforce label. Fix: refer to default-deny-coverage and namespace-quotas, and read the server log for namespace creation errors.
selftest-controls
Section titled “selftest-controls”The probe reached everything it must reach. Fix: make sure the probe image (DOCTOR_PROBE_IMAGE, default debian:12.15-slim) and listener image (DOCTOR_LISTENER_IMAGE, default busybox:1.38.0) can be pulled, the tenant RuntimeClass can start the pods, and tenant egress allows DNS and the internet. On an air-gapped cluster, point both images and DOCTOR_EXTERNAL_TARGET at internal mirrors.
selftest-apiserver
Section titled “selftest-apiserver”A tenant pod must not reach the kube-apiserver. Fix: enforce NetworkPolicy, add the apiserver or its load balancer address to tenant.extraDenyCidrs, and on Cilium keep the kube-apiserver entity deny.
selftest-nodes
Section titled “selftest-nodes”A tenant pod must not reach node management ports. Fix: firewall 10250, 22, 6443 and 50000 from the pod network at the node, add node addresses to tenant.extraDenyCidrs, and on Cilium keep the host and remote-node entity denies.
selftest-metadata
Section titled “selftest-metadata”A tenant pod must not reach 169.254.169.254. Fix: keep 169.254.0.0/16 in the tenant egress except list, and require IMDSv2 with a hop limit of 1 at the cloud.
selftest-platform
Section titled “selftest-platform”A tenant pod must not reach the server, Postgres or Redis. Fix: enforce NetworkPolicy and keep networkPolicy.platform.enabled=true.
selftest-ingress-ports
Section titled “selftest-ingress-ports”A tenant pod may reach ingress controllers on serving ports only. Controller stats and pprof ports list every tenant’s backend. Fix: keep TENANT_INGRESS_PORTS to the serving ports, and bind controller stats and pprof to localhost where the controller allows.
selftest-cross-tenant
Section titled “selftest-cross-tenant”A tenant pod must not reach an unpublished pod in another tenant namespace. Fix: enforce NetworkPolicy and check the destination namespace’s default-deny-all.
Cost and safety
Section titled “Cost and safety”A run takes one to three minutes, mostly pulling images. It starts four small pods and deletes them. Only one run can be active. If the server stops mid-run, a reaper deletes any namespace with the orchard.my/selftest label that is older than ten minutes, at the start of the next run and every ten minutes. Results are kept in memory on the server that ran the test and are lost on restart.
Run it on a staging cluster the first time you enable it on a new CNI or sandbox runtime.
Related resources
Section titled “Related resources”- Cluster settings to review - Turn on the protections that the doctor reports, with what to check first and how to undo each one.
- Security and isolation - Learn how Orchard isolates tenants, builds, and databases.
- Go to production - Prepare an instance for real users.