Security and isolation
Orchard treats every organization and every project as a separate tenant, and treats tenants as mutually untrusted. By default, it separates tenants at the network layer: each tenant namespace starts closed, and Orchard opens only the paths an app needs. It does not separate tenants at the kernel layer unless you install a sandbox runtime and turn it on. You could think of a default instance as a set of locked rooms in one building. The doors are locked, but the walls are the shared host kernel.
Some of this isolation is automatic, some holds only when you configure the cluster for it, and some Orchard does not provide at all. What is isolated lists every boundary, What you must do lists the configuration that each one depends on, and What Orchard does not guarantee lists what stays open.
If only you and people you trust deploy to your instance, the defaults suit you. If anyone can deploy code that you did not write, turn on the settings in Requirements for untrusted code.
What is isolated
Section titled “What is isolated”| Boundary | Enforced by | What you must configure |
|---|---|---|
| Organization from organization in the dashboard and API | Membership checks on every route. A user outside an organization gets a not-found response, not a permission error. | Nothing. Limit who is a superadmin, because superadmins can read every organization. |
| Project from project inside an organization | Project roles (admin, member, viewer), project-scoped API tokens, and scoped MCP grants. Tokens that Orchard mints are revoked with their parent. Org owners and admins see every project. | Give people project roles, not org admin. |
| Network between tenants | A default-deny-all NetworkPolicy in every tenant namespace. Refer to Network isolation. | A CNI that enforces NetworkPolicy. Run orchard doctor --selftest to check that yours does. |
| Network from tenants to the platform and the nodes | Tenant egress excludes the cluster’s node and API server addresses. The server, Postgres, Redis, metrics and logs accept connections only from namespaces that are not tenants. | Block the node ports at your firewall. Add public load balancer addresses with tenant.extraDenyCidrs. |
| Images | A cluster rule that lets a pod use only its own project’s images from Orchard’s registry. Refer to Images from other projects. | Keep REGISTRY_IMAGE_POLICY at its default (auto) or enforce. |
| Kernel between tenants | Shared by default. Tenant pods, builds and Postgres pods run on the node’s kernel, like ordinary containers. A sandbox RuntimeClass separates them when you turn one on. Refer to Kernel isolation. | Optional: tenantSandbox with gVisor, builderSandbox with Kata, and databases.runtimeClassName. All are off by default. |
| Nodes | Shared by default. Any organization can run on any node. | Optional: node pools and assignments. |
| Builds | One BuildKit daemon per project, a separate pusher container that holds the registry login, and rootless BuildKit by default. Refer to Builds. | Nodes that allow user namespaces. Optional: builderSandbox with Kata. |
| Build cache | With the backup proxy enabled, each build gets a short-lived key that reaches only its own project’s cache folder. | Enable the proxy and let it serve the cache bucket. |
| Secrets | Encrypted at rest, write-only in the API, and mounted only in the project’s namespace. | SECRETS_ENCRYPTION_KEY. |
| Shell access | People with edit access to a project can open a shell in its app pods. Database, pooler, Tailscale proxy, job and build pods refuse it. Refer to Shell access and databases. | Nothing. |
| Database backups | The backup proxy gives each database a key that works only for its own bucket prefix. | r2Proxy.allowedBuckets listing only the buckets you intend. |
| Objects in tenant namespaces | The admission guard refuses object shapes that Orchard never creates. | Kubernetes 1.30 or later. Optional: switch the hardening rules from warn to refuse. |
| Single sign-on, group rules and SCIM | Sessions from one organization’s identity provider see only that organization. Group rules can give admin, member or guest, never owner. | Verify the organization’s domains before you enforce SSO. |
| Quotas | A ResourceQuota and LimitRange in each namespace, plus caps on load balancers, node ports and ephemeral storage. | Set organization CPU and memory caps. Without them the namespace has no CPU or memory quota. |
What you must do
Section titled “What you must do”Orchard cannot do these for you. If one is missing, the matching guarantee does not hold. Run orchard doctor to check them, and orchard doctor --selftest to prove that the network controls work. Refer to Check isolation with orchard doctor.
| Requirement | Setting | Why |
|---|---|---|
| A CNI that enforces NetworkPolicy | Cilium, Calico, Antrea or kube-router. Plain Flannel does not enforce it. For every supported CNI, refer to Supported CNIs. On Flannel, refer to Enforce network policies on Flannel. | Every network control is a NetworkPolicy. A CNI that does not enforce it leaves every tenant able to reach every other tenant, without any error. |
| A firewall for node ports | Block 10250, 6443, 2379, 2380, 22, 9100 and 8472/udp from the internet and from tenant networks. Refer to Host firewalls. | Orchard keeps tenants away from these addresses. The firewall closes them for everyone else. |
| Public load balancer addresses | tenant.extraDenyCidrs (PLATFORM_EXTRA_DENY_CIDRS), a comma-separated list. Include the service CIDR if nodes do not report it. | Orchard finds node, pod and API server addresses itself. It cannot find a load balancer that is not one. |
| Ingress controller namespace | tenant.ingressNamespaces (TENANT_INGRESS_NAMESPACES), if the controller is not in network or orchard. | Tenant ingress is default-deny. A wrong value stops sites and certificate issuing. |
| Origin lock and proxy settings | Allow only your CDN or tunnel to reach the origin. Set clientIp.trustProxy (TRUST_PROXY) to the number of proxies in front of the server, and clientIp.trustedProxyRanges for a CDN. Refer to Rate limits and client addresses. | Per-IP limits and audit logs use this address. Traffic straight to the origin skips your CDN and WAF. |
| Reverse DNS block | Block reverse (PTR) lookups in CoreDNS. The installer does this on its own k3s cluster. Refer to Reverse DNS lookups. | Stops a pod from listing every Service name in the cluster by reverse lookup. |
| User namespaces for builds | On Ubuntu 24.04 build nodes, set kernel.apparmor_restrict_unprivileged_userns=0. The installer does this on its own host. | Default builds are rootless and need unprivileged user namespaces. Otherwise the build fails. |
| App domain on the Public Suffix List | Submit your generated app domain to the Public Suffix List. Use a different registrable domain from the dashboard. Do not set COOKIE_DOMAIN to a parent of the app domain. | Without it, apps on sibling hostnames can set cookies for each other. |
| Secrets key | Set SECRETS_ENCRYPTION_KEY. The installer does this. On a Helm install, put it in server.env. | Without it, secrets are stored in plain text in Postgres. |
| Custom-domain verification | CUSTOM_DOMAIN_VERIFICATION=true. | It starts on when SIGNUP_MODE is open. With other signup modes it starts off, so the first project to claim a hostname wins. |
| Organization caps | Set CPU and memory caps per organization. Refer to Set quotas. | Without caps one organization can use all the CPU and memory of the nodes. |
| Dedicated registry account | Use a low-privilege account in HARBOR_USERNAME, not the Harbor admin account. | All tenants’ builds share this login. |
| Registry image policy | REGISTRY_IMAGE_POLICY at enforce or auto, with DOCKER_REGISTRY_PUSH and DOCKER_REGISTRY_PULL set. The installer sets enforce. | Stops a pod from running another project’s image. |
| Builder egress | builders.registryEgress lists your registry. | Builds can only reach what this allows. |
| Control-plane network policies | Leave networkPolicy.platform.enabled on. List any namespace of your own that runs untrusted code in networkPolicy.platform.untrustedNamespaces. Refer to Control-plane namespaces. | The metrics and log stores have no authentication of their own. Do not expose VictoriaMetrics or VictoriaLogs outside the cluster. |
| Backup proxy | r2Proxy.allowedBuckets lists only the buckets you intend. Leave r2Proxy.networkPolicy.enabled on. Use a separate bucket or key for uploads. | The proxy holds the real object storage credentials. |
| Scoped build cache keys | Enable the backup proxy and let it serve the cache bucket. For a bucket other than the platform bucket, set BUILDKIT_S3_VIA_PROXY=true and add it to r2Proxy.allowedBuckets. | Without the proxy, the S3 cache is off unless BUILDKIT_S3_SHARED_KEY=true, which hands every build the instance-wide key. Single tenant only. |
| Instance SSO | Verify an organization’s domains before you enforce SSO. Keep instance-level SSO bypass off. | An identity provider is trusted for the users and groups of the organization that configured it. |
| Kubernetes Secret encryption | Turn on encryption at rest for Secrets in your cluster. | Orchard stores values as Kubernetes Secrets. |
Enforce network policies on Flannel
Section titled “Enforce network policies on Flannel”NetworkPolicy is a standard Kubernetes object, but the API server only stores it. The CNI plugin enforces it. Flannel, the default CNI on Talos and on several other distributions, does not enforce NetworkPolicy, so on a Flannel cluster every tenant policy has no effect. k3s also uses Flannel, but it embeds the kube-router network policy controller next to it, so k3s enforces policies unless you start it with --disable-network-policy.
To enforce policies on a Flannel cluster, add the following to your values file and run helm upgrade:
networkPolicy: enforcer: enabled: trueThe chart runs kube-router on every node in policy-only mode, the same controller that k3s embeds. kube-router writes the policy rules into the node firewall. It does not route pod traffic, proxy Services or write a CNI configuration, so Flannel keeps the pod network. The kube-router DaemonSet runs in the orchard-netpol namespace, which allows privileged pods.
Do not turn on the enforcer on a cluster that already enforces NetworkPolicy, such as one with Cilium, Calico, Antrea or Canal, or a k3s cluster with its built-in controller. Two controllers would write rules for the same policies. The cni-enforcement check in orchard doctor warns when it finds kube-router next to another enforcer.
When the kube-router pods are ready on every node, run the isolation self-test to confirm that the policies hold:
orchard doctor --selftestFor what the self-test sends and how to read the result, refer to The isolation self-test.
Network isolation
Section titled “Network isolation”Network isolation is on by default. Every tenant namespace has a default-deny-all NetworkPolicy for ingress and egress, so a tenant pod can only use the following paths:
| Direction | Allowed |
|---|---|
| Ingress | Pods in the same namespace, and the namespaces in TENANT_INGRESS_NAMESPACES |
| Egress | Pods in the same namespace |
| Egress | The namespaces in TENANT_INGRESS_NAMESPACES on TCP 80, 443, 8000, 8080 and 8443 only (TENANT_INGRESS_PORTS) |
| Egress | kube-dns on UDP and TCP port 53 |
| Egress | The public internet, except private and link-local ranges, and the cluster’s own node and API server addresses |
| Egress | Other tenant namespaces, which are default-deny on ingress |
| Egress | The backup proxy on TCP port 9000 |
A pod that reaches another tenant namespace can only reach what that tenant has published.
flowchart LR POD[Tenant pod] POD -->|allowed| SAME[Same namespace] POD -->|allowed| DNS[kube-dns :53] POD -->|allowed| ING[Ingress namespaces] POD -->|allowed| NET[Public internet] POD -.->|blocked| API[Kube API server] POD -.->|blocked| IMDS[Cloud metadata<br/>169.254.0.0/16] POD -.->|blocked| PLAT[Control-plane Postgres<br/>registry] POD -.->|blocked| OTHER[Other tenants'<br/>unpublished services]
The policy blocks the following:
- The Kubernetes API server and the addresses of every node, including public ones. Orchard reads them from the cluster and keeps the policy current when nodes change. Add anything else, such as a public load balancer, with
PLATFORM_EXTRA_DENY_CIDRS. - The ingress controller’s stats and debug ports. Tenants reach it on its serving ports only.
- Cloud metadata services at
169.254.0.0/16. - The ranges
10.0.0.0/8,172.16.0.0/12,192.168.0.0/16, and100.64.0.0/10, where control-plane services run. - Unpublished services of other tenants.
Orchard restores this policy if someone changes it with kubectl. If Orchard cannot apply the default-deny policy to a new namespace, it deletes the namespace. A deleted default-deny policy is recreated at once, and a sweep every five minutes catches one that a watch event missed.
With Cilium, Orchard also adds a deny-platform-egress policy to each tenant namespace that denies the host, remote-node and kube-apiserver entities (TENANT_DENY_ENTITIES), which holds even when an allow rule would match. The chart adds the same denies to the builders, sandbox, fc-builder and image-scan namespaces. Do not set TENANT_DENY_ENTITIES=none unless your ingress controller uses host networking and your apps call their own hostnames, and do not set networkPolicy.egressDeny.enabled: false. Without Cilium, Orchard approximates these denies with address exceptions. Refer to What differs without Cilium.
Control-plane namespaces
Section titled “Control-plane namespaces”The chart also closes the other direction. With networkPolicy.platform.enabled (the default), pods in the release namespace and the metrics namespace accept connections only from their own namespace and from namespaces that are not tenants. A tenant namespace, the builders, sandbox, fc-builder and image-scan namespaces cannot connect to the server, Postgres, VictoriaMetrics or VictoriaLogs, whatever their own policy allows.
There are a few exact exceptions:
- Image scan jobs can reach TCP 3001 on the server, to post their report.
- Redis accepts only the server, and requires a password that the chart generates.
- The backup proxy (
r2-proxy) accepts only tenant namespaces, the server, and the CloudNativePG operator.
Do not apply infra/k8s/orchard-metrics/30-network-policies.yaml on a shared cluster. It lets every namespace query the metrics store, and the chart’s control-plane policies replace it.
Images from other projects
Section titled “Images from other projects”Every node can pull from Orchard’s registry, so a pod could otherwise run any image it can name. A cluster rule stops this: a pod in a project’s namespace can only use that project’s images from Orchard’s registry. Images from other registries are not affected.
The REGISTRY_IMAGE_POLICY variable sets what the rule does:
| Value | Effect |
|---|---|
enforce | Kubernetes refuses the pod. The installer sets this value. |
unset or auto | The default. Orchard enforces the rule unless a current app image or a pinned POSTGRES_IMAGE_<major> on Orchard’s registry would be refused. In that case it warns and logs which images block it. |
warn | The pod runs, and Kubernetes reports a warning. |
off | Orchard removes the rule. |
Any other value enforces the rule, so a typo cannot weaken it. Orchard labels namespaces and restores the rule every five minutes.
Public TCP and UDP ports
Section titled “Public TCP and UDP ports”A tenant can publish a raw TCP or UDP port in two ways. The two have different exposure, so Orchard treats them differently.
Public ports on the organization IP. A person with edit access turns on a public port for an app or database. Orchard picks the port from 30000 to 32767, so a tenant never chooses it, and it creates a LoadBalancer Service on the organization’s public IP (or tenant.publicIp). On a cluster that uses Cilium LB-IPAM, all organizations share that IP, and Orchard checks the database and the live Services so two tenants do not get the same port. Tenants cannot create Services themselves, so they cannot take a platform port such as 80 or 443. Each port gets its own -allow-public NetworkPolicy that opens only that port on that app’s pods. The Service uses externalTrafficPolicy: Cluster, so the app sees a node address as the client address, not the real one. Treat these ports as untrusted internet-facing endpoints, and rate-limit or authenticate inside the app.
NodePort (exposePublic). A NodePort is reachable on every node address, including control-plane and builder nodes, on a port from the cluster’s node-port range. This suits a single-node install, where the node is the public address. The installer narrows the range to 30000-30100 and opens only that range in ufw. On a shared multi-node cluster that already publishes ports on a load-balancer IP, NodePorts open a second path that skips the shared edge and any cloud firewall rules you rely on, so turn them off:
tenant: nodePortExposure: falseA tenant cannot ask for a specific node port. The cluster assigns one, and an update keeps the one the Service already has. The address Orchard shows for a NodePort is PUBLIC_EXPOSURE_IP (tenant.publicIp) when you set it, and the first node’s address otherwise.
One organization can hold at most 20 public ports in total, counting NodePorts, app public ports and public databases. Change the limit with tenant.maxPublicPortsPerOrg (MAX_PUBLIC_PORTS_PER_ORG; 0 is unlimited). On a self-hosted install, a firewall rule for the node-port range is your responsibility on anything other than ufw.
Host firewalls
Section titled “Host firewalls”A ufw deny rule does not close ports 80 and 443 on a machine that runs the installer. Traffic to these ports is forwarded to pods and does not pass through the rules that ufw applies to the machine. The installer leaves your deny rule in place and warns you. To close a port, use a firewall in front of the machine.
Two host ports serve only the cluster:
| Port | Service | Who needs it |
|---|---|---|
9100/tcp | node-exporter | vmagent, which scrapes each node from the pod network. The port requires a password. |
8472/udp | Flannel VXLAN | Other nodes in the cluster. A single-node installer host has none, so nothing needs this port. |
When ufw or firewalld is active, the installer and orchardctl harden add a deny rule for both ports, after the rules that allow the pod and service networks (10.42.0.0/16 and 10.43.0.0/16). Traffic from the machine itself uses the loopback interface, which both firewalls accept. Without a host firewall, the installer prints a note, and you must close both ports in your provider’s firewall.
On a cluster installed with the Helm chart, node-exporter runs on the host network and listens on the node’s IP address on port 9100 while metrics.enabled is on. Block port 9100 at the node or cloud firewall for every source outside the cluster, and allow the pod network so that vmagent can scrape it. Node-exporter cannot listen on a private address only, because vmagent reaches it through the address that the node reports.
Reverse DNS lookups
Section titled “Reverse DNS lookups”Every pod can reach cluster DNS. With the default CoreDNS configuration, a pod can sweep the service address range with reverse (PTR) lookups and read the name of every Service, which includes the namespace names of other tenants. Orchard cannot close this from inside a tenant namespace, so the block belongs in CoreDNS.
The installer does this on its own k3s cluster. It writes a coredns-custom ConfigMap in kube-system and restarts CoreDNS. On any other cluster, add the same rule to the CoreDNS Corefile, inside the main server block:
acl in-addr.arpa ip6.arpa { block type PTR}On k3s without the installer, put the block in a coredns-custom ConfigMap under a key that ends in .override. Check the result from a tenant pod: dig -x <any Service ClusterIP> must return REFUSED.
The block also stops pods from reverse-resolving public addresses through cluster DNS, which few programs need. Forward lookups such as service.namespace.svc.cluster.local keep working, and a name that has never been published stays hard to guess, because namespace names end in a random suffix.
Kernel isolation
Section titled “Kernel isolation”Kernel isolation is off by default. Tenant pods and build jobs share the host kernel, like ordinary containers, so a container escape compromises the host.
Two optional runtimes change this. The one-command installer does not install either of them.
| Setting | Runtime | Protects |
|---|---|---|
tenantSandbox | gVisor (runsc) | Tenant app pods, with a user-space kernel. |
builderSandbox | Kata Containers | Image builds, by running the pod in a lightweight virtual machine. |
BuildKit needs privileged: true, which gVisor does not allow. Builds therefore use Kata, where a privileged container stays inside the virtual machine.
Greenhouse machines run as Kata virtual machines on the Kubernetes runner when the cluster has Kata. For more information, refer to Run Greenhouse.
Turn on kernel isolation
Section titled “Turn on kernel isolation”The runtimes are node-level and vary by distribution, so Orchard does not install them.
-
Install the runtime on every node. On Talos, use the
siderolabs/gvisorandsiderolabs/kata-containerssystem extensions. On other distributions, follow the runtime’s instructions. -
Confirm that the RuntimeClass exists:
kubectl get runtimeclass gvisor -
Run a test pod that uses the RuntimeClass.
-
Add the settings to your values file:
tenantSandbox:enabled: trueruntimeClassName: gvisorbuilderSandbox:enabled: trueruntimeClassName: kata -
Apply the change. On an installer host, edit
/etc/orchard/values.yamland runsudo orchardctl update. On a Helm install, runhelm upgrade.
Complete step 1 before step 4. If the runtime does not exist, new tenant pods fail with the following error:
no runtime for "runsc" is configuredSandbox database pods
Section titled “Sandbox database pods”CloudNativePG pods (Postgres instances and PgBouncer poolers) run on the host kernel, because the CloudNativePG Cluster resource has no runtimeClassName field. To run them under gVisor or Kata, set the following value:
databases: runtimeClassName: gvisorThe chart then installs a MutatingAdmissionPolicy that sets the runtime class on pods that carry a cnpg.io/cluster, cnpg.io/poolerName or orchard.my/tailscale label in namespaces labelled orchard.my/managed. It needs a Kubernetes version that serves MutatingAdmissionPolicy, and the RuntimeClass must exist on every node that can run databases.
This setting is optional. The default is empty, which changes nothing. Sandboxed Postgres has higher IO and latency overhead. The policy applies only to pods created after you install it. To move an existing cluster, do a switchover and then delete the old pods one at a time, so you control when each database restarts. Set databases.runtimeClassFailurePolicy: Fail to guarantee that no unsandboxed database pod is admitted. The default Ignore keeps a policy fault from blocking pod creation.
The Tailscale proxy pod is created by Orchard. It uses the tenant runtime class (TENANT_RUNTIME_CLASS), drops all capabilities, and runs in userspace mode, so it needs neither NET_ADMIN nor NET_RAW. Its image defaults to a ghcr.io/tailscale/tailscale release pinned by digest. Set TAILSCALE_IMAGE to use another. A running proxy pod keeps its settings until you turn Tailscale off and on again for the app.
Builds
Section titled “Builds”A build runs a Dockerfile from a repository, which makes it the most exposed thing Orchard runs. Orchard limits what a build can reach and what it can hold.
Builds without a sandbox runtime
Section titled “Builds without a sandbox runtime”Without builderSandbox, Orchard runs builds in rootless BuildKit (moby/buildkit:<version>-rootless). The container is not privileged, runs as uid 1000, and gets its root inside a user namespace. It uses --oci-worker-no-process-sandbox, with the seccomp and AppArmor profiles set to Unconfined, which is what rootless BuildKit needs on Kubernetes. A Dockerfile RUN step is not root on the node, but it still shares the node kernel and has a larger kernel attack surface than the default profile. Use builderSandbox when you can.
Without the process sandbox, RUN steps share the build container’s process namespace and run as the same user as buildkitd and buildctl. A RUN step can therefore read the command line and environment of every process in the build container, and the files those processes can see. Orchard keeps credentials out of command lines, but treats anything in a rootless build container as readable by the Dockerfile, so it never puts a credential there that reaches beyond the build’s own project.
The process sandbox stays off because rootless BuildKit cannot mount a fresh /proc under the masked /proc a Kubernetes container gets. procMount: Unmasked would lift the mask, but Kubernetes accepts it only with hostUsers: false, which needs user namespace and ProcMountType support in the cluster (Kubernetes 1.33 or later, containerd 2 or CRI-O, Linux 6.3 or later and idmap-capable volume filesystems). A pod user namespace also maps only 65536 IDs, which leaves no room for the subordinate ID range rootlesskit maps inside it. Orchard does not offer that mode yet.
Rootless builds need nodes that allow unprivileged user namespaces. On Ubuntu 24.04, that means kernel.apparmor_restrict_unprivileged_userns=0. When a node does not allow them, the build fails with a message that says so. Rootless builds always push from the separate pusher container, whatever BUILDER_SIDECAR_PUSH says, and the persistent daemon keeps no registry login. RUN --security=insecure and RUN --network=host are refused with a clear error in every mode.
At startup, the server logs build posture: vm-sandbox, rootless or privileged-host.
Build credential isolation
Section titled “Build credential isolation”With the default BUILDER_SIDECAR_PUSH (anything but false), no container that runs or serves a tenant’s Dockerfile holds the registry login:
- Super Builds (Depot): The build session gets only the app’s own base-image pull credentials. The image comes back as an OCI tarball, and a separate
pushercontainer, which never runs tenant code, pushes it. WithBUILDER_SIDECAR_PUSH=falseon a Kata or privileged build, the shared registry login is placed in the Depot session, so a Dockerfile can pull any image in that registry. Harbor robot accounts cannot narrow this, because their permissions apply to a whole Harbor project, not a single repository, and Orchard keeps every tenant’s images in one project. - Persistent BuildKit daemons: Each daemon pod is reachable only from build pods that carry the same
orchard.my/tenant-keylabel, and the chart’sbuildkit-daemon-network-policylimits daemon egress to the same allowlist as build pods. Daemons do not mount the registry login. The build pod pushes. Keepbuilders.registryEgresscorrect, because daemons pull and build pods push through it. - S3 build cache: When the backup proxy is enabled and serves the cache bucket, each build gets its own key, valid for two hours, that can only read and write
_build-cache/<project>/in that bucket. The proxy holds the real credentials. The key reachesbuildkitdonly asAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYin its own environment. It is never in abuildctlcommand line or in the cache attributes, which BuildKit forwards to the build’s frontend, including a Dockerfile# syntax=image. AnyAWS_*variables of the app are removed frombuildkitd’s environment and still reach the build as usual.
The cache bucket is the instance backup bucket unless BUILDKIT_S3_BUCKET names another one. For another bucket, add it to r2Proxy.allowedBuckets and set BUILDKIT_S3_VIA_PROXY=true. When the proxy does not carry the cache, the S3 cache is off and the server logs an error at startup, because the only key left is the instance-wide one. A rootless RUN step can read that key from buildkitd’s environment, and on any posture a # syntax= frontend can use it to import cache from any prefix. On a single-tenant instance, BUILDKIT_S3_SHARED_KEY=true turns the cache back on with the instance-wide key, and the server logs a warning at startup. Scoped keys cannot copy objects, so BuildKit’s touch_refresh is set far in the future and the cache never refreshes timestamps.
Shell access and databases
Section titled “Shell access and databases”People with edit access to a project can open a shell in, or run commands in, that project’s app pods. Edit access means the project Admin or Member role, or the owner or admin role in the organization. Viewers cannot.
Orchard refuses a shell in database pods, connection-pooler pods, Tailscale proxy pods, job pods and build pods. The request fails with a 403 that points to the database connection string. Pod logs stay readable.
To work in a dedicated Postgres database, run orchard db psql <id> or open the database console in the dashboard. Either one starts a short-lived psql client pod in your project’s namespace, which has the following limits:
- It runs under the tenant RuntimeClass, with a read-only root filesystem, no capabilities, no service account token, and resource limits.
- It connects as the database’s app role, never the Postgres superuser.
- It is deleted when the session ends, or after
PSQL_SESSION_TTL_SECONDS(default one hour).PSQL_SESSIONS_PER_DATABASE(default 3) caps concurrent sessions. - The pod and the credentials secret live only in your namespace, and creating them needs edit access to the project.
Orchard creates dedicated databases with the Postgres postgres superuser login turned off (enableSuperuserAccess: false), because Orchard never uses that login. Databases created by earlier versions of Orchard keep the login until you turn it off. For the steps, refer to Turn off the Postgres superuser on older databases.
Admission guard for tenant namespaces
Section titled “Admission guard for tenant namespaces”The chart installs native ValidatingAdmissionPolicy objects for namespaces that carry the orchard.my/managed label. They are a second wall behind Orchard’s own checks: if a bug ever makes the server build a bad object, the API server refuses it. They need Kubernetes 1.30 or later.
The guard refuses the following:
- Host-backed volumes, and privileged pods.
externalIPsandExternalNameServices.- LoadBalancer Services, or load-balancer IP pinning, from anyone but the server.
- Ingress annotations outside the allowlist, and a cert-manager issuer that is not configured.
- PersistentVolumeClaim hijacking, and NetworkPolicy egress that is open.
- Unsafe CloudNativePG cluster settings.
The policies are safe to install. Every policy fails open (failurePolicy: Ignore), every parameter is optional, and cert-manager, CloudNativePG and the Kubernetes controllers are recognised by the labels on what they create, not by a username that differs on every install.
The guard refuses by default. The hardening rules (drop ALL capabilities, and use the tenant RuntimeClass) only warn, because pods that earlier versions of Orchard created, and Tailscale proxy pods, can fail them, and a refusal would stop those pods from restarting. Watch kubectl get events and the API server audit log, then switch those rules to refuse too:
admissionPolicies: hardeningActions: [Deny] extraIssuers: [letsencrypt-dns01-wildcard]Set extraIssuers to every other ClusterIssuer you use for tenant hosts, or the issuer rule refuses them. Set validationActions: [Warn, Audit] to report without refusing. Set admissionPolicies.enabled: false to remove the guard. The parameter reference, the CEL tests and the fixtures for a live dry run are in infra/charts/orchard/policies/README.md.
Control-plane permissions
Section titled “Control-plane permissions”The chart creates the orchard-server ServiceAccount, a <release>-server ClusterRole, and a namespaced Role for leader-election Leases.
Orchard creates tenant namespaces at runtime, so its write permissions are cluster-scoped. They cover namespaces, pods (with exec, attach, and port-forward), Deployments, Services, Ingresses, PersistentVolumeClaims, NetworkPolicies, Jobs, CronJobs, EndpointSlices, and CloudNativePG resources, plus read and patch on nodes.
Secrets and ConfigMaps are not in that ClusterRole. A second ClusterRole, <release>-tenant-secrets, covers only those two. The server’s role can bind that one ClusterRole by name and no other. When the server creates a tenant namespace, it adds a RoleBinding there, so it gets access to Secrets only in the namespaces it creates. The chart passes the name in TENANT_SECRETS_CLUSTER_ROLE. Without that variable, the server creates a Role in each namespace instead.
If you set KUBECONFIG_PATH or KUBECONFIG_BASE64, the server uses the identity in that file and ignores the ServiceAccount, so the ClusterRole does not limit it. Use the in-cluster ServiceAccount unless the control plane runs outside the cluster it manages.
If Cilium is your CNI, set cilium.enabled: true to add the permissions for CiliumNetworkPolicy and CiliumEnvoyConfig.
What you can choose
Section titled “What you can choose”These settings cost something, so they are your decision. Each one, with what to check before and how to undo it, is on Cluster settings to review.
| Setting | What it does | Cost |
|---|---|---|
tenantSandbox.enabled (gVisor) | Tenant pods run on a user-space kernel. Refer to Kernel isolation. | Install the runtime on every node first. Some syscalls and IO are slower. |
builderSandbox.enabled (Kata) | Builds run in a lightweight virtual machine. | Needs Kata and KVM on the build nodes. |
publicTls.db.mode: perDatabase | One certificate per public database. The shared wildcard key stays out of tenants. | Needs a DNS-01 issuer and counts against Let’s Encrypt rate limits. |
databases.runtimeClassName | Runs database, pooler and Tailscale proxy pods in the sandbox. Refer to Sandbox database pods. | Needs the MutatingAdmissionPolicy API. IO and latency cost. It affects only pods created afterwards. |
tenant.nodePortExposure: false | Stops tenants opening NodePorts on every node. Refer to Public TCP and UDP ports. | On a shared multi-node cluster, only load balancer ports remain. |
admissionPolicies.hardeningActions: [Deny] | The API server refuses pods that keep capabilities or run outside the tenant RuntimeClass. Refer to Admission guard. | Read the warnings first. Older pods and Tailscale proxy pods can fail the hardening rules. |
admissionPolicies.extraIssuers, storageClasses, pgImagePrefixes, pgPreloadLibraries | Fill in the parameters of the admission rules. | Empty lists skip those rules. Wrong values refuse real objects once on Deny. |
| Pod Security labels | Orchard sets baseline enforce and restricted warn and audit on tenant namespaces. | Enforcing restricted breaks images that run as root. Do not remove the labels. |
HOSTNAME_STYLE=readable | Generated hostnames contain the app and project names. | The names show in DNS and certificate logs. The default is opaque. |
Requirements for untrusted code
Section titled “Requirements for untrusted code”Code is untrusted if someone can run a process that you did not write. This includes letting other people deploy from their own repositories. In that case, set up all of the following:
builderSandboxwith Kata: Without it, builds run arbitrary Dockerfiles rootless, on the host kernel. This is the most important item.tenantSandboxwith gVisor: Without it, a compromised app shares a kernel with everything else on the node.- Quotas: Refer to Set quotas.
- Node pools: Refer to Manage nodes and pools.
invite_onlyorclosedsignups: Refer to Close signups.
Without these settings, a single-machine Orchard that runs other people’s code has about the same exposure as giving them Docker on that machine.
What Orchard does not guarantee
Section titled “What Orchard does not guarantee”- Kernel isolation by default. Without gVisor or Kata, there is no per-tenant kernel. Tenant pods, builds and Postgres pods share the host kernel, so a container escape reaches the node and every tenant on it. Refer to Kernel isolation.
- Hardware isolation. gVisor and Kata reduce the attack surface of the kernel. Neither is a separate machine.
- Isolation without a NetworkPolicy-enforcing CNI. Orchard creates the policies and cannot tell whether they are enforced.
- Protection from the control plane itself. The server’s service account can create pods in any namespace, grant itself read access to Secrets in any namespace, and remove the isolation policies. A compromise of the server is a compromise of the cluster. Keep the server patched and limit who can reach it. Refer to Control-plane permissions.
- Separate registry credentials. All tenants’ builds use one registry login, kept in a container that does not run tenant code.
- Separate encryption keys. One key encrypts secrets for every organization.
- Secrecy from people with edit access. Anyone with edit access can open a shell in their project’s app pods and read their environment.
- Hiding names from cluster operators. Namespace names contain the organization and project slugs, and an organization transfer keeps the old name.
- Scoping at the data layer. Authorization runs in the API services. There is no row-level security in Postgres, so a missing check in one route is not caught by the database.
- A sandbox runtime you did not set. If
TENANT_RUNTIME_CLASSis unset, pods run on the node’s runtime. The hardening policy only warns. - Strict pod settings.
restrictedPod Security is not enforced andreadOnlyRootFilesystemis not set. Database pods are not sandboxed unless you setdatabases.runtimeClassName. - Confined builds. Rootless builds run with seccomp and AppArmor set to
Unconfined. WithBUILDER_SIDECAR_PUSH=falseon a Kata or privileged build, the shared registry login is placed in the build. Refer to Builds. - Network filtering beyond the policies. IPv6 egress is denied by default-deny, not filtered. Without Cilium, host-network services are protected only by the address exceptions computed from the node list. With Cilium,
allow-cilium-ingressadmits thehostandremote-nodeentities to tenant pods. - Keeping database TLS keys out of tenants. With
publicTls.db.mode: wildcard, the default, the wildcard private key is copied into each tenant namespace that has a public database. - Stable values across a
JWT_SECRETrotation. Rotating it invalidates every DNS ownership proof. - Cookie isolation by prefix. The session cookie has no
__Host-prefix. Host-only scope depends on leavingCOOKIE_DOMAINunset or scoped to the dashboard. - Quotas at the first moment. Quotas are per namespace, per-organization totals are computed by the API, and there is a short window between namespace creation and the first quota sync. There is no node-level isolation unless you use node pools.
- Erasing data. After a delete, what happens to the underlying data depends on your StorageClass and its reclaim policy.
- A bucket an organization brings. If a database uses an organization’s own bucket, that credential belongs to the organization.
What a tenant can learn about other tenants
Section titled “What a tenant can learn about other tenants”With the requirements above in place, tenant pods cannot reach the Kubernetes API, the nodes, the metrics and log stores, or other tenants’ unpublished services. These remain:
- Public hostnames of anything with its own certificate, through Certificate Transparency logs. Generated hostnames are opaque, and a wildcard certificate hides the labels. Refer to Set up wildcard certificates.
- Service names in cluster DNS, through forward lookups. The names end in a random suffix, so they are hard to guess.
- Services another tenant has published, which answer on the pod network as they do on the internet.
- On the login endpoints, the name, slug and sign-in providers of an organization that has SSO turned on, if someone guesses its slug. An organization without SSO is indistinguishable from a missing one.
The full inventory, with the code and tests behind each guarantee, is in the tenancy model in the repository.
Related resources
Section titled “Related resources”- Go to production - Work through the settings to check before other people depend on your instance.
- Cluster settings to review - Review each optional setting, what to check before, and how to undo it.
- Set up image scanning - Scan the images that Orchard builds for known vulnerabilities.
- Protect an app - Put a password or sign-in in front of an app.
- Architecture - Review the components of an instance and how they fit together.