Configure logs and request data
Orchard keeps a copy of each app’s logs, and of the ingress controller’s access log, in a log store inside the cluster. Live logs stream straight from the pods and do not need the store, but every feature that looks back in time does: log search, per-route request data, attack signals, and error grouping. This page explains how to size the store, how to point it at your ingress controller’s access log, and how to turn it off on a small machine.
| Feature | Location | Needs |
|---|---|---|
| Search over past logs, levels, context, export | Observe > Logs | The log store |
| Per-route traffic, error rates, and latency | Observe > Requests | The log store and access logs |
| Attack attempts and top sources | Security tab | The log store and access logs |
| Error grouping across replaced replicas | Observe > Errors, and deploy verdicts | The log store |
Without the store, error grouping uses the logs of the pods that are running now. A deploy verdict can then miss errors from replaced replicas.
Components
Section titled “Components”Both components run in the orchard-metrics namespace (metrics.namespace):
- VictoriaLogs is one StatefulSet named
victorialogs, with one volume. - Vector is a DaemonSet. It reads logs from pods in namespaces labeled
orchard.my/managed=true, and from ingress controller pods with the labelapp.kubernetes.io/nameset totraefikorkubernetes-ingress.
logs.enabled is on by default in the chart and the installer. It is separate from metrics.enabled. The chart sets VICTORIALOGS_URL on the server when logs.enabled is on.
Retention and disk
Section titled “Retention and disk”Two limits apply. Whichever limit is reached first takes effect.
logs.retentionis how long logs are kept. The default is14d. The installer writes7don a machine with less than 8 GB of memory.logs.maxDiskSpaceUsageis the disk limit. Above it, the oldest days are dropped. Keep this value below the volume size.
The installer sizes the volume at a tenth of free disk, between 5 GB and 20 GB, and sets the limit to 80% of that. The chart defaults are a 20Gi volume and a 16GiB limit.
To change the limits on an installer host, run the following commands:
sudo orchardctl config set logs.retention 30dsudo orchardctl config set logs.maxDiskSpaceUsage 12GiB --applyStorage class
Section titled “Storage class”logs.victorialogs.storage.storageClass is empty by default, which uses the cluster’s default StorageClass. The installer writes local-path.
A StatefulSet cannot change its StorageClass in place. The following behavior prevents failed upgrades:
- When the value is empty, the chart keeps the class of the live
victorialogsStatefulSet. - On an installer host,
orchardctl updatechecks the volume. If the volume is bound, it pins the values file to the current class. If the volume never bound, it deletes the StatefulSet and its empty claim, and recreates them on the configured class. The same check runs for VictoriaMetrics.
To move a store that holds data to another class, set the value explicitly. The store starts with an empty volume.
Chart values
Section titled “Chart values”| Value | Chart default | Installer, less than 8 GB of memory |
|---|---|---|
logs.enabled |
true |
true |
logs.retention |
14d |
7d |
logs.maxDiskSpaceUsage |
16GiB |
From free disk |
logs.victorialogs.image |
victoriametrics/victoria-logs:v1.54.0 |
|
logs.victorialogs.storage.size |
20Gi |
From free disk |
logs.victorialogs.storage.storageClass |
Empty, the cluster default | local-path |
logs.victorialogs.resources |
Requests 100m/256Mi, limits 1/1Gi |
Requests 50m/96Mi, limits 1/384Mi |
logs.vector.image |
timberio/vector:0.61.0-distroless-libc |
|
logs.vector.dataDir |
/var/lib/vector-orchard, a host path for checkpoints |
|
logs.vector.bufferEvents |
20000 |
5000 |
logs.vector.throttle.eventsPerSecond |
2000, the log rate limit per namespace |
|
logs.vector.resources |
Requests 50m/128Mi, limits 500m/512Mi |
Requests 20m/64Mi, limits 500m/256Mi |
On a small machine, reduce these values instead of turning off the store.
Turn off the log store
Section titled “Turn off the log store”On an installer host, run the following command. orchardctl update does not turn it back on.
sudo orchardctl config set logs.enabled false --applyOn a Helm install, set logs.enabled to false.
Live logs keep working. Log search, the Requests view, and the attack card are removed.
Access logs
Section titled “Access logs”The Requests view and the attack card read the ingress controller’s access log, with one JSON line for each request. Vector parses Traefik and HAProxy (the haproxytech kubernetes-ingress chart). It ignores lines from other controllers.
Orchard removes query strings before it stores a line. It also replaces IDs in paths with :id, so /users/42 and /users/43 count as one route.
Traefik
Section titled “Traefik”Traefik needs access logging on, JSON format, and the User-Agent header kept. Do not drop the ServiceName field, because Orchard uses it to match a line to an app.
On an installer host, this setup is automatic. The installer sets these values, and orchardctl update sets any that are missing, with --reuse-values on the deployed chart version, so Traefik itself does not change version.
On your own cluster, the keys depend on the Traefik chart version:
helm list -n traefikFor chart 41 and later, run the following command:
helm upgrade traefik traefik/traefik -n traefik \ --version <DEPLOYED_CHART_VERSION> --reuse-values \ --set accessLog.enabled=true \ --set accessLog.format=json \ --set accessLog.fields.headers.names.User-Agent=keepFor earlier charts, run the following command:
helm upgrade traefik traefik/traefik -n traefik \ --version <DEPLOYED_CHART_VERSION> --reuse-values \ --set logs.access.enabled=true \ --set logs.access.format=json \ --set logs.access.fields.headers.names.User-Agent=keepServiceName has the form <namespace>-<service>-<port>@kubernetes. Orchard matches it against <namespace>-<app>-<port>, and <namespace>-<app>--green-<port> for the green half of a blue-green deploy. Orchard drops lines for Traefik’s @internal services.
HAProxy
Section titled “HAProxy”HAProxy has no built-in JSON access log, so you must set a format with the field names that Vector reads: ts_ms, host, method, path, status, duration_ms, bytes, client_ip, user_agent, backend, and server.
Add the following to the kubernetes-ingress chart values:
controller: config: syslog-server: 'address:stdout, format: raw, facility:daemon' log-format: '%{+json}o %(ts_ms)[request_date(ms)] %(host)[var(txn.host)] %(method)HM %(path)HPO %(status)ST %(duration_ms)Ta %(bytes)B %(client_ip)ci %(user_agent)[var(txn.ua)] %(backend)b %(server)s' frontend-config-snippet: | http-request set-var(txn.host) req.hdr(host) http-request set-var(txn.ua) req.fhdr(user-agent)- The snippet captures the host and user agent, because the log format cannot read request headers directly.
req.fhdrkeeps commas inside the user agent. - Keep the
syslog-serverline. Without it, the controller reloads HAProxy about once a minute. Each reload refuses long-lived HTTP/2 streams, which browsers report asERR_HTTP2_SERVER_REFUSED_STREAM. - Orchard uses the backend name,
<namespace>_svc_<service>_<port>, to match a line to an app. - The stdout target cuts lines at 1024 bytes. A request with a very long path or user agent produces broken JSON, and Orchard drops it.
Verify access logs
Section titled “Verify access logs”-
Check the controller’s output. Each line should be a JSON object.
kubectl -n traefik logs deploy/traefik --tail=3 -
Check that the store has access lines:
kubectl -n orchard-metrics port-forward svc/victorialogs 9428:9428 &curl -s http://127.0.0.1:9428/select/logsql/query \--data-urlencode 'query=_stream:{kind="access"}' \--data-urlencode 'limit=5'
Observe > Requests appears on an app when it has a domain and the store has access lines for it from the last seven days. Until then, the attack card on the Security tab shows Requests aren’t recorded yet. Orchard caches this answer for about two minutes.
Client IPs show as hidden
Section titled “Client IPs show as hidden”The attack card groups private and cluster addresses (10/8, 172.16/12, 192.168/16, 100.64/10, loopback, and link-local) into one row, Hidden by the load balancer. It does not report them as a flood.
This row appears when the client address is lost before the ingress controller sees it:
- The load balancer uses source NAT: With
externalTrafficPolicy: Cluster, the default, a node can forward a packet to an ingress pod on another node, and it rewrites the source address. - A Cloudflare Tunnel: Every request arrives from
cloudflaredinside the cluster. - A LAN install: Clients on the same network have private addresses.
To keep the client address, set externalTrafficPolicy: Local on the controller’s Service. On Traefik, run the following command:
helm upgrade traefik traefik/traefik -n traefik \ --version <DEPLOYED_CHART_VERSION> --reuse-values \ --set service.spec.externalTrafficPolicy=LocalWith this setting, a node with no ready ingress pod drops the traffic it receives. Run an ingress pod on each node that receives traffic. If one node announces the address (L2 mode), one failing ingress pod on that node takes the address down. A single-machine install has no extra risk.
Behind a proxy or CDN, the client address is in a header. HAProxy can read one with src-ip-header, such as src-ip-header: cf-connecting-ip behind Cloudflare. Use this only if every request comes through that proxy. Anyone who reaches the load balancer directly can set the header to any value.
Related resources
Section titled “Related resources”- Read logs and open a shell - Stream an app’s live logs and run commands in its containers.
- Metrics and alerts - Read CPU, memory, and traffic metrics, and find current problems with apps and databases.