Skip to content
Install

Create alert rules

An alert rule tells you when an app is in trouble before your users do. Each rule watches one metric on an app, such as CPU or the 5xx rate, and fires when the metric stays past a threshold for the period you set. Orchard then sends an alert.firing notification, so you can route the alert to a team channel.

  • The Owner or Admin role in the organization, or admin access to the project. Only organization owners and admins can create organization rules.
  • A running metrics store on your instance. It is on by default (metrics.enabled in the Helm chart).
  • On an instance that uses PostHog, the Notifications beta turned on for your account. Go to Account settings > Early access.
  1. Open the app.
  2. Go to Settings > Notifications.
  3. In Alerts, select Add alert.
  4. In Metric, select the metric to watch.
  5. In Is, select above or below.
  6. In Threshold, enter the value.
  7. In For, select how long the metric must stay past the threshold.
  8. (Optional) In Min req/s, enter a minimum request rate. This field appears only for metrics that use a request rate guard.
  9. Select Add alert.

The rule appears in the Alerts list with the status OK.

MetricIDUnitComparisonDefault thresholdDefault period
CPUcpu_percent%Above855 minutes
Memorymemory_percent%Above855 minutes
Requestsrequest_ratereq/sAbove or below1005 minutes
5xx rateerror_rate%Above55 minutes
Average latencylatency_avg_msmsAbove5005 minutes
Max latencylatency_max_msmsAbove20005 minutes
CPU throttlingcpu_throttled_percent%Above2510 minutes
Hours until out of memorymemory_forecast_hourshBelow630 minutes

CPU and memory percentages compare usage across all replicas with the total limit of those replicas. If an app has no limit of its own, Orchard uses the organization default.

Requests, 5xx rate, Average latency, and Max latency need a domain on the app. Orchard skips apps without a domain.

5xx rate, Average latency, and Max latency use a request rate guard. Below the minimum request rate, Orchard does not compare the metric with the threshold. The default minimum is 0.2 requests per second.

StatusMeaning
OKThe metric is within the threshold.
Over thresholdThe metric is past the threshold, but the period has not ended.
FiringThe metric stayed past the threshold for the full period.
RecoveringThe metric is back within the threshold, but the recovery period has not ended.
No dataThe metrics store did not answer while the rule was active.
OffThe rule is paused.

Orchard evaluates rules every 30 seconds.

The recovery period equals the rule’s period, with a minimum of 2 minutes and a maximum of 10 minutes. When the recovery period ends, Orchard sends an alert.resolved notification.

A firing rule also appears above the app’s tabs and on the project’s Overview tab. If the app is a component of a status page, the component shows Degraded.

A rule does not contact a team channel by itself. To send alerts to a channel:

  1. Open the app.
  2. Go to Settings > Notifications.
  3. In Team channels, select Add channel.
  4. In Channel, select a channel or select New destination.
  5. Under App and database health, select Alert firing and Alert resolved.
  6. Select Add rule. If you are adding a new destination, select Add destination.

Orchard also sends alert events to personal notifications, in the App and database health category.

  1. Open the app.
  2. Go to Settings > Notifications.
  3. In Alerts, find the rule.
  4. Select Pause to stop the rule, or select the delete icon to remove it.

To start a paused rule, select Resume.

Pausing a rule clears its current state. The app page does not show these actions for project or organization rules. Use the CLI to change those rules.

Create a rule for a project or organization

Section titled “Create a rule for a project or organization”

A project rule applies to every app in the project. An organization rule applies to every app in the organization. The dashboard shows these rules on each app, marked From the project or From the organization, but you create them with the CLI.

To create a project rule:

orchard alert create --project <projectId> --metric memory_percent --threshold 90 --for 600

To create an organization rule, omit --project and --deployment:

orchard alert create --metric error_rate --threshold 5 --for 300
  • Each app, project, or organization can have 25 rules.
  • The period must be between 60 seconds and 24 hours. The dashboard offers periods from 1 minute to 1 hour.
  • Orchard evaluates a rule only for apps that are not stopped and that have a ready replica or a running status.