Skip to main content

Alert notifications

Use this page to send the platform's alerts to Slack or email.

⚠️ Alerts reach nobody by default — Prometheus evaluates about 36 alert rule groups, but Alertmanager discards every alert until you set up delivery. This includes the Watchdog alert that reports a dead Prometheus.

📝 Octostar's own Prometheus only — This page configures the Alertmanager that Octostar installs. If the install reports to an existing monitoring stack, configure delivery on that Alertmanager. See Installing on a shared cluster.

What you need​

  • A Slack incoming webhook, or an SMTP server.
  • Optional: an external heartbeat service (for example Healthchecks.io) for the Watchdog.

1. Set local-env.yaml​

monitoring:
  alerting:
    enabled: true
    slack:
      enabled: true
      webhookUrl: "https://hooks.slack.com/services/..."
      # criticalWebhookUrl: "https://hooks.slack.com/services/..."   # a second channel
    # email:
    #   enabled: true
    #   to: "ops@example.com"
    #   from: "octostar@example.com"
    #   smarthost: "smtp.example.com:587"
    #   authUsername: "..."
    #   authPassword: "..."
    watchdog:
      enabled: true
      heartbeatUrl: "https://hc-ping.com/<uuid>"

One Slack webhook posts to one channel. For a second channel, create a second webhook and set criticalWebhookUrl. Critical alerts then go there. Critical alerts repeat every hour in any case (severityRouting), with or without a second webhook. Email sends only critical alerts by default.

With watchdog.enabled: true, you must set heartbeatUrl. Without it, Alertmanager refuses to load its configuration.

Keep the webhooks and the SMTP password in local-env.yaml, never in defaults.yaml. An empty value in a later update does not erase a stored one.

2. Apply​

./bin/update-existing.sh --context <context>

3. Verify​

Send a test alert:

kubectl --context <context> -n octostar-main exec -it alertmanager-kube-prometheus-stack-alertmanager-0 \
  -c alertmanager -- amtool alert add \
  alertname=TestAlert severity=critical namespace=octostar-main \
  --annotation=summary="checking delivery"

The message must arrive in Slack or your mailbox.

📝 Expect real findings on first enable — The rules can report problems that already exist, such as a core pod that failed days ago or a node disk that fills up. Clear them before you tune the thresholds. A few noisy alerts, such as CPUThrottlingHigh, are muted by default. mutedAlerts in local-env.yaml changes that list. local-env.template.yaml lists every setting.

If it fails​

  • Alertmanager stays in ContainerCreating. The alertmanager-notifications Secret does not exist. Run ./bin/update-existing.sh so the preparation step runs.

  • Nothing arrives and the pod runs. Alertmanager keeps its old configuration when a reload fails. Look for errors:

    kubectl --context <context> -n octostar-main logs alertmanager-kube-prometheus-stack-alertmanager-0 \
      -c alertmanager | grep -i "error\|reload"

  • No errors, but nothing in Slack. The webhook can be revoked, or its channel archived. Post to it directly with curl.

  • An alert you expect never arrives. Check mutedAlerts and mutedAlertsInRuleNamespaces.

  • Node disk alerts work, volume alerts do not. On local-path storage, volume usage is not measured. See Storage and S3.