What pages
Emergency alerts repeat until acknowledged. High alerts bypass Pushover’s
quiet hours. Normal alerts respect them.
What deliberately stays quiet
Routine pipeline events, every successful API call, and — importantly —access_denied. Refused requests are frequent by design; a channel
that fires on each one gets muted by its owner within a week, and then the
alerts that matter go unread too.
notification_failed never raises an alert of its own, so a broken channel
cannot feed itself.
The gate-closure watchdog
On every orchestration pass the service checks the day-ahead delivery day: if it is not submitted and accepted, and gate closure is inside the warning window, it raises an emergency alert. Two details that matter:- It runs regardless of whether anything changed. Change detection drives the pipeline, but “nothing changed” is exactly the symptom of a day nobody delivered data for.
- A rejected submission counts as not submitted. The submit step having run is not the same fact as the schedules being in.
Channels
Three, and any combination of them can be active. Each has a threshold: the priority at which it starts caring. That is what keeps the loud channels for the alerts that deserve them.
Channels are independent: one failing does not stop the others, and the
recorded failure names which one. Every credential below is editable from
the console, so the on-call target moves without a deployment, and the
secrets are masked when the configuration is read back.
Slack
Create an incoming webhook for the channel. The URL is the credential — anyone holding it can post — so it is treated as a secret.Pushover
Create an application at pushover.net for the application token, and take the user key from your dashboard. Both are needed before the channel exists.
Use a group key, not a personal one: an alert that only reaches someone
on holiday has not been delivered. With a group, an emergency alert stops
repeating as soon as any member acknowledges it.
Pushover delivers only to its own clients — iOS, Android and a desktop
browser client — each a separate one-time purchase per person.
SMS
SMS is the channel that survives a dead data connection and an uninstalled app. It costs per message, so it defaults to emergencies only.
Germany and Switzerland both allow alphanumeric sender IDs (e.g.
NLEAD), which avoids renting a number — the trade-off is that the
recipient cannot reply. All four values are required before the channel is
built: a half-configured channel that fails on every alert is worse than no
channel.
Bodies are truncated so one alert cannot become a long chain of billed
segments, and a bad number does not stop delivery to the others.
iMessage is not offered, deliberately. Apple publishes no API for it;
the only routes are a Mac running AppleScript or a third-party bridge,
both of which mean maintaining a Mac as production infrastructure that
can stop delivering silently. SMS reaches the same iPhone.
Adding another channel
Notifier is a Protocol with one method. Telegram, PagerDuty or a
webhook of your own is a class that POSTs a different payload, plus one
entry in build_notifier. The rules in alert_for do not change, because
they describe what is worth an alert, not where it goes.
What this does not cover
Alerting runs inside the service, so it cannot report that the service is down. If the container is not running, neither is the code that would tell you. That needs a watchdog outside the app: an Azure Monitor alert on the orchestrator job failing or not having run, and an availability check against/health. Log Analytics is already collecting the data — the alert
rules are the missing piece, and they belong to the infrastructure, not
here.