> ## Documentation Index
> Fetch the complete documentation index at: https://docs.energy.nlead.ch/llms.txt
> Use this file to discover all available pages before exploring further.
# Alerting
> What wakes a human, how it reaches them, and what deliberately stays quiet
The event log records everything the service does. Alerting decides which
of it is worth interrupting someone for, and sends those to Slack, a phone,
or an SMS — each channel from the priority it cares about upwards.
The failure this exists for is not a loud one. A rejected submission is
visible in the console; a delivery day that quietly never got submitted is
not, and it becomes expensive at 14:30.
## What pages
| Event | Priority | Why |
| ----------------------------------------------- | --------- | ------------------------------------------------------------------ |
| Submission **not accepted** | Emergency | Schedules are not in. Carries the platform's own rejection reason. |
| **Gate closure approaching**, day not submitted | Emergency | The expensive silence. See below. |
| Accepted but **unbalanced** | High | The submission went through, but a group does not net to zero. |
| **Orchestration error** | High | The pipeline raised; steps may be half-done. |
| **Forecast generation skipped** | High | Auto-generation could not produce tomorrow's forecast. |
| Running **unauthenticated** | High | `ALLOW_UNAUTHENTICATED` is set on a running service. |
| **Orchestrator paused** | Normal | Someone stopped the service from the console. |
Emergency alerts repeat until acknowledged. High alerts bypass Pushover's
quiet hours. Normal alerts respect them.
### What deliberately stays quiet
Routine pipeline events, every successful API call, and — importantly —
**`access_denied`**. Refused requests are frequent by design; a channel
that fires on each one gets muted by its owner within a week, and then the
alerts that matter go unread too.
`notification_failed` never raises an alert of its own, so a broken channel
cannot feed itself.
## The gate-closure watchdog
On every orchestration pass the service checks the day-ahead delivery day:
if it is not submitted **and accepted**, and gate closure is inside the
warning window, it raises an emergency alert.
Two details that matter:
* It runs **regardless of whether anything changed**. Change detection
drives the pipeline, but "nothing changed" is exactly the symptom of a
day nobody delivered data for.
* A **rejected** submission counts as not submitted. The submit step having
run is not the same fact as the schedules being in.
It fires once per version of the input, so a 15-minute orchestrator does
not send the same alarm four times an hour. Correcting the forecast arms it
again — new input, new warning if that one also fails to land.
| Variable | Default | Meaning |
| --------------------------- | ------- | ------------------------------------- |
| `GATE_CLOSURE_LOCAL_TIME` | `14:30` | Deadline in local market time |
| `GATE_CLOSURE_WARN_MINUTES` | `60` | How long before it to raise the alarm |
`14:30` is the German day-ahead deadline. Confirm it against the
balancing-group contract before relying on the lead time — a warning that
arrives after the door has closed is only a post-mortem.
## Channels
Three, and any combination of them can be active. Each has a **threshold**:
the priority at which it starts caring. That is what keeps the loud
channels for the alerts that deserve them.
| Channel | Default threshold | Gets |
| ------------ | ----------------- | -------------------------------------- |
| **Slack** | everything (`-2`) | The full stream, for the team to see |
| **Pushover** | normal (`0`) | What is worth interrupting someone for |
| **SMS** | emergency (`2`) | Only what must not be missed |
Channels are independent: one failing does not stop the others, and the
recorded failure names which one. Every credential below is editable from
the console, so the on-call target moves without a deployment, and the
secrets are masked when the configuration is read back.
### Slack
Create an [incoming webhook](https://api.slack.com/messaging/webhooks) for
the channel. The URL is the credential — anyone holding it can post — so it
is treated as a secret.
| Variable | Default | Meaning |
| -------------------- | ------- | ---------------------------- |
| `SLACK_WEBHOOK_URL` | unset | The webhook |
| `SLACK_MIN_PRIORITY` | `-2` | Raise to quieten the channel |
### Pushover
Create an application at [pushover.net](https://pushover.net/apps/build)
for the **application token**, and take the **user key** from your
dashboard. Both are needed before the channel exists.
| Variable | Default | Meaning |
| ------------------------- | ------- | ---------------------------------- |
| `PUSHOVER_APP_TOKEN` | unset | Application token |
| `PUSHOVER_USER_KEY` | unset | User or **group** key |
| `PUSHOVER_DEVICE` | unset | Limit to one device; omit for all |
| `PUSHOVER_RETRY_SECONDS` | `300` | Emergency alerts repeat this often |
| `PUSHOVER_EXPIRE_SECONDS` | `3600` | …and give up after this |
| `PUSHOVER_MIN_PRIORITY` | `0` | Threshold |
Use a **group key**, not a personal one: an alert that only reaches someone
on holiday has not been delivered. With a group, an emergency alert stops
repeating as soon as *any* member acknowledges it.
Pushover delivers only to its own clients — iOS, Android and a desktop
browser client — each a separate one-time purchase per person.
### SMS
SMS is the channel that survives a dead data connection and an uninstalled
app. It costs per message, so it defaults to emergencies only.
| Variable | Default | Meaning |
| ------------------------------------------ | ------- | --------------------------------------------- |
| `TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN` | unset | Twilio credentials |
| `TWILIO_FROM` | unset | A Twilio number, or an alphanumeric sender ID |
| `TWILIO_TO` | unset | Comma-separated recipients, E.164 |
| `SMS_MIN_PRIORITY` | `2` | Threshold |
Germany and Switzerland both allow **alphanumeric sender IDs** (e.g.
`NLEAD`), which avoids renting a number — the trade-off is that the
recipient cannot reply. All four values are required before the channel is
built: a half-configured channel that fails on every alert is worse than no
channel.
Bodies are truncated so one alert cannot become a long chain of billed
segments, and a bad number does not stop delivery to the others.
**iMessage is not offered, deliberately.** Apple publishes no API for it;
the only routes are a Mac running AppleScript or a third-party bridge,
both of which mean maintaining a Mac as production infrastructure that
can stop delivering silently. SMS reaches the same iPhone.
## Adding another channel
`Notifier` is a Protocol with one method. Telegram, PagerDuty or a
webhook of your own is a class that POSTs a different payload, plus one
entry in `build_notifier`. The rules in `alert_for` do not change, because
they describe *what is worth an alert*, not where it goes.
## What this does not cover
Alerting runs inside the service, so **it cannot report that the service is
down**. If the container is not running, neither is the code that would
tell you.
That needs a watchdog outside the app: an Azure Monitor alert on the
orchestrator job failing or not having run, and an availability check
against `/health`. Log Analytics is already collecting the data — the alert
rules are the missing piece, and they belong to the infrastructure, not
here.