Skip to main content
The pipeline is split by scope: the customer only ever touches the ingest endpoint (scope external); everything downstream runs behind internal endpoints (scope process), normally driven by the orchestrator.

​
The steps

​
Ingest — POST /v1/forecasts (external)

Accepts production/consumption series per control area — or per market location (MaLo) with a malo_mapping table. Validation:
  • correct interval count for the delivery day (96/92/100, DST-aware),
  • every series complete (no gaps — a short series is rejected),
  • only active control areas (ACTIVE_CONTROL_AREAS); a subset is fine, missing areas are treated as zero.
The forecast is persisted with a content hash; nothing is processed inline. Re-posting a corrected forecast just updates the stored input — the orchestrator notices the changed hash on its next run.

​
Aggregate — POST /v1/process/delivery-days/{day}/aggregate

Pure functions compute net, open_position and total_open (algorithm) and persist them.

​
Publish — POST /v1/process/delivery-days/{day}/publish-open-position

The total open position is rendered as a CSV and written to the trader’s sFTP drop directory (atomically: temp name, then rename). This is the volume the trader executes on the EPEX day-ahead auction.

​
Submit — POST /v1/process/delivery-days/{day}/submit-schedules

Builds one schedule per balancing group — production forecast, consumption forecast, and all trades aggregated per direction and counterparty — and submits them to Engrate, one call per TSO per delivery date (per Engrate’s API model). The executed EPEX volume can be injected in the request body; it defaults to the requested open position.

​
Orchestrate — POST /v1/process/orchestrate

Called regularly by a timer (a Container Apps cron job in Azure). For every known delivery day that is not in the past, it compares the current input hash with the hash each step last ran on, and re-runs exactly the stale steps in order. No change → no action; the call is idempotent and cheap, so a tight schedule is fine.

​
Backup series for the trader

The trader contract wants a CSV for every day up to four days ahead (FORECAST_HORIZON_DAYS), while the customer sends only the next day. With the backup-csv agent switched on (it replaced AUTO_BACKUP_CSV) the orchestrator fills every day of the horizon that holds no forecast with a backup series carried forward from the customer’s own forecast:
  • Source: the customer’s forecast for the same weekday one week earlier (load shapes differ by weekday); without one, the newest customer forecast of the seven days before. While customers are in supply on the target day, a source day without any is passed over for the newest one with customers, so the first week of supply does not carry the zero forecasts from before the ramp-up. Backups never feed other backups.
  • Customer ramp-up: the volume follows the confirmed customers from the ramp-up snapshot (POST /v1/process/kundenhochlauf/quellstand). Per control area the carried series is scaled by the customers in supply on the target day against those on the source day; an area whose first customers start in between gets the source day’s average per-customer shape. Customers the source day cannot give a shape for (no customer in supply anywhere yet) stay out of the file and raise forecast_backup_unreferenced.
  • Kept current: backups from D+2 on are rebuilt on every orchestrator run and stored only when the result changed — a grown ramp-up or a newer source forecast re-publishes the trader’s file, an unchanged one writes nothing. The next day’s backup is made if missing but never rebuilt: the trader may have collected that file already, and a new version would re-arm the gate-closure alarm.
  • Ramp-up snapshot: the counts arrive with the read-only report key, so whoever holds that key influences the volumes in the backup files.
  • No race with the customer: the forecast input, the state and the versions of a day are written under one lock, and a backup is stored only when the day holds no forecast or a backup. A customer forecast that arrives while a backup is being rebuilt is never overwritten.
  • Clock changes: values are matched by local time, so a backup for the fall-back day carries the 02:00–03:00 hour twice and one for the spring-forward day drops it.
  • CSV only: a backup is aggregated and published to the trader’s sFTP like any day, but it is never submitted as schedules — not by the orchestrator and not by a manual submit-schedules call (409). A manual call also gets 409 while the positions still come from an earlier input (a backup’s, until the customer’s forecast is aggregated).
  • sFTP outage: a backup day from D+2 on whose CSV cannot be written raises backup_publish_failed once per version and error — a warning, not the orchestration_error page a real day gets — and is retried every run. After the first failed write of a run, the remaining backup days wait for the next run instead of each trying the server (every connect and file operation is bounded to 15 seconds). Tomorrow’s backup is the trader’s next pickup, so its failure pages like a real day’s. The gate-closure watchdog runs for every day, also one whose steps just failed.
  • Replaced on arrival: when the customer posts a forecast for the day, it supersedes the backup; the hash changes and the orchestrator re-publishes and submits the real data. A generated forecast (the forecast-generation agent) also replaces a backup.
  • Not in the customer API: GET /v1/forecasts does not list backup days and GET /v1/forecasts/{day} answers 404 for them, so a client that skips days already present still posts them. The Slack log does show them: the CSV entries are marked (backup series), and the warnings (forecast_backup_skipped, forecast_backup_unreferenced, backup_publish_failed) appear there; the per-build forecast_backup_created entries do not.
  • Watchdog: a day that reaches the gate with only its backup is not submitted, so the gate-closure warning fires and says the customer’s forecast has not arrived. Without a customer forecast in the last seven days the orchestrator warns once per day (forecast_backup_skipped).
The day’s state.json carries source: backup; the console’s day table shows it as backup · CSV only. Next to the active forecast_input.json the audit trail keeps the latest version of every quality (forecast_customer.json, forecast_generated.json, forecast_backup.json, with forecast_versions.json for when and how each was made), so the console’s Prognosen tab can compare a backup with the customer forecast that replaced it.

​
State machine per delivery day

Everything lives in the audit trail (state.json per day): A step is stale iff its hash differs from input_hash. Manual step calls update the same state, so manual and orchestrated operation mix safely.