> ## Documentation Index
> Fetch the complete documentation index at: https://docs.energy.nlead.ch/llms.txt
> Use this file to discover all available pages before exploring further.

# The Day-Ahead Pipeline

> External ingest, internal process steps, and the change-driven orchestrator

The pipeline is split by **scope**: the customer only ever touches the
ingest endpoint (scope *external*); everything downstream runs behind
internal endpoints (scope *process*), normally driven by the orchestrator.

```mermaid theme={null}
sequenceDiagram
    participant C as Customer
    participant B as Bridge
    participant O as Timer
    participant S as sFTP / Trader
    participant E as Engrate

    C->>B: POST /v1/forecasts (external)
    B-->>C: ack + input hash
    loop every 15 min
        O->>B: POST /v1/process/orchestrate (process)
        alt input changed
            Note over B: aggregate<br/>net, open, total_open
            B->>S: publish open-position CSV
            B->>E: submit schedules (one per TSO)
            B-->>O: actions per delivery day
        else no change
            B-->>O: no-op
        end
    end
    S->>S: trader executes on EPEX day-ahead
```

## The steps

### Ingest — `POST /v1/forecasts` *(external)*

Accepts production/consumption series per control area — or per market
location (MaLo) with a `malo_mapping` table. Validation:

* correct interval count for the delivery day (96/92/100, DST-aware),
* every series complete (no gaps — a short series is rejected),
* only active control areas (`ACTIVE_CONTROL_AREAS`); a subset is fine,
  missing areas are treated as zero.

The forecast is persisted with a content hash; **nothing is processed
inline**. Re-posting a corrected forecast just updates the stored input —
the orchestrator notices the changed hash on its next run.

### Aggregate — `POST /v1/process/delivery-days/{day}/aggregate`

Pure functions compute `net`, `open_position` and `total_open`
([algorithm](/concepts/algorithm)) and persist them.

### Publish — `POST /v1/process/delivery-days/{day}/publish-open-position`

The total open position is rendered as a CSV and written to the trader's
sFTP drop directory (atomically: temp name, then rename). This is the volume
the trader executes on the EPEX day-ahead auction.

### Submit — `POST /v1/process/delivery-days/{day}/submit-schedules`

Builds one schedule per balancing group — production forecast, consumption
forecast, and all trades aggregated per direction and counterparty — and
submits them to Engrate, **one call per TSO per delivery date**
(per Engrate's API model). The executed EPEX volume can be injected in the
request body; it defaults to the requested open position.

### Orchestrate — `POST /v1/process/orchestrate`

Called regularly by a timer (a Container Apps cron job in Azure). For every
known delivery day that is not in the past, it compares the current input
hash with the hash each step last ran on, and re-runs **exactly the stale
steps** in order. No change → no action; the call is idempotent and cheap,
so a tight schedule is fine.

## State machine per delivery day

Everything lives in the audit trail (`state.json` per day):

| Field             | Meaning                                       |
| ----------------- | --------------------------------------------- |
| `input_hash`      | Hash of the currently stored forecast         |
| `aggregated_hash` | Input hash the aggregation last ran on        |
| `published_hash`  | Input hash the CSV publication last ran on    |
| `submitted_hash`  | Input hash the Engrate submission last ran on |

A step is stale iff its hash differs from `input_hash`. Manual step calls
update the same state, so manual and orchestrated operation mix safely.