Data sources

Collecting

An adapter exists and the scheduler has streams expected from it.

Read live

Called at request time and never copied. It answers a wide date range cheaply, so a stored copy would buy nothing, and a page that needs it is unavailable for exactly as long as the publisher is.

Planned

Registered so the work has a landing place. No adapter yet.

• Regulatory feeds: the statutory publication points, and the system operators’ own. Zone-level day-ahead prices, actual and forecast load, generation by production type, installed capacity, outages, and the cash-out price a position is actually settled against. They are the backbone of the reconstruction, because they are what carries fundamentals rather than just outcomes, and because they are the numbers the market itself is reported against.
• Exchange feeds: market results. Clearing prices and traded volumes per auction, and aggregated supply and demand curves: across 13 clusters and four auctions on the coupled Continental feed, and per zone on Italy’s own, which answers an arbitrary past delivery day and is therefore read live rather than captured. The curves are the only public view of an order book anywhere in this system, and Great Britain, which clears its own auctions, has no published curve at all.
• Reference feeds: secondary compilations of prices, generation mix, load, forecasts and cross-border flows. Used as a cross-check, and as cover for areas a primary feed does not reach. A reference feed is keyed by country rather than by bidding zone, so for a country split into several zones it describes the country. The panel says so, because it changes what the number means.
• FX reference: daily EUR/GBP reference rates, so a GB price and a Continental price can be compared on the delivery day’s own rate rather than today’s.

Which upstream answers a given panel is not settled. A feed can be withdrawn, re-licensed, or replaced between one release and the next, and a page that has printed a publisher’s name to customers has made a claim about a commercial relationship that outlives the deployment. It gets quoted back, screenshotted and indexed.

The role is the part that matters for reading a number, and the part that survives a source swap. Whether a reading came from a regulatory publication or a secondary compilation changes how much weight it carries; which body published it does not. Inside the product, the collection log names every feed by name. It has to, because “which source has not run” cannot be answered by a role, and it is restricted to the operators who act on it.

Every upstream response is written to object storage before anything parses it. Parsers change, upstream formats change, and a bug found six months from now is only fixable if the bytes that produced the bad row still exist.

It also matters for a dispute. A finding that cannot be traced back to the exact document the market published is a finding that cannot be defended, and an archive that is written only after parsing succeeds is an archive missing precisely the days that went wrong.

The panel says so and degrades. It does not show yesterday’s number formatted to look like today’s.

The distinction the system insists on is between kinds of empty: data that was never published, data still being collected, data that is genuinely missing, a window that has expired, a source that is unavailable right now, and a stream that was retired. Those are six different situations and only one of them is a fault. Collapsing them into a blank cell is how a pipeline stops collecting for weeks without anyone noticing.