‹ All views

System architecture · WattIQ EPM Decision Agent

How a planner's question becomes a cited, validated decision brief

One map of the whole pipeline — inputs, the grounded analysis of World Bank–ESMAP EPM outputs, the governed LLM reasoning, the browser and export layers, the self-refreshing learning loop, and the platform that ships and stores it. Follow the four coloured circuits to trace any path end to end.

Runtime request — the answer path Grounded EPM data & provenance Learning loop — observe → validate Platform & delivery
9 Python modules 11 EPM data tools 8 browser layers 3-tier trust model durable Azure Blob store git-tag versioning + rollback

Circuit 1 — runtime

The path of a single question

Top to bottom: what happens between a planner typing a question and a decision brief rendering in their browser. The reasoning core and the analysis layer run as a loop (step 3) — the model may call data tools several times, and every figure it later states is re-derived and cited from the sheets.

01Inputwhat the user brings
User query
A natural-language question typed in the workspace search bar.
Role / audience
Planner · Ministry · Finance · Regulator · Power pool — sets the answer format & framing.
Scenario focus
Optional: pin one of the 5 SG scenarios to spotlight.
POST /api/chat/stream · role + scenario appended
02Edge & access gateRender web service · FastAPI
main.py · gate()
Auth middleware: HMAC session cookie or HTTP Basic (survives the corporate proxy that strips headers). Redirects HTML, 401s the API.
Rate limit
Per-client cap on /api/chat (default 20 / 60s).
Cache control
no-cache on the shell + scripts so a redeploy reaches testers immediately.
authenticated request → agent.answer_question_stream()
03

Reasoning core

agent.py · llm.py
system_promptThe governed “power-systems modeler” persona — Trust Tiers, the standard decision flow, data guardrails, security rules.
recall_blockSimilar past insights injected as Tier 2 — provisional, non-overriding (steers approach; never supplies a number).
LLMOpenAI API · gpt-5.5 (OPENAI_BASE_URL + MODEL) reasons, decides which tools to call, then writes the brief.
tool-call
loop cites [n]
03′

Analysis & provenance

data_tools.py · provenance.py
11 EPM toolsRead the workbooks — cost, generation mix, capacity, emissions, reliability, reserves, supply, balance, scenario summary.
SourceLedgerRecords the workbook · sheet · row behind every figure and hands the model a numbered ledger.
[n] citationsThe model may only assert a number that resolves to a ledger entry — “provenance or nothing.”
answer text + chart specs + source ledger
04Assemblyserver-side, deterministic
skills.py + catalog.json
The skill registry routes the output shape (brief · face-off · diagnostic · fiscal · regulatory · pool). One manifest generates both the prompt and the renderer.
chart builder
Each tool matrix becomes a chart spec server-side, so the figures on screen are the figures from the sheets.
payload
{ answer, charts, sources } streamed to the browser (chart first, then the narrative).
Server-Sent Events → the browser
05Display layerpoc/web · vanilla JS + ECharts
app.js
SPA shell, role picker, submits the query, consumes the SSE stream, holds state.
turn.js
Renders each answer as an aligned grid of decision cards (### steps → icon + heading + body), stat chips on figures, sources.
charts.js
ECharts option builders: spotlight bars/lines, baseline-delta / share / contrast, small-multiples, heatmap, donut; the theater story render.
theater.js
Full-screen slide deck — one finding per slide, the hero number = the chart's pinned figure, animated spotlight, keyboard nav.
skill_engine.js
Renders the skill-specific answer structure (tables, matrices, flash cards).
libs
ECharts (charts), marked+DOMPurify (safe markdown).
user clicks Export
06Export layerclient-side, branded
export.js
Per-answer: PNG chart, Markdown, portable self-contained HTML, PowerPoint (pptxgenjs).
export_brand.js
Branded DOCX & PDF (jsPDF): Watt iQ cover, country header, copyright, pagination — charts rendered full-width, spotlit.
report_export.js
The multi-section Country Report deck in every format.

Circuit 2 — learning

The learning loop — observe, recall, validate, promote

Every question and every served answer is captured, made durable in Azure, and fed back as provisional guidance. Nothing is trusted until it is corroborated by ≥2 agreeing answers and a human approves it — the model proposes, a human disposes.

Observe

Every question is logged as raw demand; every answer is distilled into a structured insight — skill, tools, sheets, figures-as-claims, recommendation, admitted gaps.

question_log.py · insights.py
Store

Both streams append through one portable interface to the durable store — two blobs in the Azure learning container.

insight_store.py → Azure Blob
Recall

A similar new question pulls prior insights back into the prompt as Tier 2 — it steers approach, and is barred from supplying any figure.

insights.recall_block()
Validate

The corroboration gate labels each pattern validated (≥2 agree), conflicting, or provisional — a human review surface.

gen_learning_digest.py → learning-digest.md
Promote

A validated pattern is proposed as a Tier-2 heuristic; a human maintainer approves it into Tier 1 (the system prompt). The loop never edits Tier 0/1 itself.

propose-heuristic → human → Tier 1
Recall feeds straight back into the Reasoning core (step 3) · a scheduled GitHub Action refreshes both docs from /api/question-log + /api/insight-log.

The checks — why an answer can be trusted

Governance: three tiers and a provenance rule

Tier 0 · immutable

Core directives. Identity, the trace-before-trust chain, data guardrails, security rules. Only a human editing the prompt file can change them — the loop never can.

Tier 1 · human-curated

Approved knowledge. Scenario map, EPM model reference, workbook map, best-practice standards. Changed only through a human-approved diff.

Tier 2 · provisional

Learned heuristics. The only region the loop may draft into — advice, lower weight, never overriding Tier 0/1.

Provenance or nothing

Every figure re-derived each run from the source sheets and cited [n]. A recalled number is context, never evidence — poisoning-resistant by design.

Circuit 3 — platform & delivery

How it ships, runs, and stores state

Source lives on GitHub; a commit to main builds the Docker image and auto-deploys on Render. The learning corpus lives outside the app in Azure Blob, so the container stays stateless and the state is portable to Azure/Foundry.

GitHub

Source of truth & release control.
  • Annotated git tags = releases & exact rollback points
  • Actions: poc-ci tests · scheduled learning-docs refresh
  • Living docs: CHANGELOG, ADRs, question bank, digest

Dockerfile

Reproducible build.
  • python:3.11-slim + requirements.txt
  • Copies poc/, the data/ EPM workbooks, the system prompt
  • Runs uvicorn; .dockerignore keeps .env out

Render

Hosts the live service.
  • Docker web service · Starter · always-on
  • Auto-deploy on push; buildFilter ignores docs/**
  • Deploy history = one-click rollback · /api/health probe

Azure Blob — the durable learning store

learning container · append blobs insights.jsonl — what was served questions.jsonl — raw demand portable: jsonl → blob → cosmos → Foundry (one env var)

Secrets & keys — Render env store

OPENAI_API_KEY · OPENAI_BASE_URL · MODEL (gpt-5.5) APP_USER · APP_PASSWORD (access gate) AZURE_STORAGE_CONNECTION_STRING · INSIGHT_STORE never in the repo — local .env is git- & docker-ignored

Reference

Every system and its role

The walk-through list — group by group, what each part is and does.

Inputs & grounding data

EPM workbooksWorld Bank–ESMAP Electricity Planning Model outputs (data/*.xlsx) — the single source of truth for every figure.
User queryThe planner's question, plus optional role and scenario focus.
catalog.jsonThe skill & audience manifest — generates both the prompt's skill section and the frontend renderer.

Python backend — poc/app

main.pyFastAPI app: routes, the auth gate, rate limiting, cache headers, the log endpoints.
agent.pyThe reasoning loop — builds the prompt, runs the LLM tool-call rounds, keeps the source ledger, streams the answer.
llm.pyThin client to the model endpoint — OpenAI API · gpt-5.5.
data_tools.pyThe 11 EPM tools that read the workbooks into comparison matrices.
provenance.pySourceLedger — ties every figure to workbook · sheet · row and numbers it [n].
skills.pyLoads the skill catalog that routes each answer's format.
insights.pyMeta-learning: extract an insight, recall similar ones, aggregate patterns.
insight_store.pyOne interface, three backends (jsonl / Azure blob / cosmos) — the portability seam.
question_log.pyThe raw-demand stream, written through the same store.

Browser — poc/web

app.jsSPA shell, role picker, streaming, state.
turn.jsAnswer rendering — decision cards, stat chips, sources.
charts.jsAll ECharts builders + the theater story render + palette & provenance-safe formatting.
theater.jsThe projected slide deck with per-slide spotlight.
skill_engine.jsSkill-specific answer structures.
export*.js / report*.jsPNG · MD · HTML · PPTX · branded DOCX & PDF · Country Report.

Learning & living docs

gen_question_bank.pyTurns the demand log into a dated, categorised, by-role question bank.
gen_learning_digest.pyPairs each question with what was served + a validation status.
gen_changelog.pyRebuilds the CHANGELOG from the annotated git tags.
GitHub Action + hookRefresh the living docs on a schedule / on commit — documentation-as-code.
docs/adr/*Architecture Decision Records — the durable “why”.

Platform & delivery

GitHubSource, tag-based releases & rollback, CI, Actions.
DockerfileReproducible image; bakes in code, workbooks, prompt.
RenderLive Docker host; auto-deploy, health check, deploy-history rollback.
Azure BlobDurable, portable learning corpus outside the app.
system_prompt…mdThe governance itself — Tiers, decision flow, guardrails, the learning-loop rules.