Problem
Marketers think in themes — "how is our security content doing?" — but analytics answers in URLs. A site with a few hundred pages has no useful answer at the page level, and hand-built taxonomies go stale the week after someone makes them.
The taxonomy had to build itself, stay stable as the site grew, and read like the business rather than like a keyword tool.
The pipeline
01 · Read
For each page: its title, meta description and body converted to Markdown (first fold first), plus the organisation's profile and ideal-customer profile.
02 · Tag
An LLM returns up to four visitor-intent topics, reusing the site's existing vocabulary so the taxonomy doesn't fork into near-duplicates; a primary and secondary topic picked by prominence in the title, meta and top headings; a 0–100 score for Awareness, Consideration, Conversion and Retention; and a neutral four-sentence summary.
03 · Cluster
Only new, unclustered topics go to a second pass that groups them into clusters named like the company's offerings. No "Misc" or "Other", every topic in exactly one cluster, and existing assignments are never overwritten, so clusters stay stable as pages are added. With only a handful of new topics, the model call is skipped.
04 · Sync
Postgres stays the system of record. Triggers on the topics table fire NOTIFY; a queue worker picks it up and syncs topics and page scores into ClickHouse.
05 · Target
Each cluster gets a pre-built audience segment when it's created, and topic and cluster affinity — including the topic a visitor arrived on — lands on the visitor's profile, so any segment can target it.
What it answers
- Cluster overview — Impact, Engagement and Conversion views, with a drill-down from cluster to topic to page.
- Funnel role — whether a topic attracts, persuades or closes, from the stage scores of the pages behind it.
- Topic synergy — which pairs of topics convert together, measured as lift over shared conversions.
- Influenced goals and audiences — which goals a topic moves and which segments engage with it.
- Page journeys — for any page, where visitors came from and went next (lag and lead over each session's ordered page path), where sessions end without a goal, and how deep into a journey the page usually sits.
Grounded recommendations
Every cluster, topic and page view ends in AI recommendations. They're retrieval-augmented, with no vector database: the retrieval key is the topic itself. Pages are already scored against every topic in ClickHouse, so a scored lookup beats a similarity search — cheaper, exact, and easy to explain.
- Retrieve the pages that score highest for the topic, with merged topics folded in, and their content as Markdown.
- Attach the metrics above — funnel role, synergy, goals, audiences — and have the model write against that payload only.
- Hardened later with a teammate: each page's Markdown is capped so the biggest topics don't overflow the context window, numbers the model quotes are swapped for the real value or stripped, empty analytics skip the paid call, truncated completions count as failures, and a fresh insight for the same range is reused instead of regenerated.
- Any recommendation goes to the agent in one click to be implemented.