90.5%
Preferred over a no-template baseline in the best model, blinded pairwise, held-out judge
Institutional finance expects a specific register: conclusion-first, figures woven into prose, calibrated hedging. We distill that register automatically from ~12,800 research documents into a single system-prompt style template, then measure it in blinded, open-book A/B tests. It wins decisively; and the gains are substantive, not cosmetic.
Both arms answer the same question from the same source excerpts, open-book, so facts are held constant. The only thing that changes is whether the corpus-derived style template is in the system prompt.
A neutral but prose-capable system prompt, explicitly told to write well-structured prose, not bullet lists. So we measure answer quality, not a prose-vs-bullets formatting artifact.
The same prompt plus a single corpus-derived template: 8 style sections (sentence construction, data integration, hedging, framing…) and a domain layer of key terms, formulas, and relationships.
Blinded pairwise preference with randomized answer order, judged by a held-out model over decisive (non-tie) pairs. The template wins decisively.
Because the answers and the primary judge are all Claude, self-preference is a real risk. So the same answers were re-scored by three independent non-Anthropic judges. Every judge favors the template.
Should each topic get its own template? We built 89 per-topic templates and tested them with oracle routing (the item's true topic, an upper bound that removes routing error). Even so, specialization does not beat a single broad template.
A three-stage pipeline, entirely corpus-derived, no hand-written style rules.
~12,800 research documents from all providers, 2024-onward, ~12 regions. Provider-balanced and asset-class-diverse: roughly two-thirds equity, one-third non-equity (credit, commodities, macro/FX).
A model reads each document and extracts a topic, region, and a style template (sentence construction, data integration, tone/hedging, etc) plus a layer of key terms, formulas, and relationships.
Per-document templates are merged hierarchically into one global template: 8 style sections plus 3 consolidated domain sections spanning equity, credit, rates/FX, and commodity conventions.
Blinded pairwise preference with randomized order, plus a separate key-fact recall metric. Open-book throughout, so facts are held constant and only the style guidance varies.