Intelligence Power per Dollar

I was thinking while using codex this weekend that not every token is equally as valuable as any other token. Much of the metrics you see are related to token generation rate or cost per token but that is only a partial picture. When I use a model I care about how much useful intelligence output I get for how much it costs and how long it takes to achieve the desired utility. Since benchmark accuracies, API price, and tokens per second are almost always documented, I was thinking how come none of them alone answers the question I actually want:

How much useful token intelligence power am I buying per dollar?

Below I am proposing a kind of FOM (Figure of Merit) that compares models on quality, speed, and cost together. I have no idea how useful this will be or how well my definitions for "token intelligence" will hold up. But I think it is a useful thought experiment. Again, the central idea is that not all tokens are equally valuable, which is indeed strongly tied to the model capabilities. In my view, the natural unit for combining value with time and money is what I would call intelligence action, by analogy with physical action and information-theoretic memory.

Why benchmarks, price, and speed are not enough

Three quantities dominate public comparisons:

Quantity	What it measures	What it misses
Benchmark score	Task accuracy on fixed evals	Dollar cost and latency
Cost per token	API price	Whether the token is any good
Throughput (token/s)	Generation speed	Quality of each token

A frontier model can score a few points higher yet cost an order of magnitude more. A cheap model can be fast yet unreliable on the tasks you care about. Leaderboards such as the Artificial Analysis Intelligence Index already compress many evals into one score, and several sites rank "value" as intelligence divided by price, including WhatLLM.org and Analytical Insider. Those are useful, but they still omit time: two models with the same intelligence-per-dollar can feel very different if one streams at 250 token/s and the other at 45 token/s.³

I want one scalar that rewards quality and thrift together, while keeping the pieces visible, including speed as an explicit tradeoff rather than something the score silently maximizes.

Naming by analogy

My take

I do not know if this is a good take/analogy but this is how I am naming/coining terms. There might already be a term for this but I do not know what it is.

In physics, action scales as energy × time. In information theory, information capacity over time scales as bit × second (I believe this is sometimes also called memory).

Language models output tokens, not joules or bits although one could think about tokens in these terms :wink:. By the same pattern, intelligence action scales as token × second, and its time-rate is what I call intelligence power (token / second), the quantity the figure of merit below actually prices out. This is just nomenclature I choose it is not a properly derived claim. The analogy is intentionally shallow, I'm just borrowing the product-of-content-and-time structure without importing the rest. The payoff is trying to gain meaningful readable units.

Token Intelligence Value Factor (TIVF)

I'm now going to describe what I call the Token Intelligence Value Factor (TIVF). It is the intelligence content of one generated token, expressed in reference-equivalent tokens. To keep units honest, I'll distinguish two kinds of token throughout: a raw-token is the physical unit you are actually billed for and that throughput $R$ and price $\tilde{c}_\mathrm{out}$ are measured in, while a ref-token is the intelligence-equivalent unit, i.e. one raw-token emitted by the reference model. So one raw-token from model $m$ is worth $\mathrm{TIVF}(m)$ ref-tokens from a fixed reference model $m_\mathrm{ref}$, and TIVF carries the unit ref-token/raw-token.

I'm going to define this as

$$ \begin{equation} \mathrm{TIVF}(m_\mathrm{ref}) \equiv 1\ \text{ref-token}/\text{raw-token}. \label{eq:tivf_def} \end{equation} $$

So TIVF is not dimensionless, its has the unit ref-token/raw-token², but importantly it is a token whose "intelligence" has been rescaled. A model with $\mathrm{TIVF}=2$ produces tokens that count double toward intelligence action relative to a defined reference model.

Capability spectrum

This is a hard one to define because intelligence is not one number. A model can score well on knowledge yet fail on agentic terminal work, or ace GPQA yet stumble on SWE-bench Pro. To try and approximate a broad spectrum, I'll assign each model six public scores $S_k(m)$, on a 0--100 scale and $k$ is the score axis, each measured at some thinking effort, $e$, if supported:

Axis $k$	Benchmark	What it probes
Knowledge	MMLU-Pro [1]	Broad multi-subject knowledge
Science	GPQA Diamond [2]	Graduate science Q&A
Coding	SWE-bench Verified [3]	Real GitHub issue resolution
Hard coding	SWE-bench Pro	Harder multi-language software engineering
Reasoning	Humanity's Last Exam w/ tools	Multidisciplinary expert reasoning
Agentic	Terminal-Bench 2.x	Shell/terminal agent coordination

I am trying to mirror the spirit of composite indices that are shown in Artificial Analysis Intelligence Index but keeps each axis visible instead of collapsing them.

Token intelligence rescaling

For each score axis $k$, I'll normalize to the reference model $m_\mathrm{ref}$, which will be GPT-4o mini ¹, and apply a mild superlinear exponent $\gamma=1.2$. The thinking is this makes sure that when more performant models do especially well on the hardest benchmarks, those factors still show up, even if many models already get top scores on easier. What We get is something like

$$ \begin{equation} \mathrm{TIVF}_k(m,e) = \left(\frac{S_k(m,e)}{S_k(m_\mathrm{ref})}\right)^{\gamma}. \label{eq:tivf_domain} \end{equation} $$

$S$ is the score of the model $m$ on the axis $k$. The reference model anchors every axis: $\mathrm{TIVF}_k(m_\mathrm{ref})=1$ for all $k$. The score ratio $S_k(m,e)/S_k(m_\mathrm{ref})$ is itself dimensionless; the ref-token base scale is injected only through the normalization $\mathrm{TIVF}(m_\mathrm{ref})\equiv 1\ \text{ref-token}/\text{raw-token}$ from eq. \ref{eq:tivf_def}.

The choice for GPT-4o mini as reference model is because it seems to be the most-used model across stacks. Although it might be that Anthropic models lead enterprise API usage.

In order to prevent a single strong axis to dominate/compensate for the weaker ones, we can take a weighted geometric mean of the axes:

$$ \begin{equation} \boxed{ \begin{aligned} \mathrm{TIVF}(m,e) &= \kappa(m)\cdot \exp\left(\sum_k w_k \ln \mathrm{TIVF}_k(m,e)\right) \\ &= \kappa(m)\cdot\prod_k \mathrm{TIVF}_k(m,e)^{\,w_k} \quad\big[\text{ref-token}/\text{raw-token}\big] \end{aligned} } \label{eq:tivf} \end{equation} $$.

Then we use equal weights (i.e., $w_k=1/6$). One thing that we need to account for is models that can handle larger context do not automatically indicate "smarter" answers, but rather address the complexity/class of problems the model can attempt to address. The inclusion of a context modifier $\kappa(m)$ from eq. \ref{eq:context_modifier} tries to address what tasks are "attemptable" by a model.

By construction $\mathrm{TIVF}(m_\mathrm{ref},e)\equiv 1$ ref-token/raw-token when every $S_k(m_\mathrm{ref})=S_k(m_\mathrm{ref})$. A model with $\mathrm{TIVF}=4$ produces tokens that count quadruple toward intelligence action relative to the reference across the full spectrum, not on a single leaderboard. Note the geometric (not arithmetic) mean was deliberate on my part. A weak axis will drag the product down rather than being averaged away by a strong one, so a hard-capability cliff (e.g. high knowledge but failing coding) is penalized rather than hidden. Any scalar can still mask a single-axis failure, which is why the per-axis $S_k$ are reported alongside TIVF.

$$ \begin{equation} \kappa(m) = 1 + \beta\cdot\ln\left(\frac{L_m}{L_\mathrm{ref}}\right), \qquad \beta = 0.087. \label{eq:context_modifier} \end{equation} $$

With $m_\mathrm{ref}$ corresponding to GPT-4o mini ($L_\mathrm{ref}=128\,\mathrm{k}$ tokens), a 1M-token model picks up roughly an 18% multiplicative bonus ($\beta=0.087$). The exponent is small on purpose so context is an enabler, not a substitute for reasoning scores. Models with context windows below $L_\mathrm{ref}$ get $\kappa<1$, an intentional penalty reflecting the narrower class of tasks they can attempt. In truth, this is probably the framework's least defensible knob since context length is a capacity constraint, not intelligence. When a task fits comfortably inside every model's window $\kappa$ rewards headroom that yields no real utility. This means we should treat $\kappa$ as a task-attemptability gate, not an intelligence premium.

Task-specific workloads can replace eq. \ref{eq:tivf} with $\sum_k w_k\ln\mathrm{TIVF}_k$ using custom $w_k$ (e.g., overweight Hard coding for agentic coding) without changing the FOM details below. One other aspect is how "intelligence action" relates to "intelligence power". I keep both: eq. \ref{eq:intelligence_action} defines the action as a cognitive footprint (ref-token$\cdot$s) that keeps TIVF visible as a token rescaling rather than collapsing into raw throughput, while its time-rate $\mathrm{TIVF}\cdot R$ is the intelligence power (ref-token/s) that the figure of merit below actually prices out.

Including Reasoning effort

Most models expose some form of effort levels (OpenAI reasoning_effort, Anthropic extended/adaptive thinking, DeepSeek thinking modes) that change three things at once: the spectrum scores $S_k(m,e)$, hidden reasoning tokens billed as output, and wall-clock throughput $R$ (OpenAI GPT-5.5, Anthropic Opus 4.8, DeepSeek V4 pricing, evals.report).

To account for this, I tag each row with an effort level $e\in{\mathrm{none},\mathrm{high},\mathrm{max},\mathrm{fast}}$ and use an effort-specific benchmark scores when available. Hidden reasoning is modeled as a kind of effective output-price multiplier:

$$ \begin{equation} \tilde{c}_\mathrm{out}(m,e) = c_\mathrm{out}(m)\cdot \Big[1 + \lambda\cdot\big(\psi(e)-1\big)\Big], \qquad \lambda = 0.65, \label{eq:effort_cost} \end{equation} $$

where $\psi(e)$ is a billed-token multiplier ($\psi=1$ for non-thinking, $\sim 2$ at default thinking, $\sim 5$ at max/xhigh effort). Throughput $R$ is taken directly from provider API measurements at the stated effort level (Artificial Analysis, June 2026 snapshot)⁵, not scaled again by a latency factor. Note $R$ here is a response output rate, which for reasoning models includes hidden reasoning tokens, so $R$ already absorbs the thinking slowdown; $\psi$ acts only on the billed price, keeping rate ($R$) and price ($\psi$) on separate factors so reasoning is not billed twice (this is an approximation, see limitations). Fast mode rows use Anthropic fast-tier pricing with throughput scaled from the standard Opus row ($\times 2.5$).

The idea is this is a proxy for compute spent per visible token, not a literal token accounting. It prevents comparing a cheap non-thinking call against a frontier model evaluated at max effort.

Figure of merit: intelligence power per dollar

Intelligence action is a kind of cognitive footprint of a generation where the TIVF a token carries, held over the wall-clock time it takes to produce,

$$ \begin{equation} \mathcal{A}_I = \mathrm{TIVF}\cdot t \quad \Big[\tfrac{\text{ref-token}\cdot\mathrm{s}}{\text{raw-token}}\Big]. \label{eq:intelligence_action} \end{equation} $$

In the framing of Principle of Least Action, intelligence action beomce a kind of latency weight that for a fixed TIVF you minimize $\mathcal{A}_I$ by driving the time $t$ down, so a model that delivers the same intelligence faster leaves a smaller sweep. What you actually use per second is the time-rate of that intelligence, so a kind of intelligence power,

$$ \begin{equation} P_I = \mathrm{TIVF}\cdot R \quad \big[\text{ref-token}/\mathrm{s}\big], \label{eq:fom_deriv} \end{equation} $$

where $R$ is the measured output raw-token/s at effort $e$. Dividing by the effective cost rate $\tilde{c}_\mathrm{out}$ (USD per raw-token) gives the FOM that has units of intelligence power per dollar:

$$ \begin{equation} \boxed{ \mathrm{FOM}(m,e) = \frac{\mathrm{TIVF}(m,e)\cdot R(m,e)}{\tilde{c}_\mathrm{out}(m,e)} \quad\Big[\tfrac{\text{ref-token}\cdot\text{raw-token}}{\$\cdot\mathrm{s}}\Big]. } \label{eq:fom} \end{equation} $$

The numerator $\mathrm{TIVF}\cdot R$ is intelligence power (ref-token/s), but note that the denominator is per unit price, not per dollar spent because dividing by the spend rate $\tilde{c}_\mathrm{out}\cdot R$ (USD/s) cancels $R$ and leaves the speed-blind value efficiency $\mathrm{TIVF}/\tilde{c}_\mathrm{out}$ (ref-token/\$). So FOM is by design value-per-dollar $\times$ throughput: $R$ stays up top to reward latency, and the raw-token in ref-token$\cdot$raw-token/($\cdot$s) is the receipt. The $\tilde{c}_\mathrm{out}$ reduces to raw output pricing when $\psi=1$. Input/cache pricing matters when using in RAG and agents and a blended $\tilde{c}$ is easy to substitute in eq. \ref{eq:fom} if your workload is input-dominated. As a reminder $m$ the model and $e$ the effort level.

Reading the number

Larger FOM means more intelligence value per dollar, weighted by throughput for the stated effort! It is a value metric, not a capability one, so compare at matched effort: a budget model can win on FOM while losing on TIVF. Price dominates the spread ($c_\mathrm{out}$ spans $\sim$100$\times$ vs $R$ $\sim$10$\times$ and TIVF $\sim$5$\times$), so FOM is, to first order, a tokens-per-dollar ranking that TIVF only tilts.

Case study: Ballpark comparison (June 2026)

Note

Snapshot date: June 2026. Pricing and throughput numbers move quickly; treat the tables as ballpark illustrations, not a live leaderboard. Also LLMs were used heavily here to generate the tables and plots. Some aspects of the pattern analysis was also done with LLMs.

The first table lists cross-vendor inputs at matched max/xhigh effort; the second lists the Claude Opus lineage. A third table shows derived TIVF and FOM from eqs. \ref{eq:tivf} and \ref{eq:fom}. $S$ is the arithmetic mean of the six axis scores (display only). Output pricing $c_\mathrm{out}$ is from official provider pages (June 2026); throughput $R$ is output token/s from Artificial Analysis provider API measurements at the stated effort⁵. Benchmark scores are from vendor cards, evals.report, BenchLM, MorphLLM, and Anthropic system cards. Coverage spans OpenAI, Anthropic, Google, DeepSeek, xAI (Grok), Moonshot (Kimi), and Alibaba (Qwen), plus one open-weight row on GroqCloud; Groq (inference host) is not Grok (xAI's model).

Input tables and derived TIVF / FOM

Cross-vendor

Model	Effort	S	SWE-Pro	HLE tools	cout	R
DeepSeek V4 Flash	max	69.3	54.0%	45.1%	$0.28 / M	106
DeepSeek V4 Flash	high	67.2	52.6%	40.3%	$0.28 / M	95
DeepSeek V4 Flash	none	61.1	49.1%	28.0%	$0.28 / M	94
GPT-4o mini (ref.)	high	33.3	12.0%	16.0%	$0.60 / M	66
Gemini 2.5 Pro	high	71.4	54.2%	51.4%	$10.0 / M	141
Gemini 3 Pro	high	74.1	55.0%	52.0%	$12.0 / M	130
Gemini 2.5 Flash	high	60.5	45.0%	38.0%	$2.50 / M	225
Gemini 3 Flash	high	67.7	50.0%	43.5%	$3.00 / M	300
Gemini 3.1 Flash	high	69.0	52.0%	45.0%	$3.00 / M	450
Kimi K2.6	high	72.8	58.6%	54.0%	$4.0 / M	88
Qwen3 Max	high	69.2	50.0%	49.5%	$6.0 / M	72
Grok 4	high	65.5	48.0%	42.0%	$15.0 / M	78
GPT-5.5	max	76.0	58.6%	52.2%	$30.0 / M	53
GPT-5.5	high	72.1	55.0%	50.0%	$30.0 / M	56

Claude Opus lineage

Model	Effort	S	SWE-Pro	HLE tools	cout	R
Claude Opus 4.5	high	67.8	45.9%	43.4%	$25.0 / M	42
Claude Opus 4.6	high	72.5	53.4%	53.1%	$25.0 / M	45
Claude Opus 4.7	high	77.0	64.3%	54.7%	$25.0 / M	45
Claude Opus 4.7	max	77.8	65.5%	55.5%	$25.0 / M	55
Claude Opus 4.8	high	79.2	69.2%	57.9%	$25.0 / M	58
Claude Opus 4.8	max	79.8	70.0%	58.5%	$25.0 / M	58
Claude Opus 4.8	fast	79.2	69.2%	57.9%	$50.0 / M	145

Derived TIVF and FOM

Model	Effort	TIVF [ref-token/raw-token]	FOM [ref-tok·raw-tok/($·s)]
DeepSeek V4 Flash	none	3.23	1.08 × 109
DeepSeek V4 Flash	high	3.76	7.73 × 108
DeepSeek V4 Flash	max	3.94	4.15 × 108
Gemini 3.1 Flash	high	4.12	3.75 × 108
Llama 3.3 70B (Groq)	high	1.18	3.58 × 108
Gemini 3 Flash	high	3.81	2.31 × 108
Gemini 2.5 Flash	high	3.28	1.79 × 108
GPT-4o mini	high	1.00	6.67 × 107
Kimi K2.6	high	3.85	5.13 × 107
Gemini 2.5 Pro	high	4.15	3.55 × 107
Claude Haiku 4.5	high	2.59	3.36 × 107
Gemini 3 Pro	high	4.33	2.84 × 107
Qwen3 Max	high	3.55	2.58 × 107
Claude Opus 4.8	fast	4.77	1.38 × 107
Grok 4	high	3.27	1.03 × 107
Claude Sonnet 4.6	high	3.70	8.22 × 106
Claude Opus 4.8	high	4.77	6.71 × 106
Claude Opus 4.7	high	4.57	4.98 × 106
GPT-5.5	high	4.19	4.74 × 106
Claude Opus 4.6	high	4.21	4.59 × 106
Claude Opus 4.5	high	3.34	3.40 × 106
Claude Opus 4.8	max	4.82	3.11 × 106
Claude Opus 4.7	max	4.64	2.83 × 106
GPT-5.5	max	4.49	2.20 × 106

Figure 1. FOM vs TIVF scatter (June 2026 snapshot).

Figure 2. FOM ranked by model and effort level.

Figure 1 plots FOM against TIVF with the point color representing provider and the marker shape representing thinking effort (two-column legend). Figure 2 ranks every row by FOM on a log scale. The bottom line from Figure 1 seems to be if you want the most intelligence power per dollar, use DeepSeek V4 Flash with high thinking. Opus 4.8 [max] clearly scores higher on TIVF ($\approx 4.8$ vs $\approx 3.8$ for DeepSeek [high]) but buys roughly 250$\times$ less intelligence power per dollar. Several patterns fall out immediately:

• Broad-spectrum TIVF separates frontier from budget models. At max effort, Opus 4.8 ($\approx 4.8$) leads GPT-5.5 ($\approx 4.5$), Gemini 3 Pro ($\approx 4.3$), and DeepSeek V4 Flash ($\approx 3.9$).
• DeepSeek wins FOM because price is low and speed is high. At \$0.28/M output and $\sim$95--106 token/s it buys roughly $\sim$250$\times$ more intelligence power per dollar than Opus 4.8 at max effort; Kimi K2.6 and Qwen3 Max trail.
• The Gemini Flash line is the best-value major-lab option. At \$3/M output and 300--450 token/s, Gemini 3.1 Flash (TIVF $\approx 4.1$, FOM $\approx 3.7\times 10^8$) slots in just under the DeepSeek rows and edges out the fast/cheap Groq Llama, with Gemini 3 Flash ($\approx 3.8$, $\approx 2.3\times 10^8$) and the older Gemini 2.5 Flash ($\approx 3.3$, $\approx 1.8\times 10^8$) trailing the lineage. Each newer generation buys both higher TIVF and higher FOM, and all three beat every frontier row on FOM by one to two orders of magnitude while carrying near-frontier TIVF.
• Grok 4 sits mid-pack on both axes. TIVF $\approx 3.3$ (near Gemini 2.5 Flash) but frontier pricing (\$15/M) wins neither leaderboard.
• When TIVF, not FOM, is the constraint, pay up. On the hard tail Opus 4.8 [max] leads SWE-Pro (70% vs 54%) and HLE w/ tools (58.5% vs 45.1%) (DataCamp, evals.report).
• Effort reshuffles both axes. DeepSeek [none] is fastest/cheapest (FOM $\approx 1.1\times 10^9$) but loses 0.7 TIVF versus [high]; GPT-5.5 [max] buys $\sim$0.3 TIVF at a steep reasoning-billing premium.
• Speed is now rewarded because $R$ sits in the FOM numerator. FOM prices intelligence power (ref-token/s) per dollar via eq. \ref{eq:fom}, so for a fixed TIVF and price a faster model delivers more intelligence per second and scores higher; the old speed paradox (where a slower model looked better) is resolved. Opus 4.8 [fast] ($R=145$) scores FOM $1.38\times 10^7$ vs [high] ($R=58$) at $6.71\times 10^6$, and Groq's fast/cheap Llama vaults to $3.58\times 10^8$ on low TIVF ($\approx 1.2$) almost entirely on speed and price.

FOM and TIVF answer different questions, so deciding which to use depends on your goal:

Your goal	Pick
Fixed budget: maximize total smart work over an hour or a day	DeepSeek V4 Flash high
Fixed budget, need higher TIVF than DeepSeek	Kimi K2.6 high
Fixed budget, easy tasks, cost and speed only	DeepSeek none
Hard agentic, coding, or research cliff	Opus 4.8 max (or GPT-5.5 xhigh)
Best single token, price irrelevant	Opus 4.8 max (highest TIVF in Figure 1)

Case Study Summary

Under the ballpark effort the split becomes:

Most intelligence power per dollar: DeepSeek V4 Flash at high thinking effort. For a fixed API budget, this row buys the most intelligence power per dollar. Kimi K2.6 (high) is the nearest runner-up if you need higher SWE-Pro/HLE scores without Opus pricing. Use DeepSeek max when the extra SWE/HLE points on the capability cliff matter; use none only when you accept lower TIVF for maximum thrift.

Most intelligence per token is Claude Opus 4.8 at max effort (then GPT-5.5 xhigh), accepting more than two orders of magnitude lower FOM.

Again key is not to read high FOM as "smartest model", so DeepSeek [high] is the value champion, not the capability champion. Confuse the two and you mis-deploy your use case.

My Thoughts

I'm in no way knowledgable enough to state if this is the correct FOM or even makes sense to the ML/AI community. I was just trying to think through Figure-of-merits due to my familarity with them in materials physics. This is just a "back of the envelope" type of analysis and proposal, not a standard.

There are a bunch of places where I think this probably falls apart or at least needs heavy caveating, so let me just list what may be the issues:

• TIVF is only as good as the scores $S_k$ I feed it. I am leaning on vendor-reported numbers, evals that were not always run at matched effort, and benchmarks that may already be contaminated. So as the old adage goes, Garbage in, garbage out still applies.
• My equal weights are just best guess. I'm set $w_k=1/6$ in eq. \ref{eq:tivf} because its easy and I have no other ideas for what it should be. In reality this needs to be tuned to the type or workload.
• The effort knob is a guess. $\psi(e)$ in eq. \ref{eq:effort_cost} is my guess for billed tokens, not a real count of reasoning tokens per request. Even the throughput $R$, which I did account for per effort row, will still shift with prompt length, caching, and region.
• Throughput and price are not fixed numbers. Reasoning modes, caching, batch APIs, and self-hosting can move $\tilde{c}$ and $R$ a lot, so I'm looking at one point in time.⁴
• No accounting for verbosity. A model that scores well while spitting out twice as many tokens looks fine here, even if it actually feels worse to use.
• My framing is output-biased. All the input-side smarts get under-sold because I plugged an output-only $\tilde{c}$ into eq. \ref{eq:fom}.
• Not exhaustive. I skipped Mistral, Cohere, etc. and I am using Artificial Analysis estimates where the provider pages did not give me matched-effort numbers.

I'd be interested to see how well I did compared to legit AI analysts. Assuming this is reasonable and if I wanted to take this past a back-of-the-envelope toy, what would they suggest? My effort for this is to make the claim as narrow as possible:

When comparing language models, treat tokens as heterogeneous and report intelligence power per dollar alongside raw scores.

Footnotes

---

1. Reference model: GPT-4o mini anchors every axis in eq. \ref{eq:tivf_domain} with $(S_\mathrm{MMLU},S_\mathrm{GPQA},S_\mathrm{SWE},S_\mathrm{SWE-Pro},S_\mathrm{HLE},S_\mathrm{TB})=(72,50,40,12,16,10)$ and $\mathrm{TIVF}\equiv 1$ by definition (eq. \ref{eq:tivf_def}), chosen as a stable cost-quality floor, not the most-used production model. ↩

2. I write the TIVF unit as ref-token/raw-token: each physical (raw) token a model emits carries TIVF reference-equivalent (ref) tokens of intelligence. Keeping the singular ("token") reflects that TIVF is a scalar field over models, one value each, like "one meter." Splitting raw-token from ref-token is what makes the downstream power (ref-token/s) and FOM (ref-token·raw-token/($·s)) units come out honest. ↩

3. Related work and how this differs. The closest cousins are intelligence index over blended price (Artificial Analysis, WhatLLM.org, Analytical Insider), which ignores latency; tokens per dollar (FriendliAI on DeepSeek V4), which assumes token homogeneity; and economics' quality-adjusted price indices, where here "quality" is an explicit $\mathrm{TIVF}$ tied to public evals rather than a hedonic regression. What I add: (1) a named token rescaling (TIVF) with a broad-spectrum formula (eqs. \ref{eq:tivf_domain}, \ref{eq:tivf}); (2) effort-aware billing for hidden reasoning tokens (eq. \ref{eq:effort_cost}) so comparisons happen at matched thinking levels; and (3) a time factor via intelligence action (eq. \ref{eq:intelligence_action}) yielding FOM (eq. \ref{eq:fom}), separating token quality, the power $\mathrm{TIVF}\cdot R$, and power per dollar. ↩

4. Prices and throughputs are a snapshot, not fixed; the same headline price hides very different real cost depending on reasoning modes, caching, batch APIs, and self-hosting (Finout on Claude Opus 4.7 pricing). ↩

5. Output pricing from official provider pages (OpenAI, Anthropic, Google, DeepSeek, xAI, Moonshot, Alibaba Model Studio, Groq). Throughput $R$ is output token/s from Artificial Analysis (June 2026), matched to each row's effort; Moonshot, xAI, and Alibaba use AA estimates. Llama 3.3 70B uses GroqCloud (Groq the host, not xAI Grok). Reasoning models bill hidden tokens as output, and AA's $R$ already counts them, so the price multiplier $\psi(e)$ (eq. \ref{eq:effort_cost}) and $R$ partially overlap; I keep $\psi$ purely on price so reasoning load is billed once, an approximation rather than exact token accounting. ↩↩

References

[1] Y. Wang, X. Ma, others, {MMLU-Pro}: A More Robust and Challenging Multi-Task Language Understanding Benchmark, arXiv Preprint arXiv:2406.01574. (2024). https://doi.org/10.48550/arXiv.2406.01574.

[2] D. Rein, others, {GPQA}: A Graduate-Level Google-Proof {Q\&A} Benchmark, arXiv Preprint arXiv:2311.12022. (2024). https://doi.org/10.48550/arXiv.2311.12022.

[3] C.E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, K.R. Narasimhan, {SWE}-bench: Can Language Models Resolve Real-World {GitHub} Issues?, in: International Conference on Learning Representations, 2024. https://doi.org/10.48550/arXiv.2310.06770.

Reuse and Attribution

What in the World is a World Model?

It's clear that LLMs+RL+<Insert ML/AI Approach> is leading to some pretty impressive AI tools that can solve problems¹ in novel and unique ways. Additionally, LLMs have shown that when the task or thinking is best defined through language, code, or mathematics, these AI tools can clearly outperform humans at some of these tasks. Software engineering benchmarks [1] clearly show rapidly advancing capabilities in autonomous issue resolution, though the resolved fractions on real-world repositories remain far below typical human developers. However, the internal mechanisms or state representations, which are clearly rich and capturing something we humans would call "intelligence", these AI tools do not, as far as I know², out of the box, maintain a reliable internal "simulator" of state and dynamics. They can roll out scenarios in language/code/math (chain-of-thought, tree search, and similar), but those rollouts are often inconsistent or ungrounded when you need faithful prediction over many steps of inputs, information, constraints, goals, and actions [2]. This seems to me to be a clear limitation of the current SOTA, but keep in mind I'm not a researcher in this space, 🤷‍♂️.

Note

There are a lot of references for this post and I did not read most of them all in great detail, but rather "ran" through them and then asked questions to clarify my understanding using Gemini and other LLMs.

If I take a moment here to think about how a human thinks internally it seems we excel at abstraction and planning in a model space of the world we've experienced. Take the scenario of when you or I think about a ball rolling down a hill, we have this mental model for what the ball will do based on our knowledge of the environment around us that we have observed. Now true, a LLM does have knowledge of the governing physics, possible initial/boundary conditions, and thus can numerically simulate the physics exactly but that is not what we do, we often do this by approximate forward simulation in the head, like an "intuitive physics engine" [3]. This knowledge can be shaped early by maturation and perception³, and it is directly reinforced from our continued observations of the same interactions and patterns. We humans have a world model [4] that we use to make decisions and predictions about the world around us; the same idea was formalized in AI as a learned simulator of environment dynamics [5]. Our brains likely evolved for this because it allows us to make informed decisions towards our actual actions and goals ... not a reseacher here just thinking out loud.

Again just stop and think about it, when you are preparing for say a trip or presentation how you mentally play things out the scenarios that may occur. Or think about when you are doing any lab or simulation work, how you mentally scope various scenarios and hypothesize about the outcomes. As you do these things you narrow down on the possible set of actions and targeted goals you want to achieve based on how your mental perspective of these form.

I still don't know if I truly gauge or grasp what a world model is, but I think what it tries to represent is the abstract planning of actions and goals based on the latent (i.e. largely unconscious) knowledge we have acquired. The better the predictive model, the better our real world actions and goals are often defined and executed ... though "world model" is an analogy here it seems Craik's internal models, episodic simulation of the future, and motor internal models in the brain are clearly related ideas, not one single thing [6].

Take riding a bike. Everyone who has learned to ride a bike will tell you that if they rode a bike for 5 years and then stopped for 10 years and then tried again to ride the bike, they often recover riding ability within 10 minutes or so. This is the procedural memory for skilled actions that is stored separately from everyday declarative memory [7]. Truth though is it seems to be that the persistence is usually explained by your implicit motor circuits (basal ganglia, cerebellum, and related systems), not by consciously rehearsing the physics in your mind's eye before you mount, although though mental imagery does help. Still, I argue that there is a motor "model" of riding a bike such that it encodes balance, pedaling, and steering in a given context. If we did the same thing but instead of riding the bike on a road we told you to ride the bike on a tightrope or snow, your learned skills for riding on a road would still be invoked (i.e. you would think about how it might feel to ride a bike on a tightrope) but because that latent representation was never tuned or updated with actual feedback from those environments (i.e., you never actually rode a bike on a tightrope or snow), you would likely struggle [8]. Interestingly though, you could imagine that some might do better than others because their prior experience and internal forward models are richer, so mind-planning in the tightrope action space is more robust.

LeCun's World Model

Yann LeCun has been fairly vocal about world models as part of a broader cognitive architecture for embodied AI. He sees it as a orthogonal direction to LLMs. My understanding on his position is that systems that only predict the next token (or pixel) in some observation space will struggle to reach the kind of sample-efficient reasoning and planning that animals show. The hypothesis is because they waste capacity on unpredictable surface detail instead of learning the proper abstract state and dynamics. The recipe he and collaborators have worked on is a configurable predictive world model: encode what is observed, then roll forward in a compact latent space (optionally conditioned on actions), and use those "imagined" trajectories to choose goals and controls [9].

The way, to my understanding, that this idea departs from classic supervised vision is exactly what I was gesturing in the cat example. Don't train a network to output the label "cat" from pixels and don't ask the model to reconstruct every RGB value either. Instead have encoders that map observations into a high-dimensional latent representation, and a predictor that forecasts how that representation should evolve. For just static images the "action" conditioning the predictor can be absent (e.g., I-JEPA⁴ [10]). From this we get a self-supervised prediction in representation space and avoids forcing two augmented views of a cat to share one embedding via contrastive learning. In ref. [9] it seems he actually argues against leaning on contrastive losses and favors regularized, predictive objectives instead, but not sure I understand, per usual.

So what does JEPA (Joint-Embedding Predictive Architecture) look like? Rather than me go through the details, for which my visuals may not be as clear, The interview with LeCun on Welch Labs had some really good visuals and explanations:

There are many variations of JEPA, each built on the same overall idea ofmapping observed data into latent representations with a predictor and training via prediction errors in latent space rather than on raw outputs [9]. There is I-JEPA for images, V-JEPA for video, action-conditioned versions for control tasks, and hierarchical JEPA for planning at multiple time scales [11],[9].

The LeWorldModel [12] is a recent implementation of an action-conditioned small encoder-predictor trained and includes a SIGReg in the loss that keeps latents from collapsing. There is no reconstructiong the pixels for forward frames; everything is done in latent space. At test time it encodes start and goal images, rolls the predictor forward under candidate actions, and picks the sequence whose final latent is closest to the goal. The paper's claim to fame is that the planning (i.e. action conditioned predictor) in latent space is cheap comapred to using pixels. An interesting thing here is that nothing in the self-supervised loss contains any biasing towards known physics (i.e., how 3D classical mechanics should work).

How does a world model connect to physics? Probably not a drop-in replacement for a classical PDE solver on a macroscopic grid or an atomistic MLIP, but the idea of a JEPA-style scientific world model makes sense to me because it matches what I've seen from learned simulators and neural operators, such as Fourier Neural Operators [13] to learn a compact state, predicted evolution under physics biases, rolled forward without reconstructing every emergent degree of freedom at each step.

Footnotes

---

1. Frontier LLM have clearly shown in mathematics and perhaps elsewhere solutions that had yet to been-thought-of by humans. The clear demonstration for this is Erdos problems where solutions that experts state are novel have been produced by frotnier models from OpenAI and Anthropic [14–15]. ↩

2. Keep in mind, I am not a frontier researcher in AI research in any shape or form but more of a domain specialist applying AI tooling. ↩

3. I use "bootstrapping" to mean that without any prescribed inductive biases, the correct mental model exist and can be used to make the correct prediction or action. I don't know if this is the correct wording. A good example is the visual cliff experiment [16]: crawling infants on a glass floor where one half looks solid and safe while the other looks like a cliff. Most refuse to cross to their mother when she calls from the cliff side, consistent with a bootstrapped visual model of depth. ↩

4. Take the visible patches of an image as context and train a predictor to match the latent embeddings of other masked regions in the same image, where the target embeddings come from a slow-moving target encoder (exponential moving average of the context encoder) so the problem does not collapse to a trivial constant latent representation. ↩

References

[1] C.E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, K.R. Narasimhan, {SWE}-bench: Can Language Models Resolve Real-World GitHub Issues?, in: International Conference on Learning Representations, 2024. https://doi.org/10.48550/arXiv.2310.06770.

[2] S. Yao, D. Yu, J. Zhao, I. Shafran, T.L. Griffiths, Y. Cao, K. Narasimhan, Reasoning with Language Model is Planning with World Model, in: Empirical Methods in Natural Language Processing, 2023. https://doi.org/10.18653/v1/2023.emnlp-main.507.

[3] P.W. Battaglia, J.B. Hamrick, J.B. Tenenbaum, Simulation as an Engine of Physical Scene Understanding, Proceedings of the National Academy of Sciences. 110 (2013) 18327--18332. https://doi.org/10.1073/pnas.1306572110.

…

Show remaining references

[4] K.J.W. Craik, The Nature of Explanation, Cambridge University Press, Cambridge, 1943. https://books.google.com/books?id=wT04AAAAIAAJ.

[5] J. Schmidhuber, Making the World Differentiable: On Using Self-Supervised Fully Recurrent Neural Networks for Dynamic Reinforcement Learning and Planning in Non-Stationary Environments, (1990). https://www.idsia.ch/~juergen/FKI-126-90.pdf.

[6] D. McNamee, D.M. Wolpert, Internal Models in Biological Control, Annual Review of Control, Robotics, and Autonomous Systems. 2 (2019) 339--364. https://doi.org/10.1146/annurev-control-060117-105206.

[7] N.J. Cohen, L.R. Squire, Preserved Learning and Retention of Pattern-Analyzing Skill in Amnesia: Dissociation of Knowing How and Knowing That, Science. 210 (1980) 207--210. https://doi.org/10.1126/science.7414331.

[8] K.E. Adolph, Specificity of Learning: Why Infants Fall Over a Veritable Cliff, Psychological Science. 11 (2000) 290--295. https://doi.org/10.1111/1467-9280.00258.

[9] Y. LeCun, A Path Towards Autonomous Machine Intelligence, (2022). https://openreview.net/forum?id=BZ5a1r-kVsf.

[10] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, N. Ballas, Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. https://doi.org/10.48550/arXiv.2301.08243.

[11] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F.R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, N. Ballas, {{V-JEPA} 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning}, arXiv Preprint arXiv:2506.09985. (2025). https://doi.org/10.48550/arXiv.2506.09985.

[12] L. Maes, Q.L. Lidec, D. Scieur, Y. LeCun, R. Balestriero, {{LeWorldModel}: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels}, arXiv Preprint arXiv:2603.19312. (2026). https://doi.org/10.48550/arXiv.2603.19312.

[13] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Fourier Neural Operator for Parametric Partial Differential Equations, in: International Conference on Learning Representations, 2021. https://doi.org/10.48550/arXiv.2010.08895.

[14] G. DeepMind, Advancing Mathematics Research with {AI}-Driven Formal Proof Search, (2026). https://doi.org/10.48550/arXiv.2605.22763.

[15] {OpenAI}, An OpenAI model has disproved a central conjecture in discrete geometry, (2026). https://openai.com/index/model-disproves-discrete-geometry-conjecture/.

[16] E.J. Gibson, R.D. Walk, The Visual Cliff, Scientific American. 202 (1960) 64--71. https://doi.org/10.1038/scientificamerican0460-64.

Reuse and Attribution

DFT is gettings its share of AI

I made a post on LinkedIn the other day sharing a recent preprint[1] by Microsoft AI that introduces a significant advancement in exchange-correlation functionals (XC) for DFT. This, to me, seemed like an important piece of news given that much of the data used for training the recent explosion in foundational interatomic potentials for atomistics uses massive DFT datasets.

The biggest achievement seems to be the ability for Microsoft AI's DFT XC functional, called Skala, to achieve near chemical accuracy (i.e., 1 kcal/mol) for molecular systems while doing it with reasonable computational cost. What this means is that we may get a new rung added to the chemistry Jacob's ladder of accuracy for DFT. I wanted to illusrate this so I made a diagram of the chemistry Jacob's ladder with the new Skala XC added as shown in Figure 1. I got the idea to annotate it from Keiran Rowell's blog[2].

The idea of Jacobs Ladder in chemistry, I think, was first introduced by J. Perdew in 2013(?) [3], I believe. The primary idea is that like the in the biblical story, Jacob dreams of a ladder reaching from earth to heaven and God standing above it promising Jacob divine blessing and protection. In the spirit of this, we the scientist dream of reaching chemical accuracy using our ladder of DFT approximations. Why is this important? Well, chemical accuracy with DFT would enable accurate enough predictions in reasonable computational time to improve discovery, design, and understanding of chemicals and materials. In particular understanding reaction mechanisms and kinetics would be much more tractable if chemical accuracy is reached.

Figure 1. Computational chemistry Jacob's ladder for DFT with the Microsoft AI Skala XC added.

DFT XC Overview

First, let me briefly summarize some basics of density functional theory (DFT) to explain why the exchange-correlation (XC) functional is so important. The Nobel-worthy Hohenberg–Kohn theorems and the formulation of the Kohn–Sham equations established that the total ground-state (i.e., 0 K) energy of a many-body electronic system can be written as:

$ \begin{equation} E[\rho] = T_s[\rho] + V_{ext}[\rho] + J[\rho] + E_{xc}[\rho] \end{equation} \label{eq:dft_energy} $

The first three terms are kinetic energy, external potential energy, and Coulombic repulsion energy. The last term is the XC energy which is the only term we don't know exactly and is crucial because it encapsulates the many-body quantum mechanical effects of the electrons whereas the others are single-particle terms.

Throughout the successful history of DFT, different approximations have been used for the XC term. The first was local density approximation (LDA), with the idea being that the XC energy is simply a function of the local density of the electrons. For all its simplicity, LDA did work reasonably for some systems but shortcomings were quickly discovered.

Then the generalized gradient approximation (GGA) was introduced, where the XC energy is now not only a function of the local density but also the gradient of the density. This introduced the idea of non-local XC functionals (i.e., the energy now depends on how the density changes with position, rather than just at that position).

After GGA, what followed was a "flavor-of-the-month" approach toward improving XC functions. There were hybrid functionals that mixed DFT XC with the Hartree-Fock exchange. Then meta-GGA was introduced with the idea of adding higher-order derivatives to the density. Add to that expensive XC functionals that include perturbation quantum chemistry methods. All have their improvements and shortfalls. You can see all the major players in the rungs of the ladder in Figure 1 showing how different XC functionals improve the range of accuracy as you climb up the ladder.

These are just the formalisms; there are also ad-hoc tweaks and specializations that get added based on your domain use and expertise. The limitations of such approaches are:

1. Handcrafted Features: Most functionals rely on fixed analytic forms and known constraints.
2. Non-locality Deficiency: Electron correlation is inherently non-local; local/semi-local functionals cannot fully capture this.
3. Empirical Tuning vs. Physical Justification: Empirical functionals may generalize poorly or violate constraints.
4. Slow Progress at Higher Rungs: Despite decades of work, hybrid/double-hybrid methods still fall short of universal chemical accuracy (error < 1 kcal/mol).

Skala Neural XC

The efforts by Microsoft AI represent a significant leap in XC functional design. Unlike traditional functionals that have relied on hand-crafted mathematical forms, Skala incorporates deep learning (i.e., Neural Networks) to achieve near-chemical accuracy while maintaining computational efficiency comparable to meta-GGA functionals[1]. Skala is particularly impressive because it navigates the trade-off between accuracy and computational cost. The Microsoft AI team has designed it to bridge the gap between semi-local functionals (fast but less accurate) and hybrid/double-hybrid functionals (accurate but computationally expensive).

Architecture and Design Philosophy

As mentioned, Skala XC is a neural network architecture trained to learn the non-local electron density interactions without requiring the full computational burden of exact exchange calculations. Like meta-GGA, it starts with seven semi-local density-derived features, but the design then employs what the authors call a "coarse-fine grid structure" that captures long-range density correlations consistent with multipole-like behavior.

For the exchange, the authors use LDA but incorporate a neural enhancement factor $f_\theta$:

$ \begin{equation} E_{xc}^\theta[\rho] = -\frac{3}{4}\left(\frac{6}{\pi}\right)^{1/3} \int \left(\rho_\uparrow(r)^{4/3} + \rho_\downarrow(r)^{4/3}\right) f_\theta[x[\rho]](r) \, dr \label{eq:skala_xc} \end{equation} $

Not knowing too much about XC design, this does seem clever though because $f_\theta$ operates on a feature vector $x[\rho]$ that encodes both local and non-local density information while maintaining computational efficiency (would like to understand this but above my head for now). This is reminiscent of delta-learning in atomistic ML, though Skala performs a direct functional approximation rather than a residual correction atop an existing functional.

What data was used?

The Microsoft AI team used a quantum chemistry dataset that seems only they could have curated, something like 150,000 data points spanning thermochemistry, conformational energies, noncovalent interactions, and ionization potentials. This was all done with gold-standard wavefunction methods like CCSD(T) to generate the data, 🤯. Their MSR-ACC/TAE dataset alone includes about 80,000 total atomization energies with errors below 1 kcal/mol, which is just staggering when you consider the computational expense of generating such reference data. The datasets alone might be useful for others.

Peformance Metrics

I'm not too familiar with benchmarks in this space, but they mention the challenging W4-17 benchmark and Skala achieves a mean absolute error (MAE) of around 1.0 kcal/mol, outperforming more established functionals. On the GMTKN55 benchmark, it scores a WTMAD-2 of 3.89 kcal/mol, which puts it in competition with the best hybrid functionals while requiring significantly less computational resources.

Another thing that is super interesting is that for large atomic numbers that are out of distribution for the trained Scala XC, They are showing that it maintains very good accuracy. This is usually a failure point for data-driven ML models where out of distribution data is not well handled particularly well [4]. The authors also method that the self-consistent field convergence is stable, crucial for practical application in DFT --- since, unlike post-SCF models, Skala is trained and used self-consistently.

Redefining Jacob's Ladder?

So going back to the Jacob's ladder of DFT functionals in Figure 1. Whether Skala constitutes a sixth rung or a bypass of the traditional ladder remains obviously open to interpretation, but it clearly represents a shift. I'm not a veteran in this space so hard for me to be definitive, but it probably represents some departure to non-locality from semi-local features to effectively bypass the traditional way of doing things.

It will be interesting to see from the "Kings of DFT" to see how they view Skala XC. Based on the Microsoft AI preprint this functional(s) is showing systematic improvement through use of high-quality data and training procedures, and the authors claim it can encode known physics through appropriate constraints and design. On the other hand, they lack the theoretical transparency and interpretability that traditional functional forms offer will probably be a critiqued, although existing functionals are not perfect either.

I'm just interested to see how this gets used for the other down stream applications in solid-state physics and materials science. Will we get DFT training sets for Materials Project structures that range in the 1 million of structures and all have chemical accuracy? If so that could in turn make these foundation models even better for materials discovery and classical atomistic modeling.

---

References

[1] G. Luise, C.-W. Huang, T. Vogels, D.P. Kooi, S. Ehlert, S. Lanius, K.J.H. Giesbertz, A. Karton, D. Gunceler, M. Stanley, W.P. Bruinsma, L. Huang, X. Wei, J.G. Torres, A. Katbashev, B. Máté, S.-O. Kaba, R. Sordillo, Y. Chen, D.B. Williams-Young, C.M. Bishop, J. Hermann, R. van den Berg, P. Gori-Giorgi, Accurate and scalable exchange-correlation with deep learning, (2025). DOI.

[2] K. Rowell, An Ersatz Ansatz, Blog (2023). https://keiran-rowell.github.io/guide/2023-04-12-compchem-methods-basics (accessed June 21, 2025).

[3] J.P. Perdew, Climbing the ladder of density functional approximations, MRS Bull. 38 (2013) 743–750. DOI.

[4] K. Li, A.N. Rubungo, X. Lei, D. Persaud, K. Choudhary, B. DeCost, A.B. Dieng, J. Hattrick-Simpers, Probing out-of-distribution generalization in machine learning for materials, Commun Mater 6 (2025) 1–10. DOI.

---

Reuse and Attribution