Rankscale variance study · De'Longhi (US)

How consistent are AI engines when they recommend brands?

We asked nine AI engines the same questions repeatedly - 1,722 executions in total - and measured whether they return the same brands, in the same order. This report shows how stable each engine is and where the variation comes from.

7 topics × 9 engines = 63 cells 1,722 executions ~46 days tracked Source: Rankscale REST API

Key findings

The short version

One headline score answers "same answer twice?". Three sub-metrics explain in what way an engine varies.

The headline

No engine is highly consistent, and most of the variation is in which brands appear, not their order

For every engine, the exact set of brands named changes from run to run (Lineup Stability about 54%). The order of the brands that do appear is far steadier (Order Stability about 76%), and the top pick holds about 71% of the time. Appearing once does not mean you appear the next time.

API vs GUI

Provider APIs are more consistent than live-web interfaces

The four [API] engines hold the top of the consistency leaderboard; the live-web [GUI] engines (Google AI Overview, ChatGPT) sit at the bottom.

Stabilization

Scores settle by 10-15 runs

Consistency scores move around for the first few executions, then flatten. About 25 runs per prompt is enough to trust the number.

Range

Best to worst: 35.8% → 22.0%

Claude 4.5 Haiku [API] leads; Google AI Overview [GUI] trails - a 14-point spread in overall Consistency.

How we measure it

One headline score, three sub-metrics

The unit is a single prompt run repeatedly on a single engine. For each run we take the answer, extract the ordered list of brands it names (matched against Rankscale's own tracked brand set), and compare those lists across runs. One headline number, plus three that explain it.

RBO, top-weighted

Consistency

How repeatable the answer is overall, weighting the top of the list most heavily. The headline score. Read it by comparing engines, not against 100%.

Jaccard overlap

Lineup Stability

Whether the engine names the same brands each time.

Kendall τ, rescaled

Order Stability

Whether it keeps those brands in the same order. 50% means no better than random ordering.

Modal #1 share

Top Pick Stability

How often it names the same brand first. This is the number brands care about most.

Deliverable 1

AI Engine leaderboard

Each engine averaged across its 7 topics, sorted by Consistency. [API] = provider API; [GUI] = live browser interface.

AI EngineConsistencyLineupOrderTop Pickavg runs

Consistency (RBO) is top-weighted and runs low in absolute terms - compare engines to each other, not to 100%. Lineup / Order / Top Pick are directly interpretable.

Deliverable 2

Consistency by topic

Every engine × topic cell. Darker = more consistent. The averages hide strong topic effects: some cells are near-perfect, others barely above noise.

Consistency15% → 40%+  ·  rounded to whole %
Claude 4.5 Haiku [API] returns an identical list every run on "bean to cup" (100%) - a topic De'Longhi dominates - yet drops to ~20% on manual espresso. Google AI Overview [GUI] is the most volatile across the board.

Deliverable 3

How many runs a score needs to stabilize

Consistency recomputed on the first k executions, for growing k. Solid lines = [API] engines, dashed = [GUI].

Every curve flattens by roughly 10-15 runs - the estimate converges, and the engine ranking is stable from ~k=10 on. Chart capped at k=26, where all nine engines still have all seven topics supporting each point.

Method & caveats

How to read these numbers

Read with these in mind
  • Brands seen through Rankscale. Extraction is limited to Rankscale's tracked brand set per prompt; brands it doesn't track are invisible, which biases all scores slightly upward.
  • Real-world drift is included. Over ~46 days the market itself shifts, so a little "inconsistency" is the world changing, not the model. Isolating pure model stochasticity would need dense same-day repeat runs.
  • Consistency (RBO) is a comparative scale. Top-weighted overlap on short lists runs low in absolute terms - use it to rank engines, not as a "70% = good" gauge.
  • Extraction is deterministic string-matching against Rankscale's alias lists (first-mention order = position), not a language model - fast and reproducible, with minor matching noise.

Data: De'Longhi (US), pulled once via POST /v1/metrics/search-terms-report using the rankscale-api-skill (all 63 search terms, answer texts included), analyzed offline. The same workspace's standard metrics and Brand Rank tables are on the worked examples page. Reproducible against the same workspace.