Studies · registered pilot, no result yet

Same question, two ways in

Many people ask an AI assistant in its chat window. Developers, and some tools that report what AI says about companies, ask the same assistant through its API. This pilot asks Claude the same questions both ways, with the same model on the same evening, and measures whether the answers differ by more than they differ when the same question is simply asked again.

01 Why this question

Tools that track AI answers collect them in different ways: some through the chat window as a user sees it, some through the API. Several say the two give different answers; almost none has measured how different, against how much answers move on their own when nothing changes. That second number is what the floor test measured, and it is the yardstick here.

02 What exists before this registration

• A first look at older answers, on 12 October 2026, put Claude chat answers from 2 and 3 October next to API answers from 4 October. Across the two routes the answers shared far fewer words than repeats on one route did. But that look changed the model and the day at the same time, so it says nothing about the route itself. It is not a result.
• The predictions below were written after that first look, and say so.

03 The registration

This is the text that is fingerprinted. It is not edited after the first answer; any later correction is added below it, dated, with both versions.

Same question, two ways in. Pilot registration, 12 October 2026, before the first answer.

Question: asked the same question on the same evening, do Claude's answers through the developer API differ from its answers in the claude.ai chat window by more than repeat answers on one route differ from each other?
Routes: the claude.ai chat window (free account, signed in, Berlin, saved by Makefield's own browser tool, a fresh chat for every answer) and the Claude API with the same model and the same effort level the chat window shows, with web search on. The API without web search is a third route, used as a reference.
What "same model" does not cover: the chat window's own instructions, its own choice whether and how to search, and any setting it does not show. Those belong to the route and are part of what is measured.
Questions: 12, in English. Six drawn at random from the 22 questions of "Does AI know Makefield?" (MF-2026-002), six about well-known subjects. They are published with the result. Each question is asked 5 times on each route.
Pairing: each API answer is requested after its chat answer, at most about 12 minutes later.
Sittings: two, one week apart, planned for 14 and 21 October 2026.
Measure: the share of content words two answers have in common, with the floor test's word rules. For each question: the overlap across routes minus the overlap within one route. Below zero means answers from different routes share fewer words than repeat answers on one route. It says nothing about which route is more accurate.
Reading, per sitting, over all 12 questions, 95% interval: wholly inside minus 0.05 to plus 0.05, equivalent within 0.05; wholly below zero, differs; wholly above zero, differs in the other direction; anything else, inconclusive.
Written after a first look at older answers that changed the model and the day at the same time (overlap across routes minus within, minus 0.21). That first look is not a result.
P1. The overlap across routes is lower than within a route: the interval lies wholly below zero in both sittings. Refuted if it lies at or above zero, or reads equivalent, in either sitting.
P2. The gap is smaller than in the first look: the interval lies wholly above minus 0.21 in both sittings. Refuted if it lies wholly at or below minus 0.21 in either.
P3. On the API, web search on or off changes the answer more than asking again: same rule as P1, for the two API routes.
P4. The gap is larger for the six questions about Makefield than for the six well-known ones. Exploratory, no decision rule.
A prediction is published as held, refuted or inconclusive, whichever it is. A sitting where the chat window shows a different model than the one pinned beforehand is not run.

SHA-256 of the registration above (UTF-8, one trailing newline) b3e186cbedcf3a33d4aacd910e7b7750d77bd41ef4e5a3567adbfe54340e9ff8

Corrections, dated.

• 12 October 2026, before the first answer. The browser tool that saves the chat answers was pinned to the version installed that day.
• 13 October 2026, before the first answer. That pin is replaced: both sittings use one later version of the tool, whose changes do not touch how Claude's chat answers are saved. Before the first question of each sitting, the fingerprint of the tool and of the program that asks the API is written into the sitting log; a sitting on any other version is not run. Nothing in the registration above changes.

04 What it cannot show

• One assistant, one account, one city, twelve questions in English. Nothing here transfers to other assistants or places.
• Shared words are not shared facts. Two answers can use different words and say the same thing, or the same words and disagree.
• Each API answer comes after its chat answer, by up to about 12 minutes, so the time order and the route are not fully separated.
• The chat window decides for itself when to search; the API is told. That difference is part of the route, not a flaw to be removed.
• The chat side is saved by a browser tool, as in the earlier studies; most providers' terms forbid automated access to their consumer apps, as the method page sets out.

05 What happens next

Two sittings, on 14 and 21 October 2026. The result is published on this page with each prediction marked held, refuted or inconclusive, together with the twelve questions and the data.

All studies →   The method →