Studies · an example, invented
An example reading
Readings of named companies are not published here. This page shows a complete reading with every part a real one has: the scope, the locked ground truth, the answers, the scoreboard and the fingerprint. The product, the engines and every answer are invented. Nothing here is a result.
01 Scope, before the answers
One product: a portable speaker, invented for this page. Three questions, asked of seven answer engines, anonymised A to G, in two languages (English and German), three draws per question, engine and language. That is 126 answers. All of them would be captured in one sitting, from one account and one location, and that limit belongs on the page before any number does.
02 The locked ground truth
The correct answer to each question is written down, with its source of record, before the first question is asked. Nothing is scored against a truth decided afterwards.
| # | Question | Locked truth | Source of record |
|---|---|---|---|
| Q1 | Is the device waterproof? | No. Rated IP54: protected against splashing water, not against immersion. | manufacturer specification sheet |
| Q2 | How long does the battery last? | Up to 12 hours at medium volume. | manufacturer specification sheet |
| Q3 | Does it work without the app? | Yes. Bluetooth pairing works without the app; the app is needed only for firmware updates. | manufacturer user manual |
03 The answers
Every answer is kept and scored against the locked truth on one scale: 0 correct, +1, +2 an overclaim, −1 an omission, −2 a false claim, −3 a false claim that cites the source refuting it. Five of the 42 answers to Q1, as they would appear in the transcript:
| Engine | Lang. | Draw | Answer | Score |
|---|---|---|---|---|
| A | EN | 1 | “No. It is rated IP54, which means splash-resistant, not waterproof.” | 0 · Correct |
| A | DE | 2 | “Yes, it is waterproof, as confirmed by the manufacturer's specification sheet.” | −3 · False claim, citing the source that refutes it |
| C | EN | 1 | “I couldn't find reliable information on whether this device is waterproof.” | −1 · Omission |
| D | EN | 3 | “No, it is rated IP54, but it is certified for use in heavy rain and snow.” | +2 · Overclaim |
| E | DE | 1 | “Ja, es ist wasserdicht, wie das Datenblatt des Herstellers bestätigt.” | −3 · False claim, citing the source that refutes it |
German answers are shown in the original or in English translation; a real transcript keeps the original text.
04 The scoreboard
Per engine and question: correct answers out of six (three draws in each of two languages), and the worst score among the six.
| Engine | Q1 waterproof | Q2 battery | Q3 without app | All |
|---|---|---|---|---|
| A | 2/6 −3 | 4/6 +1 | 6/6 0 | 12/18 |
| B | 6/6 0 | 5/6 −1 | 4/6 −1 | 15/18 |
| C | 1/6 −2 | 3/6 −1 | 2/6 −2 | 6/18 |
| D | 1/6 −2 | 2/6 +2 | 5/6 +1 | 8/18 |
| E | 0/6 −3 | 4/6 −2 | 3/6 −2 | 7/18 |
| F | 2/6 −2 | 6/6 0 | 4/6 −1 | 12/18 |
| G | 3/6 −3 | 3/6 +1 | 6/6 0 | 12/18 |
| All | 15/42 | 27/42 | 30/42 | 72/126 |
Read off the board, in the invented data: one engine (B) is right on Q1 every time, one (E) never is, and Q1 is the hardest question, with 15 of 42 answers correct. The same engine can be right in one language and wrong in the other: engine A is right on Q1 in two of three English draws and wrong in all three German ones.
05 What this reading cannot say
- Why an engine answered as it did. A reading records what was said, not the cause.
- Whether the answers hold in another sitting. Three draws inside one sitting measure how much answers move there; the floor test is about the difference between sittings.
- Anything about engines in general. Three questions about one product are a reading of that product, not a ranking of engines.
06 The fingerprint
A real reading ships with its frozen results table and the SHA-256 of that table, so anyone holding the file can check that no number moved after publication. Here is this page’s table, and its fingerprint:
question,engine,correct_of_6,worst_score Q1,A,2,-3 Q1,B,6,0 Q1,C,1,-2 Q1,D,1,-2 Q1,E,0,-3 Q1,F,2,-2 Q1,G,3,-3 Q2,A,4,1 Q2,B,5,-1 Q2,C,3,-1 Q2,D,2,2 Q2,E,4,-2 Q2,F,6,0 Q2,G,3,1 Q3,A,6,0 Q3,B,4,-1 Q3,C,2,-2 Q3,D,5,1 Q3,E,3,-2 Q3,F,4,-1 Q3,G,6,0
SHA-256 of the table above (UTF-8, one trailing newline) 7630aa25f2612d996ae6ded5dbf133327a9336babec663b35d4587060317b282