Field note

Why I never type 'search the web' into an AI visibility test

The instinct to add 'search the web' to the prompt is the thing that breaks the test.

Denes Csaszar · Makefield · Berlin · not yet published

When you ask an AI assistant about your brand, sometimes it answers from memory — stale, generic, clearly not looking anything up. The obvious fix is to add "search the web" to your question and ask again. It's a reasonable instinct. It also quietly ruins the test.

Two things break at once.

First, you've changed the question. If you're comparing several engines — and you should be, because they don't agree — the wording has to be identical across all of them, or you're no longer comparing the same thing. "What does AI say about us?" and "What does AI say about us? Search the web." are two different questions.

Second, the engines don't behave alike. Some search on their own; some prefer to answer from memory for anything they think they already know. So if you add the instruction only to the ones that need it, every engine ends up getting a slightly different prompt — and any difference in the answers might be the wording you added, not the engine itself.

The fix is to treat it as two separate dials, set outside the question:

• Personalisation and memory: off. So the engine isn't answering from your own history (that's its own trap).
• Web search: on, using the engine's own toggle or mode — never by typing it into the prompt. The question stays word-for-word identical everywhere; "search was on" becomes a condition you record, not text you add.

There's a deeper reason this matters. Whether an engine chooses to search is part of its behaviour — part of what you're actually measuring. A real buyer doesn't force it to search either; they get whatever it decides to do. So the honest move is to note what each engine did — searched, or answered from memory — and leave the question alone.

One caveat: a few engines only search when asked in the text and give you no toggle. Don't fight it — write that down as that engine's quirk, a constant you note, rather than patching it with extra words that wreck the comparison everywhere else.

Keeping the question fixed is what makes an audit reproducible — see also ask the same question twice, get two answers.