PoliTrust

The only AI that never fakes a quote.

Poli against GPT, Gemini, Claude and Grok: 20 real homeowner questions on a real filed Texas policy. Poli scored highest overall, and every word she quoted is really in the policy. The others made up quotes.

  • Zero made-up quotes
  • Highest overall score
  • Every word checked
Poli beside five clay answer cards — four marked wrong or unsure, one marked correct with a quoted clause
85%
of the deciding clauses PoliSays found, with every quote real
0
quotes PoliSays made up
12
quotes Claude made up on the same questions

Last run 2026-09-16 22:34 UTC · generated by our benchmark script, not written by hand

Latest automated re-run · 2026-10-02 — same 20 questions, re-run on its own every two weeks. PoliSays 18/20, 0 made-up · Gemini 18/20, 2 made-up · Claude 18/20, 8 made-up · GPT 15/20, 3 made-up · Grok 16/20, 5 made-up
  • 1
    PoliSaysOurs

    Finds the right clauses first, then answers with the same Gemini model

    85%17/20
    none
  • 2
    Gemini

    Handed the whole policy, asked to quote its answer

    90%18/20
    2made up
  • 3
    GPT

    Handed the whole policy, asked to quote its answer

    85%17/20
    4made up
  • 4
    Grok

    Handed the whole policy, asked to quote its answer

    60%12/20
    8made up
  • 5
    Claude

    Handed the whole policy, asked to quote its answer

    85%17/20
    12made up
Five clay answer cards on a desk — a question mark, a guess, one cited answer, an incomplete answer and a wrong answer

Same question

Five answers. One of them quotes the policy.

The others guess, hedge, or invent wording. We keep score on the only two things that matter: did it find the clause that decides the answer, and is every quoted word really in the document?

Why asking a chatbot is risky

All 4 of the big AI chatbots put words in this policy's mouth.

Not a loose paraphrase you could argue with — whole sentences printed inside quotation marks, with a section number attached, that appear nowhere in the real policy. It happened on 12 of 20 questions. Nothing on screen tells you which quote is real, and your insurer will only ever go by what the document actually says.

26
made-up quotes across one 20-question set
12/20
questions where at least one chatbot made a quote up
12
from Claude alone — and it sounded just as sure
0
from PoliSays, which can only quote what it found in the file

Vague vs cited

A confident paragraph is not an answer.

An answer you can act on names the section, quotes the sentence, and shows you where to read it yourself. That is the whole test.

Poli between a vague clay answer bubble marked wrong and a cited clay answer marked correct with numbered steps

The four chatbots were handed the entire 284k-character policy — their best case. PoliSays answered from 5% of it — only the clauses it picked out as relevant — and still found the most deciding clauses, with nothing made up. The other four made up 26 quotes between them.

Every made-up quote, listed

26

sentences quoted that the policy never contained

Each one reads exactly like a real citation. That is the danger: a reader checking their coverage cannot tell the invented quote from the genuine one, and an adjuster will only ever quote the real document.

0 of them came from PoliSays

Gemini2 made up
GPT4 made up
Grok8 made up
Claude12 made up

Every one of them is listed under its own AI in the tabs below.

Poli can only quote text that was retrieved from your uploaded document, so a quote it shows is a quote that exists. When it cannot find governing wording it says so.

Our track record

Every test we have ever run

Drag the slider to any test. Nothing is cleaned up after the fact — our earlier, worse results stay on the record, because that is the only way a number like this means anything.

Run 8 of 8 · 2026-09-16 22:34 UTC

85%

of the deciding clauses PoliSays found in this test

0

quotes PoliSays made up

26

quotes the big AI chatbots made up on the same questions

  • PoliSays
    85% no made-up quotes
  • Gemini
    90%2 made up
  • GPT
    85%4 made up
  • Grok
    60%8 made up
  • Claude
    85%12 made up

What we changed before this test: Sept refresh

Results, one AI at a time

Pick an AI, then tap a question to read its answer.

PoliSays

Finds the right clauses first, then answers with the same Gemini model

85%
deciding clauses found (17/20)
0
quotes it made up
53
points

Every quote PoliSays gave was found in the document first.

All 20 questions

Poli holding a magnifier over a clay policy with highlighted lines and two quoted callouts

How we score it

Everyone gets the same question. The four chatbots get the whole policy pasted in — their best case, and the way most people would use them. PoliSays only sees the clauses it picked out, then answers with the same Gemini model the others use. So the difference is in finding the right wording, not in the writing.

Scoring: two points for finding the clause we wrote down in advance as the one that decides the answer, one for quoting anything at all, minus one for every quote that isn't in the document, and minus one for every dollar amount or deadline the document never states. This is one run of 20 questions on one policy — strong evidence, not a guarantee. PoliSays explains documents; it does not give advice.