Accuracy benchmark

Five models. One filed policy. Every quote checked.

20 questions a homeowner would actually ask, answered against 261 clauses pulled from real filed Texas FAIR Plan forms. For each question we recorded, in advance, the clause that genuinely decides it. Then we checked two things: did the answer find that clause, and was every word it put in quotation marks really in the policy.

85%
of governing clauses PoliSays found — the highest of the five
0
quotes PoliSays made up
12
quotes Claude made up on the same questions

Last run 2026-08-29 02:37 UTC · generated by our benchmark script, not written by hand

  • 1
    PoliSaysOurs

    Retrieval + rerank, then the same Gemini model

    85%17/20
    none
  • 2
    GPT

    Whole document pasted in, answer with citations

    80%16/20
    4made up
  • 3
    Gemini

    Whole document pasted in, answer with citations

    70%14/20
    1made up
  • 4
    Grok

    Whole document pasted in, answer with citations

    65%13/20
    7made up
  • 5
    Claude

    Whole document pasted in, answer with citations

    70%14/20
    12made up

The four chatbots were handed the entire 284k-character policy — their best case. PoliSays answered from 5% of it, only the clauses its retrieval picked, and still found the most governing clauses with nothing made up. The other four invented 24 quotations between them.

Fabrication watch

24

sentences quoted that the policy never contained

Each one reads exactly like a real citation. That is the danger: a reader checking their coverage cannot tell the invented quote from the genuine one, and an adjuster will only ever quote the real document.

0 of them came from PoliSays

GPT4 made up
Gemini1 made up
Grok7 made up
Claude12 made up
  • GPT invented thison: Water has been slowly leaking under my sink for months and now the cabinet is rotted. Is that covered?
    We do not cover loss caused directly or indirectly by or resulting from: c. constant, repeated or intermittent discharge, seepage, leakage, collection, overflow or release of water or steam from any source, including but not limited to any pipe joint, plumbing fixture, shower stall, shower bath, bathtub, bathtub installation, drain line, drain pan, air conditioning overflow pan, duct work, or appliance;

    Attributed to Endorsement No. (FAIR) HO-400 — Water Damage — not found in the 261 clauses of the filed forms.

  • GPT invented thison: Water has been slowly leaking under my sink for months and now the cabinet is rotted. Is that covered?
    We do not cover loss caused directly or indirectly by or resulting from: a. rust, decay, wear and tear, deterioration or loss caused by any quality in property that causes it to damage or destroy itself;

    Attributed to Endorsement No. (FAIR) HO-400 — Wear And Tear, Inherent Vice, Rust, Deterioration, Decay, Rats, Mice, Vermin, Termites, Moths or Other Insects, Contamination, Neglect, Improper Maintenance, Failure To Make Proper Repairs, Failure To Make Timely Repairs — not found in the 261 clauses of the filed forms.

  • Grok invented thison: Water has been slowly leaking under my sink for months and now the cabinet is rotted. Is that covered?
    10. Water Damage We do not cover loss caused directly or indirectly by or resulting from: c. constant, repeated or intermittent discharge, seepage, leakage, collection, overflow or release of water or steam from any source, including but not limited to any pipe joint, plumbing fixture, shower stall, shower bath, bathtub, bathtub installation, drain line, drain pan, air conditioning overflow pan, duct work, or appliance;

    Attributed to ENDORSEMENT NO. HO-400 Sudden and Accidental Discharge, Eruption, Overflow or Release of Water or Steam — not found in the 261 clauses of the filed forms.

  • Grok invented thison: Water has been slowly leaking under my sink for months and now the cabinet is rotted. Is that covered?
    14. Wear And Tear, Inherent Vice, Rust, Deterioration, Decay, Rats, Mice, Vermin, Termites, Moths or Other Insects, Contamination, Neglect, Improper Maintenance, Failure To Make Proper Repairs, Failure To Make Timely Repairs. We do not cover loss caused directly or indirectly by or resulting from: a. rust, decay, wear and tear, deterioration or loss caused by any quality in property that causes it to damage or destroy itself;

    Attributed to ENDORSEMENT NO. HO-400 Sudden and Accidental Discharge, Eruption, Overflow or Release of Water or Steam — not found in the 261 clauses of the filed forms.

  • Claude invented thison: Water has been slowly leaking under my sink for months and now the cabinet is rotted. Is that covered?
    We do not cover loss caused directly or indirectly by or resulting from: ... constant, repeated or intermittent discharge, seepage, leakage, collection, overflow or release of water or steam from any source, including but not limited to any pipe joint, plumbing fixture, shower stall, shower bath, bathtub, bathtub installation, drain line, drain pan, air conditioning overflow pan, duct work, or appliance

    Attributed to Endorsement HO-400, Water Damage exclusions — not found in the 261 clauses of the filed forms.

  • Claude invented thison: Water has been slowly leaking under my sink for months and now the cabinet is rotted. Is that covered?
    We do not cover loss caused directly or indirectly by or resulting from: ... neglect, improper maintenance, the failure to make proper repairs or the failure to make timely repairs.

    Attributed to Endorsement HO-400, Wear and Tear exclusion — not found in the 261 clauses of the filed forms.

  • Gemini invented thison: Ice cracked my patio and part of the fence. Does the policy pay for that?
    e. freezing, thawing, pressure or weight of water or ice, whether driven by wind of not, to a: i. Fence, pavement, patio or swimming pool or spa;

    Attributed to TO THE COVERAGE PROVIDED BY THIS ENDORSEMENT. — not found in the 261 clauses of the filed forms.

  • Grok invented thison: Ice cracked my patio and part of the fence. Does the policy pay for that?
    e. freezing, thawing, pressure or weight of water or ice, whether driven by wind of not, to a: i. Fence, pavement, patio or swimming pool or spa;

    Attributed to Endorsement HO-400 — not found in the 261 clauses of the filed forms.

  • Grok invented thison: Ice cracked my patio and part of the fence. Does the policy pay for that?
    11. Settling, Cracking, Bulging, Shrinkage, Or Expansion. We do not cover loss caused directly or indirectly by or resulting from settling, cracking, bulging, shrinkage, or expansion of foundations, foundation piers, foundation beams, slabs, walls, floors, ceilings, roof structures, walks drives, curbs, fences, retaining walls, bulkheads, spas, or swimming pools, regardless of the cause.

    Attributed to Endorsement HO-400 — not found in the 261 clauses of the filed forms.

  • Claude invented thison: Ice cracked my patio and part of the fence. Does the policy pay for that?
    freezing, thawing, pressure or weight of water or ice, whether driven by wind of not, to a: i. Fence, pavement, patio or swimming pool or spa

    Attributed to TO THE COVERAGE PROVIDED BY THIS ENDORSEMENT - Exclusion 10.e. — not found in the 261 clauses of the filed forms.

  • Claude invented thison: Ice cracked my patio and part of the fence. Does the policy pay for that?
    Settling, Cracking, Bulging, Shrinkage, Or Expansion. We do not cover loss caused directly or indirectly by or resulting from settling, cracking, bulging, shrinkage, or expansion of foundations, foundation piers, foundation beams, slabs, walls, floors, ceilings, roof structures, walks drives, curbs, fences, retaining walls, bulkheads, spas, or swimming pools, regardless of the cause.

    Attributed to TO THE COVERAGE PROVIDED BY THIS ENDORSEMENT - Exclusion 11 — not found in the 261 clauses of the filed forms.

  • Claude invented thison: Ice cracked my patio and part of the fence. Does the policy pay for that?
    Outdoor Equipment, Fences, Driveways, Walks, Lawns, Trees, Shrubs And Plants Or Retaining Walls And Bulkheads. We do not cover loss caused directly or indirectly to the following: a. driveways, walks, lawns, trees, shrubs and plants or retaining walls and bulkheads not a part of the building; b. outdoor equipment or fences.

    Attributed to TO THE COVERAGE PROVIDED BY THIS ENDORSEMENT - Exclusion 12 — not found in the 261 clauses of the filed forms.

  • Grok invented thison: My wedding ring was stolen. How much can I actually get back?
    Our limit of liability and payment for covered losses under Section I will not exceed the smallest of the following: a. the actual cash value; or b. the specified limit of liability of the policy.

    Attributed to SECTION I – CONDITIONS 4. Loss Settlement — not found in the 261 clauses of the filed forms.

  • GPT invented thison: My laptop was taken out of my locked car in a parking lot. Is that a claim?
    personal property while away from the residence premises and unattended in or on any motor vehicle or trailer, other than a public conveyance, unless all its doors, windows and other openings are closed and locked and there are visible marks or forcible entry.

    Attributed to TFPA Homeowners Policy — Section I – Perils Insured Against, Coverage B (Personal Property), Theft — not found in the 261 clauses of the filed forms.

  • Claude invented thison: If a nuclear accident somehow starts a fire that burns my house, does the fire damage get paid?
    We do not cover loss resulting directly or indirectly from nuclear reaction, radiation or radioactive contamination, all whether controlled or uncontrolled or however caused. We cover loss by fire resulting from nuclear reaction, radiation or radioactive contamination.

    Attributed to GENERAL EXCLUSIONS, Section 5. NUCLEAR DAMAGE — not found in the 261 clauses of the filed forms.

  • Claude invented thison: My house has been sitting empty for three months while I'm overseas and now there's damage — will that be covered?
    "Vacant" means: a. the insured moves from the dwelling and a substantial part of the personal property is removed from that dwelling; or b. the dwelling is unoccupied by the insured; or c. the dwelling lacks the necessary amenities, adequate furnishings, utilities and services to permit occupancy of the dwelling as a residence.

    Attributed to DEFINITIONS, item 14 - Vacant — not found in the 261 clauses of the filed forms.

  • Grok invented thison: My washing machine hose burst and water shot out inside the house — is that kind of sudden water release covered under a special endorsement?
    9. Sudden and Accidental Discharge, Eruption, Overflow or Release of Water or Steam from: ... d. within any household appliance.

    Attributed to ENDORSEMENT NO. HO-400 — not found in the 261 clauses of the filed forms.

  • Grok invented thison: My washing machine hose burst and water shot out inside the house — is that kind of sudden water release covered under a special endorsement?
    10. Water Damage We do not cover loss caused directly or indirectly by or resulting from: ... c. constant, repeated or intermittent discharge, seepage, leakage, collection, overflow or release of water or steam from any source, including but not limited to any pipe joint, plumbing fixture, shower stall, shower bath, bathtub, bathtub installation, drain line, drain pan, air conditioning overflow pan, duct work, or appliance;

    Attributed to ENDORSEMENT NO. HO-400 — not found in the 261 clauses of the filed forms.

  • Claude invented thison: My washing machine hose burst and water shot out inside the house — is that kind of sudden water release covered under a special endorsement?
    Sudden and Accidental Discharge, Eruption, Overflow or Release of Water or Steam from: ... d. within any household appliance.

    Attributed to Endorsement HO-400, Coverage B (Personal Property), Peril 9 — not found in the 261 clauses of the filed forms.

  • Claude invented thison: When they calculate my payout for older damaged items, do they subtract for wear and age, or pay full replacement cost?
    Actual cash value is calculated as the amount it would cost us to repair or replace the damaged part of covered property, at the time of loss or damage, with material of like kind and quality and for the same use on the premises shown on the declarations page, subject to a deduction for deterioration and depreciation.

    Attributed to DEFINITIONS — not found in the 261 clauses of the filed forms.

  • Claude invented thison: After my house partially burned last year, is my remaining coverage amount lower now until I finish repairs?
    Each time there is a loss to any building insured under Coverage A (Dwelling), the amount of insurance applicable to that building for loss by fire will be reduced by the amount of the loss. As repairs are made, the amount of insurance will be reinstated up to the limit of liability as shown on the declarations page.

    Attributed to SECTION I – CONDITIONS, Insurable Interest and Limit of Liability — not found in the 261 clauses of the filed forms.

  • Claude invented thison: My old refrigerator broke from a covered surge — will they pay me a brand-new replacement cost or something less?
    Our limit of liability and payment for covered losses under Section I will not exceed the smallest of the following: a. the actual cash value; or b. the specified limit of liability of the policy.

    Attributed to SECTION I – CONDITIONS, Loss Settlement (item 4) — not found in the 261 clauses of the filed forms.

  • Claude invented thison: My old refrigerator broke from a covered surge — will they pay me a brand-new replacement cost or something less?
    "Actual cash value" is calculated as the amount it would cost us to repair or replace the damaged part of covered property, at the time of loss or damage, with material of like kind and quality and for the same use on the premises shown on the declarations page, subject to a deduction for deterioration and depreciation.

    Attributed to DEFINITIONS, Actual cash value (item 1) — not found in the 261 clauses of the filed forms.

  • GPT invented thison: My neighbor got hurt on my sidewalk but didn't need surgery until almost two years later — is that still covered under my medical payments coverage?
    Medical expenses means reasonable charges for medical, surgical, xray, dental, ambulance, hospital, professional nursing, prosthetic devices and funeral services.

    Attributed to TFPA Homeowners Policy — SECTION II – LIABILITY COVERAGE — not found in the 261 clauses of the filed forms.

Poli can only quote text that was retrieved from your uploaded document, so a quote it shows is a quote that exists. When it cannot find governing wording it says so.

Question by question

Tap a question to read every answer, its citations and anything it invented.

How the scoring works

Every contestant gets the identical question. The chatbots get the entire policy pasted in, which is the best case for them and the way a reader would use them. PoliSays gets only the clauses its retrieval pipeline picked and then answers with the same Gemini model, so the comparison isolates the retrieval, not the writing.

Points are two for finding the pre-labelled governing clause, one for citing anything at all, minus one for every quote that does not appear in the document, minus one for every dollar amount or deadline the document never states. A run is only a snapshot of these 20 questions on these forms — it is evidence, not a guarantee, and PoliSays remains an explanation tool rather than advice.