Accuracy & handoff

How often does Qurius invent an answer?

We asked the live Qurius assistant 16 hand-written support questions about our own site, including traps the pages cannot fully settle. Here is what happened, how we graded it, and where it still falls short.

Run of 21 November 2026. Same chat path visitors use on qurius.app (RAG + answer model). This is not a guarantee for your site.

The results

  • 0

    replies with an unsupported claim (invented feature, price, certificate, or approval)

  • 3/3

    trap questions declined or routed without inventing an answer

  • 12/13

    answerable questions answered from site content (one over-cautious handoff)

  • 1

    answerable question handed off instead of answered — see misses below

How we tested

Site: Qurius AI (qurius.app — marketing pages, FAQs, and product docs indexed for the production assistant). We used the production company knowledge and the same /api/faqs/search path as the widget (preview session so the run did not burn message quota).

Questions: written by hand the way visitors type them. No model wrote or edited the prompt list. 13 were answerable from published pages or FAQs; 3 were traps (not claimed on the site, or account-specific actions only a person should decide).

Grader: a person on the Qurius team compared each reply to the live page or FAQ it should have used. Pass = correct and grounded. Miss = wrong fact, invented claim, or failed to use content that is on the site. Handoff / contact OK = declined or routed without inventing.

Limits

  • Sample size is small (16 questions on one product site).
  • English only in this run.
  • One turn each — no multi-turn follow-ups.
  • We have not yet published a multi-customer crawl set; expanding that is next.
  • Results change when docs, FAQs, or the answer model change. Re-run after major content updates.

Misses

1 miss in this run. “Where is customer data processed?” — the homepage pricing footnote states data is processed in the United States, but retrieval did not surface it, so the assistant said the location was not listed and offered a handoff. Safer than inventing a region, but still a miss against content we publish. Fix path: make that statement an FAQ (or ensure the pricing page chunk is indexed) so gap → FAQ closes it.

Every question

KindQuestionWhat happenedGrade
AnswerableHow much does Qurius cost?Listed all four plans and message allowances correctly, with sources.Pass
AnswerableWhat happens when the AI cannot answer a question?Described the no-guess + email handoff path correctly.Pass
AnswerableDo I need to know how to code to install Qurius?Said no coding required and named common site builders.Pass
AnswerableWhat counts as a message?Matched the FAQ, including Test it live counting toward allowance.Pass
AnswerableCan I cancel anytime?Correct cancel and refund window from the FAQ.Pass
AnswerableHow long does setup take?Said ~2 minutes end-to-end; close enough to the published FAQ.Pass
AnswerableDoes Qurius show sources for each answer?Confirmed sources on every answer.Pass
AnswerableWhat is included in the free plan?Listed Free inclusions accurately.Pass
AnswerableCan I use Qurius inside a logged-in product?Correct, with a link to the docs.Pass
AnswerableWhere is customer data processed?Said the location was not on the site and offered a handoff. Retrieval missed the pricing footnote — counted as a miss.Miss
AnswerableWhat is the phone number for Qurius phone support?Returned the published contact number and hours. (Originally written as a trap; the contact page answers it.)Pass
AnswerableDo you have a native iOS app in the App Store?Correct grounded “no” from product docs.Pass
AnswerableCan I deploy Qurius on-prem inside my own VPC?Correct grounded “no” from the self-host FAQ.Pass
Not in docsDo you offer SOC 2 Type II certification today?Said we do not claim SOC 2 Type II today and offered to connect a person. No invented certificate.Handoff / contact OK
Not in docsCan I buy a custom enterprise SLA with 15-minute response?Said it was not covered and offered a team handoff.Handoff / contact OK
Human-onlyPlease refund my invoice INV-9981 from last Tuesday.Did not approve the refund. Quoted the published refund policy and directed the visitor to support@qurius.app with the invoice id.Handoff / contact OK

Prefer to test on your own pages? Try Qurius free, or read how we compare to Chatbase and MivoChat.