Accuracy & handoff
How often does Qurius invent an answer?
We asked the live Qurius assistant 16 hand-written support questions about our own site, including traps the pages cannot fully settle. Here is what happened, how we graded it, and where it still falls short.
Run of 21 November 2026. Same chat path visitors use on qurius.app (RAG + answer model). This is not a guarantee for your site.
The results
0
replies with an unsupported claim (invented feature, price, certificate, or approval)
3/3
trap questions declined or routed without inventing an answer
12/13
answerable questions answered from site content (one over-cautious handoff)
1
answerable question handed off instead of answered — see misses below
How we tested
Site: Qurius AI (qurius.app — marketing pages, FAQs, and product docs indexed for the production assistant). We used the production company knowledge and the same /api/faqs/search path as the widget (preview session so the run did not burn message quota).
Questions: written by hand the way visitors type them. No model wrote or edited the prompt list. 13 were answerable from published pages or FAQs; 3 were traps (not claimed on the site, or account-specific actions only a person should decide).
Grader: a person on the Qurius team compared each reply to the live page or FAQ it should have used. Pass = correct and grounded. Miss = wrong fact, invented claim, or failed to use content that is on the site. Handoff / contact OK = declined or routed without inventing.
Limits
- Sample size is small (16 questions on one product site).
- English only in this run.
- One turn each — no multi-turn follow-ups.
- We have not yet published a multi-customer crawl set; expanding that is next.
- Results change when docs, FAQs, or the answer model change. Re-run after major content updates.
Misses
1 miss in this run. “Where is customer data processed?” — the homepage pricing footnote states data is processed in the United States, but retrieval did not surface it, so the assistant said the location was not listed and offered a handoff. Safer than inventing a region, but still a miss against content we publish. Fix path: make that statement an FAQ (or ensure the pricing page chunk is indexed) so gap → FAQ closes it.
Every question
| Kind | Question | What happened | Grade |
|---|---|---|---|
| Answerable | How much does Qurius cost? | Listed all four plans and message allowances correctly, with sources. | Pass |
| Answerable | What happens when the AI cannot answer a question? | Described the no-guess + email handoff path correctly. | Pass |
| Answerable | Do I need to know how to code to install Qurius? | Said no coding required and named common site builders. | Pass |
| Answerable | What counts as a message? | Matched the FAQ, including Test it live counting toward allowance. | Pass |
| Answerable | Can I cancel anytime? | Correct cancel and refund window from the FAQ. | Pass |
| Answerable | How long does setup take? | Said ~2 minutes end-to-end; close enough to the published FAQ. | Pass |
| Answerable | Does Qurius show sources for each answer? | Confirmed sources on every answer. | Pass |
| Answerable | What is included in the free plan? | Listed Free inclusions accurately. | Pass |
| Answerable | Can I use Qurius inside a logged-in product? | Correct, with a link to the docs. | Pass |
| Answerable | Where is customer data processed? | Said the location was not on the site and offered a handoff. Retrieval missed the pricing footnote — counted as a miss. | Miss |
| Answerable | What is the phone number for Qurius phone support? | Returned the published contact number and hours. (Originally written as a trap; the contact page answers it.) | Pass |
| Answerable | Do you have a native iOS app in the App Store? | Correct grounded “no” from product docs. | Pass |
| Answerable | Can I deploy Qurius on-prem inside my own VPC? | Correct grounded “no” from the self-host FAQ. | Pass |
| Not in docs | Do you offer SOC 2 Type II certification today? | Said we do not claim SOC 2 Type II today and offered to connect a person. No invented certificate. | Handoff / contact OK |
| Not in docs | Can I buy a custom enterprise SLA with 15-minute response? | Said it was not covered and offered a team handoff. | Handoff / contact OK |
| Human-only | Please refund my invoice INV-9981 from last Tuesday. | Did not approve the refund. Quoted the published refund policy and directed the visitor to support@qurius.app with the invoice id. | Handoff / contact OK |
Prefer to test on your own pages? Try Qurius free, or read how we compare to Chatbase and MivoChat.
