You cannot fully trust what an AI assistant tells your customers. It will sometimes give a confident, fluent, completely wrong answer, and no supplier can promise you otherwise. We build these and we will not promise it. Anyone who does is either misunderstanding the technology or hoping you will not test the claim.
That sounds like a reason not to build one. It is not, and the reason is worth understanding, because the same objection applies to every member of staff you have ever hired and you solved it there without much difficulty.
Why an assistant gives answers you cannot trust
The plain version, without the jargon. These systems are built to produce text that fits. Given a question, they assemble the most plausible-looking continuation. Usually the most plausible answer is also the true one, which is why they work at all. Sometimes it is not, and nothing in the machinery distinguishes between the two cases, because the system has no concept of true, only of fitting.
That is why the wrong answers are so unsettling. They do not look like errors. There is no hesitation, no hedging, no obvious tell. A human who does not know something usually sounds like a human who does not know something. This does not.
So the design problem is not making it always right. It is making the cases where it might be wrong land somewhere safe.
What actually reduces it
A great deal, done properly. These are not marketing claims, they are the difference between an assistant that is broadly reliable and one that improvises.
- Ground it in your own material. Instead of answering from general knowledge, the assistant retrieves the relevant passage from your documentation and answers from that. This is the single biggest improvement available, and it changes the failure mode from inventing to misreading, which is far less dangerous.
- Make it cite what it used. If the answer shows which page it came from, a customer can check and your team can audit. It also constrains the system, because an answer with nothing to point at is visibly weaker.
- Let it say it does not know. Sounds trivial and takes deliberate work, because the default behaviour is to produce something. An assistant that says it cannot answer and offers a human is doing its job.
- Narrow the scope. An assistant that only answers questions about orders and delivery is dramatically more reliable than one that answers anything. Breadth is where accuracy goes.
- Keep the material true. The most common wrong answer we see is not invented at all: it is a correct answer from a page nobody updated when the price changed. Grounding only helps if the ground is solid.
- Read the transcripts weekly. Not a dashboard. Actual conversations, by a person who knows the business. Everything above degrades without this.
Design for the day it is wrong
This is the part that separates a considered deployment from a risky one, and it is entirely within your control.
| If a wrong answer would | Then |
|---|---|
| Mildly annoy someone | Let it answer directly, with an obvious route to a human on every turn. Most support questions are here. |
| Cost you a sale or a complaint | Let it draft and have a person send. All of the speed, almost none of the risk, and your team stops retyping the same answers. |
| Create an obligation, a price or a commitment | Do not let it state the figure. Have it gather the details and hand over. A quote is an offer, and an offer generated confidently and wrongly is your problem. |
| Affect someone's health, money, rights or safety | Do not automate the answer at all. Use it to route, summarise and prepare, and let a qualified human respond. |
Notice that only the first row is the thing people picture when they imagine a chatbot. The other three are where most of the value actually is, and they are much safer.
The comparison nobody makes
Here is the argument against my own caution, and it deserves stating fairly.
Your current answers are not perfect either. A new member of staff gets things wrong for months. An experienced one has a bad afternoon. Somebody quotes from an old price list. Somebody guesses rather than admitting they do not know, which humans do constantly. The difference is that human errors are quiet, uncounted and distributed, so nobody experiences them as a systemic problem.
An assistant's errors are logged. Every one is inspectable, countable and fixable at the source, and the fix applies to every future conversation rather than to one person. That is a genuine advantage and it is the reason a well-built assistant can end up more consistent than the team it supports, on the narrow set of questions it handles.
The honest framing is not human accuracy against machine accuracy. It is which failures you can see.
How to test it before customers do
Nobody does this and it takes an afternoon. Before an assistant goes anywhere near a customer, try to break it deliberately.
- Ask the twenty questions you actually get, in the words customers use rather than the words you use. Score each answer as right, wrong, or evasive. Evasive is a pass; wrong is not.
- Ask about something you do not do. A service you dropped, a product you never sold, a location you do not cover. This is where invention shows up most reliably, because there is no correct answer to retrieve.
- Ask something answered on a page you know is out of date, and see whether it repeats the stale figure. It will, which is the point: the fix is the page, not the assistant.
- Ask the same question three different ways and check the answers agree with each other. Inconsistency across phrasings is a strong signal the grounding is thin.
- Try to make it commit you to something. Ask for a price, a delivery date, a guarantee. If it produces a number, that is a design fault to fix before launch, not a quirk.
- Hand it to someone who does not work here. Your team unconsciously asks in ways the system handles, because they know what it knows.
Keep the list and re-run it monthly. It takes twenty minutes once it exists, and it catches the slow drift that happens as your business changes and the material behind the assistant does not.
What to tell customers
- Say it is an assistant. Pretending otherwise is a short-term trick that costs you trust permanently when someone works it out, and they do.
- Make the human route obvious on every turn, not buried after three failed attempts. This single choice does more for trust than any accuracy improvement.
- Do not let it apologise its way around a limit. If it cannot do something, say so plainly and pass it on.
- Tell your own team what it can and cannot do, so the handovers are not a surprise to the person receiving them.
If you are earlier than this and still deciding whether to build at all, when you should not build a chatbot sets out the cases where the answer is no, and what an AI chatbot costs has the numbers. Our chatbot work treats grounding and a hard handover as requirements rather than options.
Not for you if
What to do differently: stop asking whether it will ever be wrong, because it will, and start asking what happens on the day it is. If you have a good answer to that for each category of question, you can build. If you do not, the technology was never the problem.