When AI makes things up: what it means for your business

Short answer: AI tools sometimes write things that sound right and are wrong. People call this "hallucination." You cannot switch it off. You can design around it: make the tool answer from your own records, make it show where each fact came from, and make a person check before anything leaves the building.

If you are new to this, start with where to start with AI and why ChatGPT is not the same as AI for your business. This post explains the one failure every owner should understand before they trust a draft.

What "made it up" looks like on a normal day

It is rarely dramatic. It looks like this:

A customer letter that gives the wrong date for their last service.
A summary that says a client "agreed" to something they only asked about.
A price that looks like your prices, but is not on any invoice.
A rule or a deadline stated with confidence, from nowhere.

The tool is not lying. It writes the most likely next words. When it does not have the fact, it may still write a sentence that has the shape of a fact.

Why "from your own records" matters

A general chatbot answers from what it learned on the public internet. It does not know your customers, your jobs, or your prices. When you ask about them, it guesses.

A tool set up for your business should do something different. It should look up the record first, then write. And it should show you the record: the invoice number, the work order, the email date. If a fact in the draft has no source next to it, treat it as a guess.

This is the main reason we build tools that read the business's own files and cite them. It does not make errors impossible. It makes them easy to see.

Review rules that catch most problems

Every fact needs a source. Names, dates, amounts, and promises must point to a record. No source, no send.
"Not found" is a good answer. Tell staff to trust a draft more when it admits a gap, not less.
Numbers come from the system, not the draft. Totals, balances, and prices are copied from your accounting or job software. The draft does not calculate them.
Laws and rules get checked by a person. If a draft mentions a law, a deadline, or a regulation, someone checks it at the source or with your adviser.
A named person sends. The person who presses send owns the message. That is true with or without AI.

Where the risk is highest

Anything that goes to a customer or client.
Anything with money in it.
Anything that sounds like advice: legal, medical, tax, or safety.
Anything about a person's eligibility, such as a tenant, an applicant, or a loan.

In those areas, AI should prepare and a person should decide. That is not a limit of today's tools. It is how a responsible business works.

What to ask a vendor

When the tool does not know, does it say so?
Can I see the record behind each fact in a draft?
Can it send anything without a person approving it?
Where does my data go while it works? If the answer is unclear, ask about an on-premises option.

A realistic first step

Pick one routine document, such as a follow-up letter. Have the tool draft ten from real records. Have the person who usually writes them check every fact against the file. Count the errors and note what kind they are. That one afternoon will tell you more than any demo.

If you want drafts that show their sources and wait for your people, talk with us.