Observed Signal · Aug 19, 2026 · Opinion / Analysis · Source: UX Collective · Impact: 3/5 · Sentiment: Negative
Large Language Models (LLM) & AI Market: AI Lies Confidently; UX Must Expose Uncertainty
The article argues that contemporary large language models routinely produce confident but incorrect answers (hallucinations) because model training and scoring often reward confident guessing over admitting uncertainty. The author recommends building a "harness" around models — UI and runtime guardrails that show step‑by‑step reasoning, force the model to flag uncertainty, and allow selective prediction (abstaining when unsure). The piece cites academic work and industry reports (including a KPMG survey) showing widespread reliance on unchecked AI outputs and rising hallucination rates in newer reasoning‑focused models. The author describes product design patterns (step‑level feedback, cognitive forcing functions, selective prediction) and points to toolkits such as NVIDIA’s NeMo Guardrails as examples of runtime enforcement that do not require changing the base model.
The piece highlights rising LLM hallucination risks, product UX patterns (harness, selective prediction) and industry survey data (KPMG) showing widespread reliance on unchecked outputs — relevant to any organization deploying generative AI in products or workflows.
Key Takeaways & Evidence Grounding
- KPMG found 58% of employees rely on AI output without checking it, 57% said they have made mistakes because of it, and only 41% work somewhere with any AI policy.
- Researchers from OpenAI and Georgia Tech (Nature paper) showed that current training and evaluation incentives reward confident guessing, which encourages hallucinations.
- The author’s product team implemented a "harness" that forces models to show reasoning, flag uncertainty, and abstain when below a confidence threshold (selective prediction).
- NVIDIA researchers published NeMo Guardrails, a runtime toolkit that enforces rules between user and model without modifying the underlying model.
- An analysis cited (Scott M. Graffius) reports that some newer reasoning‑focused models hallucinate on roughly one third to one half of open‑ended factual questions in 2025.
Connected Companies & Entities
5 Entities mappedAnthropic
Foundation model company selling AI assistants and model APIs.
“Right now, what you actually get from Claude or ChatGPT is a play‑by‑play that tells you nothing....”
NVIDIA
Accelerated computing company spanning AI software, cloud and gaming.
“Traian Rebedea and his team at NVIDIA built one of the better‑known versions of it, a toolkit that sits between the user and the model and e...”
KPMG
Enterprise consulting network monetising transformation, risk and AI services.
“In 2025, KPMG found that 58% of employees admit to relying on AI output without checking whether it’s accurate, and 57% say they’ve already ...”
Medium
Subscription-based digital publishing platform for readers and writers.
“Get Dan Maccarone’s stories in your inbox....”
OpenAI
Foundation model company selling AI software, APIs and subscriptions.
“A team of researchers from OpenAI and Georgia Tech showed that models hallucinate because the way we train and score them rewards confident ...”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.