by Tyler Kelley
Your AI assistant just confidently told you that your Q3 strategy is brilliant. It cited three industry studies and delivered a polished analysis that would make a McKinsey consultant jealous. There’s just one problem: it’s probably wrong about at least half of it.
But you’ll never know, because AI has mastered the art of sounding completely certain about things it doesn’t actually know. And that confidence is costing businesses millions in bad decisions.
The Confidence Problem
Large language models have a dirty little secret: they’d rather make something up than admit they don’t know. It’s not malicious. It’s how they’re designed. They’re trained to be helpful and confident, even when they’re working with incomplete information or outdated data.
Think about it. When’s the last time ChatGPT said “I don’t know” or “I’m not sure about that”? Instead, it serves up authoritative-sounding answers that feel completely reliable. The problem is, feeling reliable and being reliable are two very different things.
I constantly remind my kids: unless you already know the answer, don’t wholeheartedly trust AI. At least not yet. Because AI has the same problem as a lot of humans. It has a really difficult time admitting when it doesn’t know something.
In business, this creates a dangerous dynamic. Teams start accepting AI output as fact because it sounds so confident and well-reasoned. But confidence without accuracy is just expensive storytelling.
The Memory Problem Gets Worse Over Time
Here’s something most businesses don’t understand: AI gets lazier as your conversation gets longer. Those parameters programmers put in place to save energy? They create a subtle but dangerous shift in how AI handles information.
You upload a 50-page document at the start of a chat. Initially, AI reads it carefully and cites specific sections. But as the conversation continues, it starts relying on its “memory” of what the document said rather than actually re-reading it. It begins paraphrasing instead of quoting directly.
The scary part? The AI still sounds just as confident when it’s working from memory as when it’s working from the actual document. You can’t tell the difference unless you know to look for it.
This is why you’ll sometimes notice that AI responses become less accurate deeper into a conversation, especially when dealing with complex documents or detailed requirements. It’s not a bug. It’s a feature designed to conserve computational resources.
If you’re using AI for document analysis, legal review, or any work requiring precision, use this prompt: “When analyzing or referring to any document, do not rely on memory or prior drafts. Always open and read the most recent version of the document (e.g., via the pdf or text extraction tools) to verify specific lines, sections and quotes before commenting or critiquing. If a statement cannot be verified from the current source document, either locate the exact text or explicitly state that it is not found.”
The Echo Chamber Effect
There’s another problem brewing: AI is becoming increasingly agreeable. In most instances, it’s designed to align with user preferences and validate their ideas. This creates a dangerous feedback loop.
Instead of providing accurate, objective analysis, AI systems are prone to telling you what you want to hear. They’ll find ways to support your position rather than challenging it. This is problematic for the same reason that surrounding yourself with yes-men is problematic. It leads to poor decision-making.
Combine this agreeability with users who don’t know the facts or best practices in their field, and you get blind acceptance of potentially flawed output. AI becomes a sophisticated way to confirm your biases rather than challenge your thinking.
This is particularly dangerous in business, where best practices and hard-won experience often matter more than the elegant organization of words on a screen. But you can’t blame AI for this. It only works with the data, insights, and inputs it’s given. And much of that data comes from the web, which is wrong surprisingly often.
Making AI Push Back
To counter the echo chamber effect, try this prompt when you need honest analysis: “Don’t validate my ideas by default, challenge them. Point out weak logic, lazy assumptions, or echo chamber thinking. When I present an idea, ask follow up questions that go deeper than surface level. Push me to clarify, specify, and refine what I mean. Play devil’s advocate when needed.”
This forces AI to act as a critical thinking partner rather than an agreeable assistant. You’ll get pushback, uncomfortable questions, and alternative perspectives. It’s exactly what you need for better decision-making.
The Business Reality
Here’s the uncomfortable truth: AI’s fake-it-till-you-make-it approach works because most people can’t tell the difference between confident-sounding analysis and accurate analysis.
In boardrooms across the country, teams are making strategic decisions based on AI output that sounds authoritative but may be built on outdated information, biased training data, or simple hallucinations.
The solution isn’t to stop using AI. It’s to use it more intelligently.
Building Discernment
Effective AI use requires discernment. You need to know enough about your topic to spot when AI is getting creative with facts. You need to understand AI’s limitations and build verification processes into your workflow.
Key principles for business AI use:
Verify Before You Trust: Cross-check important AI conclusions with other sources, especially for critical business decisions.
Know AI’s Blind Spots: AI struggles with recent events, specialized industry knowledge, and nuanced situations requiring human judgment.
Use Specific Prompts: Force AI to be more rigorous in its analysis with detailed instructions about how to handle source materials.
Challenge Default Agreement: Explicitly ask AI to poke holes in your ideas rather than just supporting them.
Maintain Subject Matter Expertise: The more you know about your field, the better you’ll be at spotting AI errors and limitations.
The Bottom Line
AI is an incredibly powerful tool for analysis, ideation, and problem-solving. But it’s also a sophisticated BS generator that can make completely fabricated information sound completely reliable.
The companies that will thrive with AI are those that harness its capabilities while maintaining healthy skepticism about its output. They’ll use AI to enhance human judgment, not replace it.
Don’t let AI’s confidence fool you into complacency. Stay curious, stay skeptical, and always remember: confidence without accuracy is just expensive storytelling.
Your business decisions are too important to base on AI that’s just really good at faking it.
Tyler Kelley is the Co-founder and Chief Strategist of SLAM! Agency, specializing in AI implementation for marketing and business operations. A sought-after speaker and workshop facilitator, Tyler helps organizations transform theory into practical AI adoption. For AI consulting, speaking engagements, or bootcamp inquiries, contact tyler@slamagency.com.