by Tyler Kelley
Your team is using AI every day. They’re faster than they’ve ever been. Their outputs look professional and comprehensive.
And you have no idea if any of it is actually good.
You see, most teams are using AI to avoid the hard thinking work. They’re skipping from brief to deliverable without doing the strategic work in between. They’re hoping AI will figure out what to do so they don’t have to.
That’s not a capability gap. It’s a choice. And it’s making teams weaker, not stronger.
The real problem isn’t that people don’t know how to use AI. It’s that they’re using it to replace thinking instead of enhance it. And most leaders can’t tell the difference.
This framework gives you a way to tell.
The AI Effectiveness Assessment
Use this in one-on-ones, team meetings, or project reviews. You’re listening for patterns in how people talk about their work.
Question 1: Walk me through how you approached this project.
What You’re Testing: Whether they used AI to extend their thinking or replace it.
Effective Use Sounds Like:
• “I started by mapping what we knew about the customer’s decision process, then used AI to explore whether that held across different segments”
• “I had a hypothesis about their competitive positioning, so I used AI to gather data I couldn’t compile manually”
• “The first output was too generic, so I added more context about their specific market dynamics”
Ineffective Use Sounds Like:
• “I asked AI to create a strategy framework and it gave me something comprehensive”
• “I fed it the brief and got back a good starting point”
• “I used [specific prompt/framework] and the output was solid”
What This Reveals: Effective users start with their own thinking. They know what problem they’re solving. They use AI as a research and exploration tool.
Ineffective users start with AI. They’re hoping the tool will figure out what to do. They’re outsourcing the thinking.
Red Flag: If most of your team describes their process in terms of what they asked AI to do rather than what they were trying to figure out, they’re not adding value above what any team with the same tools could produce.
Question 2: What did you learn that you didn’t know before?
What You’re Testing: Whether their understanding deepened or just their output increased.
Effective Use Sounds Like:
• “I thought their main competitor was X, but the research showed customers actually see Y as the alternative”
•“I assumed price was the main objection, but when I dug deeper, it’s actually about implementation complexity”
•“The data revealed a pattern I hadn’t noticed: their best customers all share this specific characteristic”
Ineffective Use Sounds Like:
•“I learned some industry statistics I didn’t have before”
•“I got a better understanding of best practices in this space”
•“The output gave me a good framework to work from”
What This Reveals: Effective users can point to specific insights that changed their understanding. Their mental model of the problem evolved.
Ineffective users got information but gained no insight. They have deliverables but no deeper understanding.
Red Flag: If your team consistently can’t articulate what changed in their thinking, they’re using AI as a production tool, not a thinking tool. The work will be adequate but never distinctive.
Question 3: Why did you choose this approach over alternatives?
What You’re Testing: Whether they exercised judgment or accepted what seemed good enough.
Effective Use Sounds Like:
•“We considered three positioning directions. This one works better because their customers care more about reliability than innovation”
•“The data suggested going broad, but given their limited resources, focused depth makes more strategic sense”
•“AI generated five options. This one was actually the weakest on paper, but when I thought through implementation, it’s the only one that fits their capabilities”
Ineffective Use Sounds Like:
•“This approach seemed comprehensive”
•“The output looked solid and covered all the bases”
•“We went with what AI suggested because it made sense”
What This Reveals: Effective users considered alternatives. They applied domain knowledge and judgment. They made deliberate choices they can defend.
Ineffective users took the path of least resistance. They accepted the first decent option without critical evaluation.
Red Flag: If your team consistently struggles to explain why they chose one direction over another, they’re not applying expertise. They’re just using tools. This makes your team interchangeable with any other team using the same AI.
Question 4: Where did the AI output need the most work?
What You’re Testing: Whether they’re evaluating critically or accepting outputs as-is.
Effective Use Sounds Like:
•“The strategic framework was solid, but the examples were generic. I replaced them with scenarios specific to their industry”
•“The tone was too formal for their brand. I rewrote the key sections to match how they actually communicate”
•“It missed the political dynamics inside their organization. I adjusted the recommendations to account for that”
Ineffective Use Sounds Like:
•“It was pretty good. I just cleaned up some formatting”
•“Minor edits mostly. Fixed a few phrases”
•“Not much, honestly. The output was comprehensive”
What This Reveals: Effective users are quality control. They know the difference between good and generic. They add value by applying expertise AI doesn’t have.
Ineffective users are pass-through. If the output looks professional, they ship it. They’re not adding meaningful value.
Red Flag: If most outputs require only light editing, your team isn’t exercising judgment. They’re not catching gaps because they don’t know what good looks like. Clients will notice this before you do.
Question 5: What would you do differently next time?
What You’re Testing: Whether capability is growing or staying static.
Effective Use Sounds Like:
•“I’d start with customer interviews before using AI for research. That context would have made the exploration more targeted”
•“I’d push back harder on the first outputs. I accepted ‘good enough’ when I should have gone deeper”
•“I’d structure my initial questions differently. I was too broad at first and had to narrow down later”
Ineffective Use Sounds Like:
•“The process worked well. I’d probably do the same thing”
•“Maybe use a different prompt template”
•“I’d give AI more context upfront so the first output is closer”
What This Reveals: Effective users are learning. They’re identifying what worked and what didn’t. They’re building skill, not just completing tasks.
Ineffective users are repeating a process. They’re not developing deeper capability. They’re just getting faster at producing the same level of work.
Red Flag: If your team isn’t identifying ways to improve their thinking process, they’re not growing. They’re just getting faster at producing the same level of work. That’s not sustainable advantage.
Scoring Your Team
You don’t need formal scores. You’re looking for patterns.
If most answers show effective use signals: Your team is using AI to amplify expertise they already have. The work is getting better, not just faster. Keep developing domain knowledge and judgment. That’s your competitive advantage.
If most answers show ineffective use signals: Your team is using AI as a thinking replacement, not a thinking tool. They’re producing adequate work that lacks distinctive insight. This is fixable, but it requires developing the underlying skills, not better AI techniques.
If answers are mixed: You have some people building real capability and others just using tools. The gap will widen. Invest in developing the thinking skills across the team. n
Tyler Kelley is the Co-founder and Chief Strategist of SLAM! Agency, a marketing execution and creative operations agency. He advises businesses on leveraging AI to drive growth and innovation. For questions or to explore these predictions further, email Tyler at tyler@slamagency.com.