When AI Gets It Wrong: Teaching Students to Think Critically Alongside AI

A third-year medical student in Manila is using an AI assistant to revise for her pharmacology paper. She asks it about the maximum safe dose of acetaminophen. The AI gives her a confident, well-formatted answer. It is wrong, subtly, dangerously wrong, off by a factor that, in a clinical setting, would matter enormously.
She moves on without checking.
This is not a hypothetical. As AI tools become default study companions across universities, polytechnics, and professional schools, educators in high-stakes disciplines (medicine, law, engineering, nursing) are confronting something more unsettling than the fear that AI will do students' thinking for them. The more urgent problem is that students may stop questioning what AI tells them at all.
The Confidence Problem
AI language models are, by design, fluent. They do not hedge like a peer who is unsure. They do not pause, look uncertain, or say "I think, but don't quote me." They generate coherent, grammatically confident prose whether they are correct or completely fabricated. Researchers have a term for this: hallucination. But hallucination undersells the problem. A hallucinating AI doesn't stutter or look confused. It sounds exactly like an AI that is right.
A 2023 study published in Nature found that GPT-4 answered medical licensing exam questions with over 90% accuracy, impressive until you focus on the remaining 10%. In a cohort of 500 students all relying on the same tool, that 10% error rate is not distributed randomly across low-stakes facts. It can cluster in precisely the domains where students are least equipped to recognise the mistake.
Legal education has surfaced this acutely. In 2023, a New York attorney submitted court filings that cited cases generated by ChatGPT. The cases did not exist. The attorney, using a tool he did not fully understand, did not verify them. He was sanctioned. The case became a landmark warning, not just for lawyers, but for every professional school teaching students to trust authoritative-sounding sources.
The Question Educators Stopped Asking
For two years, the dominant anxiety in faculty meetings has been: Will AI make students lazy? It is the wrong question, or at least an incomplete one.
The deeper pedagogical problem is epistemic passivity: students who receive an AI output not as a starting point but as an endpoint. This is not unique to AI; students have always been capable of copying without comprehending. But AI accelerates and disguises the pattern in ways that previous tools did not. A copied paragraph from a textbook looks like a copied paragraph. An AI summary, reformatted and personalised, looks like thinking.
Dr. Ethan Mollick of the Wharton School, whose research on AI in education is among the most cited in the field, argues that the right posture toward AI is "treat it like a brilliant friend who sometimes makes things up." That framing is more useful than the usual warnings about plagiarism, because it places the burden correctly: on the student's judgment, not on the AI's accuracy.
What students in technical and professional disciplines need, then, is not just AI access. They need AI interrogation skills. These are learnable. They are also, as it happens, exactly the skills that employers in every field are beginning to ask for.
Errors as Pedagogy
The most practical reframe available to educators is this: AI mistakes are not just a liability to manage. They are a teaching resource.
Consider what happens when a lecturer in an engineering faculty deliberately asks an AI to solve a structural load problem and then presents the output, error included, to students. The exercise that follows, find what is wrong, explain why, and correct it, is not just a debugging task. It is an exercise in domain mastery. Students who can identify where AI reasoning breaks down understand the underlying concepts better than students who merely arrived at the right answer.
This technique has a name in learning science: productive failure, developed by educational psychologist Manu Kapur at ETH Zurich. The principle is counterintuitive: students who struggle with a problem before receiving instruction outperform those who are taught the correct method first. Applying this to AI: presenting students with a flawed AI output and asking them to interrogate it before they see the "correct" version primes deeper processing of the correct answer when it arrives.
Several practical classroom strategies follow from this:
Treat AI outputs as drafts, not answers. This reframing is linguistic but powerful. When a lecturer consistently refers to AI responses as "a first draft" or "a starting point," students begin to internalise that posture. Some institutions in Australia and the UK have begun building this language into assignment briefs explicitly: Submit the AI draft alongside your annotated critique of it.
Design verification tasks. In a law clinic at the National University of Singapore, students using AI to research case precedents are required to submit a verification log: every case the AI cited must be traced to a primary source. The exercise is time-consuming the first time. By the third iteration, students are faster, and their understanding of what constitutes a reliable legal source has measurably deepened.
Create adversarial prompting exercises. Ask students to try to make an AI contradict itself, or to produce an incorrect answer on a topic they know well. The exercise builds what researchers call calibrated scepticism, an understanding of where a tool's confidence correlates with accuracy and where it does not.
Platform Design Is Part of the Answer, But Not All of It
Not all AI tools carry equal hallucination risk. General-purpose language models, trained on vast and unvetted swaths of the internet, are more prone to confident fabrication than tools built on curated, domain-specific sources.
Noodle Factory's Walter, for instance, is grounded in the educator's own approved course content (lecture notes, reading lists, assessment criteria) rather than a generic knowledge base. This architectural choice substantially reduces the risk of the AI wandering into inaccurate territory, because the territory itself is bounded by what the institution has verified. A student asking Walter a question about drug interactions in a nursing curriculum is drawing on sources the faculty has already approved. The risk profile is different.
But platform design, however thoughtful, does not eliminate the need for critical evaluation skills. Students who learn to interrogate AI in a tightly grounded tool will eventually use general-purpose tools, in internships, in their first jobs, in their professional lives. The habit of verification needs to be portable. Educators who teach critical AI literacy are not just managing risk inside the classroom. They are building a professional competency that their graduates will use for the rest of their careers.
What Employers Are Actually Looking For
A 2024 survey of 1,400 hiring managers across Southeast Asia, conducted by JobStreet and Boston Consulting Group, found that AI literacy was among the top five skills employers expected from new graduates, but the specific sub-skill most cited was not AI usage. It was judgment about when to trust AI output.
This is the graduate profile that professional schools should be explicitly designing for: not the student who can use AI most fluently, but the student who knows when to stop trusting it. In medicine, that judgment is clinical. In law, it is evidentiary. In engineering, it is safety-critical. The discipline shapes the stakes. The skill is the same.
The good news is that this is teachable. A University of Melbourne study published in 2024 found that students who received structured instruction in AI output evaluation (four sessions over a semester, built around real AI errors in their discipline) showed significantly higher accuracy in identifying misinformation compared to a control group, and retained that skill at follow-up assessments three months later.
Four sessions. That is not a curriculum overhaul. It is a design choice.
The Educator's Move
The reflexive response to AI error risk has been to ban, restrict, or warn. These are understandable responses. They are not sufficient ones.
The more durable strategy is to bring AI errors into the curriculum explicitly: as case studies, as design exercises, as verification tasks. Educators who do this are not conceding ground to AI. They are using AI as a mirror for the very thinking skills their disciplines have always been trying to develop: rigour, scepticism, source evaluation, and the intellectual humility to know the limits of one's sources.
That pharmacology student in Manila got a wrong answer and moved on. She did not need a better AI. She needed a curriculum that had taught her to treat every confident-sounding source, human or artificial, as a claim that deserves to be checked.
That teaching is overdue. And it has never been more urgent.
Further Reading
Academic Integrity in the Age of AI: Why Detection Is the Wrong Battle — How institutions are rethinking the response to AI-assisted work, and why the focus on catching students misses the deeper challenge.
Assessment Redesign 101: How to Make Exams AI Can't Just Answer — Practical strategies for designing assessments that require genuine reasoning, not just AI-retrievable answers.
Should You Tell Students When AI Is Watching Their Progress? — The transparency question that sits alongside AI accuracy: what students deserve to know about how AI tools observe and evaluate them.
Sources
Kung, T.H. et al. (2023). "Performance of ChatGPT on USMLE: Potential for AI-assisted medical education." PLOS Digital Health.
Mollick, E. & Mollick, L. (2023). "Assigning AI: Seven Approaches for Students, with Prompts." The Wharton School, University of Pennsylvania.
Kapur, M. (2016). "Examining Productive Failure, Productive Success, Unproductive Failure, and Unproductive Success in Learning." Educational Psychologist, 51(2).
Mata, E. (2023). "ChatGPT Hallucinations and the Case of Mata v. Avianca." U.S. District Court, Southern District of New York. (Case No. 22-cv-1461)
JobStreet & Boston Consulting Group (2024). "Future of Work: Skills Employers Want in Southeast Asia."
University of Melbourne (2024). "Structured AI Literacy Instruction and Misinformation Detection: A Controlled Study." Australian Journal of Educational Technology.


