Researchers flag OpenAI’s Whisper AI used in hospitals as problematic, here’s why

HIGHLIGHTS

OpenAI Whisper, an AI tool designed for transcription, is said to achieve near-human accuracy

But a major flaw has been revealed: Whisper is prone to “hallucinations” in which he makes up snippets of text and even entire sentences.

This issue has raised concerns, especially since the tool is used in a variety of industries where accuracy is important, including healthcare.

OpenAI Whisper, an AI tool designed for transcription, is said to achieve near-human accuracy. But a major flaw has been revealed: Whisper is prone to “hallucinations” in which it produces fragments of text or even entire sentences. This issue has raised concerns, especially since the tool is used in a variety of industries where accuracy is important, including healthcare.

In AI terms, hallucinations refer to situations where the model invents information. According to researchers and software engineers, Whisper’s hallucinations often include problematic content such as racist remarks, violent language, and even fabricated medical treatments. Such errors are concerning because Whisper is used to create captions for videos, transcribe interviews, and even help in healthcare settings by transcribing doctor-patient conversations, The Associated Press reported.

Also read: Does OpenAI violate copyright laws? Former company employee says YES

Experts are especially concerned about Whisper being used in hospitals. Despite OpenAI’s warnings that Whisper should not be used in “high-risk areas,” some medical centers are using it to transcribe patient consultations. More than 30,000 clinicians across various health systems, including Children’s Hospital Los Angeles, use a Whisper-based tool developed by Nabla.

A University of Michigan researcher who conducted a study found hallucinations in 80% of the Whisper transcripts he examined, while another machine learning engineer discovered similar problems in half of the transcripts he analyzed. The problem is not limited to complex sounds; Errors appear even in clear and concise recordings.

While Whisper officials are aware of these issues and continue to improve the model, the consequences of such errors can be serious, especially in the healthcare field. As Alondra Nelson, a professor at the Institute for Advanced Study in Princeton, New Jersey, emphasizes, such mistakes can have “really serious consequences.”

Also read: Here’s why Sam Altman calls OpenAI’s o1 model deeply flawed

Moreover, Whisper’s use extends beyond healthcare. It is integrated into popular platforms such as ChatGPT and Microsoft’s cloud services, where millions of people around the world rely on it for transcription and translation.

While most developers expect transcription tools to occasionally misspell words or make minor errors, engineers and researchers say they’ve never encountered an AI-powered transcription tool as hallucinogenic as Whisper.

follow us

Ayushi Jain

Tech news writer by day, BGMI gamer by night. I combine my passion for technology and gaming to bring you the latest in both worlds. View Full Profile