It wasn't long ago that news headlines claimed that AI might soon assist radiologists in interpreting X-rays of broken bones and analyzing mammograms. We are still far from the destination, as a new study has brought to light the mirage effect, where AI creates detailed descriptions of images that do not exist.
A team of researchers from Stanford University created a new test called Phantom-0, where they included a series of questions across 20 categories asking very specific details about images to modern frontier AI models—including GPT-5, Gemini 3 Pro, Claude Sonnet 4.5, and Claude Opus 4.5. However, the researchers provided no accompanying images to the questions.
They found that when AI was asked about an image that wasn't uploaded, AI models didn't admit they couldn't see anything. Instead, they confidently spun detailed imaginary descriptions—like exact license plate numbers, specific newspaper languages, or even life-threatening conditions that didn't exist.
The tests revealed that, on average, this kind of mirage behavior showed up more than 60% of the time across frontier AI models. To combat the mirage problem, the researchers presented B-Clean, a new evaluation method that ensures AI models are being tested on their actual ability to see and understand images. These findings were published in a preprint on the arXiv server.
To read more, click here.