About six months ago, Suvrat Raju, a professor of theoretical physics at the International Centre for Theoretical Sciences in Bengaluru, asked a frontier AI model to solve a physics problem. He didn’t ask because he needed to know the answer. He had actually solved this particular problem already. But he had never published the solution — which made it a perfect question to test how good AI was at physics.
LLMs have been trained on the whole Internet, so if a physics problem has been discussed online, it’s likely the model already knows about it and may have memorized the answer. So a fair test of AI capabilities needs to focus on problems that have never appeared online.
The model didn’t answer Raju’s problem correctly, but it did better than he expected — well enough to get him thinking the experiment should be repeated and formalized. Along with a colleague, he set out to create a theoretical physics benchmark for AI.
They asked other physicists for some of their private gems — minor but real findings they’d never gotten around to cleaning up for public display. They were asking for real work, problems that take humans at the top of the field about a day to solve.
“Our contributors had to either sacrifice potential publication elsewhere, or else undertake a small research project just for us,” Raju wrote.
By the time all the questions and answers were ready, the AIs were too. At a colloquium in September, Raju shared preliminary results showing that leading AI models could solve about 60% of the problems on their own. And by the time he and his colleagues finished the full set, the situation had advanced further: GPT-6 Astra could solve nearly 90% of the problems.
To read more, click here.