What does PhD-level intelligence even mean?

One of the benchmarks beloved of tech bros for their large language models is what they describe as ‘PhD-level intelligence’. The recently released GPT-5 from OpenAI was supposed to come equipped with its own doctoral certification, gown and mortar board.

Elon Musk, who never likes to be outdone by the likes of Sam Altman, had already described the latest version of Grok as ‘better than PhD level in everything’.

Better than a PhD?

This reminds me of people on TikTok saying that AI will have an ‘IQ’ of 200 or 2,000 or 2,000,0000, without realising that we actually find it hard to distinguish between IQs of, say, 160 and 180 in humans.

What’s the PhD obsession all about exactly? Why is it the magic benchmark? Well, if you work in Silicon Valley, the pattern is usually that a company is founded by an Ivy League drop-out, who then employs people with PhDs in computer science and related fields to make their vision real. You need a whole bunch of people qualified at this level to tackle the complexities of what you’re typically trying to build.

Then add in all the hype about what’s known as Artificial General Intelligence (AGI) - the point at which AI is supposed to be broadly better than most humans at pretty much everything in every domain. Some people think this moment is coming in two years, others think 20 years, while some hold-outs maintain it will never be reached. I guess we need a way of knowing if we’ve got there.

Because I don’t have a PhD in deep learning (just my humble BSc and MSc in sociology and applied psychology respectively), I don’t feel in a very good position to judge whether, or exactly when, the milestone of AGI will be reached. But I do share the consensus view that we ain’t there yet.

We all know that if we ask chatty g about the development of moraines in glacial regions of the Alps, it will provide us with some fantastically complex geological summary that would sit well in a university seminar. But if we give it a simple riddle or a request that it counts the number of letters in Borussia Mönchengladbach, smoke starts to billow from an overheated data centre in Northern California. (Joke: who’s the most unpopular guy on the terraces at Borussia Mönchengladbach? The one who shouts “Give me a B!”)

I’ve known a number of people over the years with PhDs. Some personally, some professionally.

One doctor (non-medical), with whom I’ve held meetings on a few occasions, has a superhuman memory and clearly many more neurons firing than I do. He nevertheless ended up making some decidedly human and rather dubious judgments in his professional career. Well, imho he did, anyway.

I also remember, as an undergraduate, being assigned a tutor at university who’d written a beautifully erudite book on sociological theory, but when you went into his office, he honestly couldn’t have told you what time of the day it was or what day of the week. In some of the professors’ lairs, you had to clamber over large piles of discarded files and books to make it to the desk - a bit like camera crew in one of those extreme hoarding shows on TV.

The point is that having a very deep expertise in a very narrow field of study does not make you worldly wise, practical, sensible or someone who is good with a join-the-dots book. And I wonder if we’re seeing elements of this problem play out with the AI too. It seems possible to make a bot very clever in a million specific fields, while struggling to get it to exercise common sense or to contextualise more straightforward challenges.

I’m not belittling in any way what has been achieved, which is mind-blowing and enough in itself to change the fabric of society. Very often, large language models will make surprising interconnections and come up with suggestions that are quite ingenious. (I experienced this myself when posing a conundrum a couple of years ago about payment systems for public transportation to an earlier version of ChatGPT.)

But the LLMs can also seemingly have blind spots and find that a load of silly stuff still trips them up. I imagine it’s a bit like sending Albert Einstein shopping at Lidl and discovering that he’s forgotten the cottage cheese and the Zoofari interactive cat toy from the middle aisle.

For most of human history, people have got by just fine without having PhD-level intelligence to help them through their everyday lives. The ostensible vision of the tech gurus (and half of the wannabes on LinkedIn) seems to be that when we do have this kind of back-up available to us, we’ll no longer be jumping on the 88 bus or watching Destination X on the BBC, but will empower ourselves to become rip-roaring entrepreneurs like them.

We’ll drop out of top universities - rendered irrelevant and redundant by AI - to pursue our dreams. And thankfully, we won’t have to splash out on PhD-level employees, because invisible agents will be whirring away in our pockets making everything real.