However, a recent paper from Apple researchers found o1 and many other language models have significant trouble solving genuine mathematical reasoning problems. Their experiments show the outputs of these models seem to resemble sophisticated pattern-matching rather than true advanced reasoning. This indicates superintelligence is not as imminent as many have suggested.
Will AI keep getting smarter?
Some people think the rapid pace of AI progress over the past few years will continue or even accelerate. Tech companies are investing hundreds of billions of dollars in AI hardware and capabilities, so this doesn’t seem impossible.
If this happens, we may indeed see general superintelligence within the “few thousand days” proposed by Sam Altman (that’s a decade or so in less scifi terms). Sutskever and his team mentioned a similar timeframe in their superalignment article.
Many recent successes in AI have come from the application of a technique called “deep learning”, which, in simplistic terms, finds associative patterns in gigantic collections of data. Indeed, this year’s Nobel Prize in Physics has been awarded to John Hopfield and also the “Godfather of AI” Geoffrey Hinton, for their invention of Hopfield Networks and Boltzmann machine, which are the foundation for many powerful deep learning models used today.
General systems such as ChatGPT have relied on data generated by humans, much of it in the form of text from books and websites. Improvements in their capabilities have largely come from increasing the scale of the systems and the amount of data on which they are trained.
However, there may not be enough human-generated data to take this process much further (although efforts to use data more efficiently, generate synthetic data, and improve transfer of skills between different domains may bring improvements). Even if there were enough data, some researchers say language models such as ChatGPT are fundamentally incapable of reaching what Morris would call general competence.
One recent paper has suggested an essential feature of superintelligence would be open-endedness, at least from a human perspective. It would need to be able to continuously generate outputs that a human observer would regard as novel and be able to learn from.
Existing foundation models are not trained in an open-ended way, and existing open-ended systems are quite narrow. This paper also highlights how either novelty or learnability alone is not enough. A new type of open-ended foundation model is needed to achieve superintelligence.
What are the risks?
So what does all this mean for the risks of AI? In the short term, at least, we don’t need to worry about superintelligent AI taking over the world.
But that’s not to say AI doesn’t present risks. Again, Morris and co have thought this through: as AI systems gain great capability, they may also gain greater autonomy. Different levels of capability and autonomy present different risks.
For example, when AI systems have little autonomy and people use them as a kind of consultant – when we ask ChatGPT to summarise documents, say, or let the YouTube algorithm shape our viewing habits – we might face a risk of over-trusting or over-relying on them.
In the meantime, Morris points out other risks to watch out for as AI systems become more capable, ranging from people forming parasocial relationships with AI systems to mass job displacement and society-wide ennui.
What’s next?
Let’s suppose we do one day have superintelligent, fully autonomous AI agents. Will we then face the risk they could concentrate power or act against human interests?
Not necessarily. Autonomy and control can go hand in hand. A system can be highly automated, yet provide a high level of human control.
Like many in the AI research community, I believe safe superintelligence is feasible. However, building it will be a complex and multidisciplinary task, and researchers will have to tread unbeaten paths to get there.
This article is republished from The Conversation under a Creative Commons license. Read the original article.