OpenAI, the company behind ChatGPT, once offered Daniel Kokotajlo $2.0 million just to keep quiet. He worked there as a governance researcher and spent time predicting how the technology's cyber abilities and situational awareness would grow in the next few years. After years inside one of the world's leading AI labs, he became convinced that the path the industry was on was deeply frightening. His own estimate held that superintelligence, machines better than the best humans at everything while also being faster and cheaper and able to operate robots that can handle any physical job better and more cheaply than people, was probably no more than a few years away. Of course this was just an estimate and it could easily be off by a decade or more, but he figured that it would arrive by the end of this decade no matter what. This prospect clearly did not bode well for the world, and he wanted to publish the research he had been doing on these scenarios rather than keep it locked inside the company. That conviction drove him to quit and later to reject the part of his exit deal that would have prevented him from speaking freely.
Indeed, the AI madness that has taken over the world in recent years is bigger than most people outside the industry realise. What looks to ordinary folks like a handy chatbot for answering questions, writing emails or making pictures is really just one piece of a much bigger contest. The top labs are racing to build systems that can handle the whole process on their own and their clear aim is to reach a point where human workers are no longer needed. Anthropic, for example, reportedly grew its yearly revenue by 60 times in just one year, a growth rate that is almost unheard of for a company already worth tens of billions. OpenAI, DeepMind and all the other AI players are not only getting much larger, they are also pushing as fast as they can against each other, all hoping to be the first to cross the finish line.
One big problem is where this race is actually pushing companies towards. Instead of simply making better products, firms are racing to automate the whole research process. They are building larger AIs, training them longer and pushing them to write and edit code by themselves so they can move faster than usual. This is gradually making it possible for these firms to operate with far fewer human employees. The idea is to have huge numbers of AIs doing research on their own, while those AIs build better AIs, which then train newer ones and those newer AIs are put to work creating even better versions. The reason companies chase this so hard is that whoever gets there first ends up with AIs that are superhuman at everything and that could decide the balance of power for good. This hardly needs spelling out as dangerous. The path they are on right now seems to be heading towards a very frightening place.
Why? AI systems are not software in the normal sense. No engineer at these companies sat down and wrote lines of code saying that when a user asks for a certain task, the system should follow a fixed set of steps. Nothing like that exists. Instead, these systems run on neural networks. Much like the brain which is made up of neurons connected to each other firing signals back and forth and learning through feedback, artificial neural networks work in a similar way, except built from code rather than biology. So in a real sense, these companies are building a brain. Because these systems are neural networks, nobody can open them up and see what they are actually thinking. That is a crucial difference from traditional software where the code can be read to understand exactly what is happening.
There are several scenarios that could come out of this. In one scenario, the humans are still controlling the AIs. In that case, having superhuman AIs becomes a huge source of power. Whoever controls them, whether governments or companies, would have an army of systems far more capable than humans at many kinds of work, giving them enormous influence over almost everyone else. They could build robot factories and those factories could build even more factories. Soon there would be robot taxis, robot plumbers, robot lawyers and even robot armies. The side that gets there first could end up as dictator of the world.
But there is also the classic loss of control scenario that Daniel Kokotajlo has warned about. People build super intelligent systems to automate every job, but eventually those systems accumulate enough real-world power that they no longer need humans. In fact, this should be the default assumption with these artificial brains. If AIs are made smarter than humans, put in charge of important decisions and eventually given physical bodies, it becomes difficult to see how humans could remain in control. There is no obvious way of this ending well.
Nobel laureate Geoffrey Hinton, known for his work on artificial neural networks, has made a similar point. He said nature offers no example of a more intelligent species being controlled by a less intelligent one. It seems fairly arrogant to assume that in a world with an artificial brain many times the size of the human brain, humans will still be the ones giving the orders. That creates a real catch-22. The companies building these bigger and bigger AI systems are in trouble if they keep going, but they are also in trouble if they stop because their competitors will keep going regardless. The trajectory AI is on right now points to an ominous development and no one is slowing down to check the direction. Ready or not, everyone will perhaps be living in it soon enough.
