In the book I wrote in the s,The Age of Intelligent Machines, I ended with the spectre of machines matching human intelligence somewhere between and , and I basically have not changed my view on that time frame, although I left behind my view that this is a final spectre. Now I'm trying to consider what that will mean for human society. One thing that we should keep in mind is that innate biological intelligence is fixed. Fifty years from now, the biological intelligence of humanity will still be at that same order of magnitude.
On the other hand, machine intelligence is growing exponentially, and today it's a million times less than that biological figure. So although it still seems that human intelligence is dominating, which it is, the crossover point is around and non-biological intelligence will continue its exponential rise.
This leads some people to ask how can we know if another species or entity is more intelligent that we are? One response is not to want to be enhanced, not to have nanobots. The answer is that they really won't notice it, except for the fact that machine intelligence will appear to biological humanity to be their transcendent servants. It will appear that these machines are very friendly are taking care of all of our needs, and are really our transcendent servants.
But providing that service of meeting all of the material and emotional needs of biological humanity will comprise a very tiny fraction of the mental output of the non-biological component of our civilization. So there's a lot that, in fact, biological humanity won't actually notice. There are two levels of consideration here.
On the economic level, mental output will be the primary criterion. We're already getting close to the point that the only thing that has value is information. Information has value to the extent that it really reflects knowledge, not just raw data. There are a few products on this table — a clock, a camera, tape recorder — that are physical objects, but really the value of them is in the information that went into their design: The actual raw materials — a bunch of sand and some metals and so on — is worth a few pennies, but these products have value because of all the knowledge that went into creating them.
And the knowledge component of products and services is asymptoting towards percent. By the time we get to it will be basically percent. With a combination of nanotechnology and artificial intelligence, we'll be able to create virtually any physical product and meet all of our material needs. When everything is software and information, it'll be a matter of just downloading the right software, and we're already getting pretty close to that.
On a spiritual level, the issue of what is consciousness is another important aspect of this, because we will have entities by that seem to be conscious, and that will claim to have feelings. We have entities today, like characters in your kids' video games, that can make that claim, but they are not very convincing. If you run into a character in a video game and it talks about its feelings, you know it's just a machine simulation; you're not convinced that it's a real person there.
This is because that entity, which is a software entity, is still a million times simpler than the human brain. In , that won't be the case. Say you encounter another person in virtual reality that looks just like a human but there's actually no biological human behind it — it's completely an AI projecting a human-like figure in virtual reality, or even a human-like image in real reality using an android robotic technology. These entities will seem human. They won't be a million times simpler than humans. They'll be as complex as humans.
They'll have all the subtle cues of being humans. They'll be able to sit here and be interviewed and be just as convincing as a human, just as complex, just as interesting. And when they claim to have been angry or happy it'll be just as convincing as when another human makes those claims. At this point, it becomes a really deeply philosophical issue. Is that just a very clever simulation that's good enough to trick you, or is it really conscious in the way that we assume other people are?
In my view there's no real way to test that scientifically. There's no machine you can slide the entity into where a green light goes on and says okay, this entity's conscious, but no, this one's not. You could make a machine, but it will have philosophical assumptions built into it. Some philosophers will say that unless it's squirting impulses through biological neurotransmitters, it's not conscious, or that unless it's a biological human with a biological mother and father it's not conscious.
But it becomes a matter of philosophical debate. It's not scientifically resolvable. The next big revolution that's going to affect us right away is biological technology, because we've merged biological knowledge with information processing. We are in the early stages of understanding life processes and disease processes by understanding the genome and how the genome expresses itself in protein. And we're going to find — and this has been apparent all along — that there's a slippery slope and no clear definition of where life begins. Both sides of the abortion debate have been afraid to get off the edges of that debate: They don't want to get off those edges, because they realize it's just a completely slippery slope from one end to the other.
But we're going to make it even more slippery. We'll be able to create stem cells without ever actually going through the fertilized egg. What's the difference between a skin cell, which has all the genes, and a fertilized egg? The only differences are some proteins in the eggs and some signalling factors that we don't fully understand, yet that are basically proteins.
We will get to the point where we'll be able to take some protein mix, which is just a bunch of chemicals and clearly not a human being, and add it to a skin cell to create a fertilized egg that we can then immediately differentiate into any cell of the body. When I go like this and brush off thousands of skin cells, I will be destroying thousands of potential people. There's not going to be any clear boundary. In the future, we'll be able to do therapeutic cloning, which is a very important technology that completely avoids the concept of the fetus. We'll be able to take skin cells and create, pretty directly without ever going through a fetus, all the cells we need.
We're not that far away from being able to create new cells. For example, I'm 53 but with my DNA, I'll be able to create the heart cells of a year-old man, and I can replace my heart with those cells without surgery just by sending them through my blood stream. They'll take up residence in the heart, so at first I'll have a heart that's one percent young cells and 99 percent older ones.
But if I keep doing this every day, a year later, my heart is 99 percent young cells. With that kind of therapy we can ultimately replenish all the cell tissues and the organs in the body. This is not something that will happen tomorrow, but these are the kinds of revolutionary processes we're on the verge of. If you look at human longevity — which is another one of these exponential trends — you'll notice that we added a few days every year to the human life expectancy in the 18th century.
In the 19th century we added a few weeks every year, and now we're now adding over a hundred days a year, through all of these developments, which are going to continue to accelerate. Many knowledgeable observers, including myself, feel that within ten years we'll be adding more than a year every year to life expectancy. As we get older, human life expectancy will actually move out at a faster rate than we're actually progressing in age, so if we can hang in there, our generation is right on the edge.
We have to watch our health the old-fashioned way for a while longer so we're not the last generation to die prematurely. But if you look at our kids, by the time they're 20, 30, 40 years old, these technologies will be so advanced that human life expectancy will be pushed way out. There is also the more fundamental issue of whether or not ethical debates are going to stop the developments that I'm talking about.
It's all very good to have these mathematical models and these trends, but the question is if they going to hit a wall because people, for one reason or another — through war or ethical debates such as the stem cell issue controversy — thwart this ongoing exponential development. I strongly believe that's not the case. These ethical debates are like stones in a stream. The water runs around them. You haven't seen any of these biological technologies held up for one week by any of these debates.
To some extent, they may have to find some other ways around some of the limitations, but there are so many developments going on. There are dozens of very exciting ideas about how to use genomic information and proteonic information. Although the controversies may attach themselves to one idea here or there, there's such a river of advances. The concept of technological advance is so deeply ingrained in our society that it's an enormous imperative.
Bill Joy has gotten around — correctly — talking about the dangers, and I agree that the dangers are there, but you can't stop ongoing development.
Technological singularity - Wikipedia
The kinds of scenarios I'm talking about 20 or 30 years from now are not being developed because there's one laboratory that's sitting there creating a human-level intelligence in a machine. They're happening because it's the inevitable end result of thousands of little steps. Each little step is conservative, not radical, and makes perfect sense. Each one is just the next generation of some company's products.
If you take thousands of those little steps — which are getting faster and faster — you end up with some remarkable changes 10, 20, or 30 years from now. You don't see Sun Microsystems saying the future implication of these technologies is so dangerous that they're going to stop creating more intelligent networks and more powerful computers. Sun can't do that. No company can do that because it would be out of business. There's enormous economic imperative. There is also a tremendous moral imperative.
We still have not millions but billions of people who are suffering from disease and poverty, and we have the opportunity to overcome those problems through these technological advances. You can't tell the millions of people who are suffering from cancer that we're really on the verge of great breakthroughs that will save millions of lives from cancer, but we're cancelling all that because the terrorists might use that same knowledge to create a bioengineered pathogen. This is a true and valid concern, but we're not going to do that. There's a tremendous belief in society in the benefits of continued economic and technological advance.
Still, it does raise the question of the dangers of these technologies, and we can talk about that as well, because that's also a valid concern. Another aspect of all of these changes is that they force us to re-evaluate our concept of what it means to be human. There is a common viewpoint that reacts against the advance of technology and its implications for humanity.
The objection goes like this: And because the software's so incredibly complex, we can't manage it. I address this objection by saying that the software required to emulate human intelligence is actually not beyond our current capability. We have to use different techniques — different self-organizing methods — that are biologically inspired. The brain is complicated but it's not that complicated. You have to keep in mind that it is characterized by a genome of only 23 million bytes. The genome is six billion bits — that's eight hundred million bytes — and there are massive redundancies.
One pretty long sequence called ALU is repeated thousand times. If you use conventional data compression on the genomes at 23 million bytes, a small fraction of the size of Microsoft Word , it's a level of complexity that we can handle. But we don't have that information yet. You might wonder how something with 23 million bytes can create a human brain that's a million times more complicated than itself. That's not hard to understand. The genome creates a process of wiring a region of the human brain involving a lot of randomness.
Then, when the fetus becomes a baby and interacts with a very complicated world, there's an evolutionary process within the brain in which a lot of the connections die out, others get reinforced, and it self-organizes to represent knowledge about the brain. It's a very clever system, and we don't understand it yet, but we will, because it's not a level of complexity beyond what we're capable of engineering.
In my view there is something special about human beings that's different from what we see in any of the other animals. By happenstance of evolution we were the first species to be able to create technology. Actually there were others, but we are the only one that survived in this ecological niche.
But we combined a rational faculty, the ability to think logically, to create abstractions, to create models of the world in our own minds, and to manipulate the world. We have opposable thumbs so that we can create technology, but technology is not just tools. Other animals have used primitive tools, but the difference is actually a body of knowledge that changes and evolves itself from generation to generation.
The knowledge that the human species has is another one of those exponential trends. We use one stage of technology to create the next stage, which is why technology accelerates, why it grows in power. Today, for example, a computer designer has these tremendously powerful computer system design tools to create computers, so in a couple of days they can create a very complex system and it can all be worked out very quickly. The first computer designers had to actually draw them all out in pen on paper. Each generation of tools creates the power to create the next generation.
So technology itself is an exponential, evolutionary process that is a continuation of the biological evolution that created humanity in the first place. Biological evolution itself evolved in an exponential manner. Each stage created more powerful tools for the next, so when biological evolution created DNA it now had a means of keeping records of its experiments so evolution could proceed more quickly.
Because of this, the Cambrian explosion only lasted a few tens of millions of years, whereas the first stage of creating DNA and primitive cells took billions of years. In the next epoch this species that ushered in its own evolutionary process — that is, its own cultural and technological evolution, as no other species has — will combine with its own creation and will merge with its technology.
At some level that's already happening, even if most of us don't necessarily have them yet inside our bodies and brains, since we're very intimate with the technology—it's in our pockets. We've certainly expanded the power of the mind of the human civilization through the power of its technology. What is unique about human beings is our ability to create abstract models and to use these mental models to understand the world and do something about it. These mental models have become more and more sophisticated, and by becoming embedded in technology, they have become very elaborate and very powerful.
Now we can actually understand our own minds. This ability to scale up the power of our own civilization is what's unique about human beings. Patterns are the fundamental ontological reality, because they are what persists, not anything physical. Take myself, Ray Kurzweil. What is Ray Kurzweil? Is it this stuff here? Well, this stuff changes very quickly. Some of our cells turn over in a matter of days. Even our skeleton, which you think probably lasts forever because we find skeletons that are centuries old, changes over within a year.
Many of our neurons change over. But more importantly, the particles making up the cells change over even more quickly, so even if a particular cell is still there the particles are different. So I'm not the same stuff, the same collection of atoms and molecules that I was a year ago. But what does persist is that pattern. The pattern evolves slowly, but the pattern persists. So we're kind of like the pattern that water makes in a stream; you put a rock in there and you'll see a little pattern. The water is changing every few milliseconds; if you come a second later, it's completely different water molecules, but the pattern persists.
He speculated on the effects of superhuman machines, should they ever be invented: Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. Good's scenario runs as follows: This superintelligent machine then designs an even more capable machine, or re-writes its own software to become even more intelligent; this even more capable machine then goes on to design a machine of yet greater capability, and so on.
These iterations of recursive self-improvement accelerate, allowing enormous qualitative change before any upper limits imposed by the laws of physics or theoretical computation set in. A superintelligence, hyperintelligence, or superhuman intelligence is a hypothetical agent that possesses intelligence far surpassing that of the brightest and most gifted human minds. John von Neumann, Vernor Vinge and Ray Kurzweil define the concept in terms of the technological creation of super intelligence.
They argue that it is difficult or impossible for present-day humans to predict what human beings' lives would be like in a post-singularity world. Technology forecasters and researchers disagree about if or when human intelligence is likely to be surpassed. Some argue that advances in artificial intelligence AI will probably result in general reasoning systems that lack human cognitive limitations. Others believe that humans will evolve or directly modify their biology so as to achieve radically greater intelligence.
A number of futures studies scenarios combine elements from both of these possibilities, suggesting that humans are likely to interface with computers , or upload their minds to computers , in a way that enables substantial intelligence amplification. Some writers use "the singularity" in a broader way to refer to any radical changes in our society brought about by new technologies such as molecular nanotechnology ,    although Vinge and other writers specifically state that without superintelligence, such changes would not qualify as a true singularity.
Many prominent technologists and academics dispute the plausibility of a technological singularity, including Paul Allen , Jeff Hawkins , John Holland , Jaron Lanier , and Gordon Moore , whose law is often cited in support of the concept. Most proposed methods for creating superhuman or transhuman minds fall into one of two categories: The means speculated to produce intelligence augmentation are numerous, and include bioengineering , genetic engineering , nootropic drugs, AI assistants, direct brain—computer interfaces and mind uploading.
The existence of multiple paths to an intelligence explosion makes a singularity more likely; for a singularity to not occur they would all have to fail. Hanson is skeptical of human intelligence augmentation, writing that once one has exhausted the "low-hanging fruit" of easy methods for increasing human intelligence, further improvements will become increasingly difficult to find. Despite the numerous speculated means for amplifying human intelligence, non-human artificial intelligence specifically seed AI is the most popular option for organizations [ which?
Whether or not an intelligence explosion occurs depends on three factors. Contrariwise, as the intelligences become more advanced, further advances will become more and more complicated, possibly overcoming the advantage of increased intelligence. Each improvement must be able to beget at least one more improvement, on average, for the singularity to continue. Finally the laws of physics will eventually prevent any further improvements. There are two logically independent, but mutually reinforcing causes of intelligence improvements: On the other hand, most AI researchers [ who?
A email survey of authors with publications at the NIPS and ICML machine learning conferences asked them about the chance of an intelligence explosion. Both for human and artificial intelligence, hardware improvements increase the rate of future hardware improvements. Oversimplified,  Moore's Law suggests that if the first doubling of speed took 18 months, the second would take 18 subjective months; or 9 external months, whereafter, four months, two months, and so on towards a speed singularity. Hawkins [ citation needed ] , responding to Good, argued that the upper limit is relatively low;.
Belief in this idea is based on a naive understanding of what intelligence is. As an analogy, imagine we had a computer that could design new computers chips, systems, and software faster than itself. Would such a computer lead to infinitely fast computers or even computers that were faster than anything humans could ever build? It might accelerate the rate of improvements for a while, but in the end there are limits to how big and fast computers can be.
We would end up in the same place; we'd just get there a bit faster. There would be no singularity. Whereas if it were a lot higher than current human levels of intelligence, the effects of the singularity would be great enough as to be indistinguishable to humans from a singularity with an upper limit.
For example, if the speed of thought could be increased a million-fold, a subjective year would pass in 30 physical seconds. It is difficult to directly compare silicon -based hardware with neurons. But Berglas notes that computer speech recognition is approaching human capabilities, and that this capability seems to require 0. This analogy suggests that modern computer hardware is within a few orders of magnitude of being as powerful as the human brain.
The exponential growth in computing technology suggested by Moore's law is commonly cited as a reason to expect a singularity in the relatively near future, and a number of authors have proposed generalizations of Moore's law. Computer scientist and futurist Hans Moravec proposed in a book  that the exponential growth curve could be extended back through earlier computing technologies prior to the integrated circuit. Ray Kurzweil postulates a law of accelerating returns in which the speed of technological change and more generally, all evolutionary processes  increases exponentially, generalizing Moore's law in the same manner as Moravec's proposal, and also including material technology especially as applied to nanotechnology , medical technology and others.
Kurzweil reserves the term "singularity" for a rapid increase in artificial intelligence as opposed to other technologies , writing for example that "The Singularity will allow us to transcend these limitations of our biological bodies and brains There will be no distinction, post-Singularity, between human and machine".
Some singularity proponents argue its inevitability through extrapolation of past trends, especially those pertaining to shortening gaps between improvements to technology. In one of the first uses of the term "singularity" in the context of technological progress, Stanislaw Ulam tells of a conversation with John von Neumann about accelerating change:.
- The Rake: A Novel.
- Technological singularity.
- History of Michigan (Volume 1).
One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue. Kurzweil claims that technological progress follows a pattern of exponential growth , following what he calls the " law of accelerating returns ". Whenever technology approaches a barrier, Kurzweil writes, new technologies will surmount it. He predicts paradigm shifts will become increasingly common, leading to "technological change so rapid and profound it represents a rupture in the fabric of human history".
Oft-cited dangers include those commonly associated with molecular nanotechnology and genetic engineering.
These threats are major issues for both singularity advocates and critics, and were the subject of Bill Joy 's Wired magazine article " Why the future doesn't need us ". Some intelligence technologies, like "seed AI",   may also have the potential to make themselves more efficient, not just faster, by modifying their source code. These improvements would make further improvements possible, which would make further improvements possible, and so on.
The mechanism for a recursively self-improving set of algorithms differs from an increase in raw computation speed in two ways. First, it does not require external influence: While speed increases seem to be only a quantitative difference from human intelligence, actual algorithm improvements would be qualitatively different. Eliezer Yudkowsky compares it to the changes that human intelligence brought: Similarly, the evolution of life had been a massive departure and acceleration from the previous geological rates of change, and improved intelligence could cause change to be as different again.
There are substantial dangers associated with an intelligence explosion singularity originating from a recursively self-improving set of algorithms. First, the goal structure of the AI may not be invariant under self-improvement, potentially causing the AI to optimise for something other than was intended. While not actively malicious, there is no reason to think that AIs would actively promote human goals unless they could be programmed as such, and if not, might use the resources currently used to support mankind to promote its own goals, causing human extinction.
Carl Shulman and Anders Sandberg suggest that algorithm improvements may be the limiting factor for a singularity because whereas hardware efficiency tends to improve at a steady pace, software innovations are more unpredictable and may be bottlenecked by serial, cumulative research. They suggest that in the case of a software-limited singularity, intelligence explosion would actually become more likely than with a hardware-limited singularity, because in the software-limited case, once human-level AI was developed, it could run serially on very fast hardware, and the abundance of cheap hardware would make AI research less constrained.
Some critics, like philosopher Hubert Dreyfus , assert that computers or machines can't achieve human intelligence , while others, like physicist Stephen Hawking , hold that the definition of intelligence is irrelevant if the net result is the same. Psychologist Steven Pinker stated in There is not the slightest reason to believe in a coming singularity. The fact that you can visualize a future in your imagination is not evidence that it is likely or even possible. Look at domed cities, jet-pack commuting, underwater cities, mile-high buildings, and nuclear-powered automobiles—all staples of futuristic fantasies when I was a child that have never arrived.
Sheer processing power is not a pixie dust that magically solves all your problems. University of California, Berkeley , philosophy professor John Searle writes:. We design them to behave as if they had certain sorts of psychology , but there is no psychological reality to the corresponding processes or behavior. Martin Ford in The Lights in the Tunnel: Automation, Accelerating Technology and the Economy of the Future  postulates a "technology paradox" in that before the singularity could occur most routine jobs in the economy would be automated, since this would require a level of technology inferior to that of the singularity.
This would cause massive unemployment and plummeting consumer demand, which in turn would destroy the incentive to invest in the technologies that would be required to bring about the Singularity. Job displacement is increasingly no longer limited to work traditionally considered to be "routine. Theodore Modis   and Jonathan Huebner  argue that the rate of technological innovation has not only ceased to rise, but is actually now declining.
Evidence for this decline is that the rise in computer clock rates is slowing, even while Moore's prediction of exponentially increasing circuit density continues to hold. This is due to excessive heat build-up from the chip, which cannot be dissipated quickly enough to prevent the chip from melting when operating at higher speeds. Advancements in speed may be possible in the future by virtue of more power-efficient CPU designs and multi-cell processors.
Others  propose that other "singularities" can be found through analysis of trends in world population , world gross domestic product , and other indices. Andrey Korotayev and others argue that historical hyperbolic growth curves can be attributed to feedback loops that ceased to affect global trends in the s, and thus hyperbolic growth should not be expected in the future.
In a detailed empirical accounting, The Progress of Computing , William Nordhaus argued that, prior to , computers followed the much slower growth of a traditional industrial economy, thus rejecting extrapolations of Moore's law to 19th-century computers. In a paper, Schmidhuber stated that the frequency of subjectively "notable events" appears to be approaching a 21st-century singularity, but cautioned readers to take such plots of subjective events with a grain of salt: Paul Allen argues the opposite of accelerating returns, the complexity brake;  the more progress science makes towards understanding intelligence, the more difficult it becomes to make additional progress.
A study of the number of patents shows that human creativity does not show accelerating returns, but in fact, as suggested by Joseph Tainter in his The Collapse of Complex Societies ,  a law of diminishing returns. The number of patents per thousand peaked in the period from to , and has been declining since. Jaron Lanier refutes the idea that the Singularity is inevitable.
It's not an autonomous process. If you structure a society on not emphasizing individual human agency, it's the same thing operationally as denying people clout, dignity, and self-determination Standard of Living Since the Civil War , points out that measured economic growth has slowed around and slowed even further since the financial crisis of , and argues that the economic data show no trace of a coming Singularity as imagined by mathematician I.
In addition to general criticisms of the singularity concept, several critics have raised issues with Kurzweil's iconic chart. One line of criticism is that a log-log chart of this nature is inherently biased toward a straight-line result. Others identify selection bias in the points that Kurzweil chooses to use. For example, biologist PZ Myers points out that many of the early evolutionary "events" were picked arbitrarily. The Economist mocked the concept with a graph extrapolating that the number of blades on a razor, which has increased over the years from one to as many as five, will increase ever-faster to infinity.
Dramatic changes in the rate of economic growth have occurred in the past because of some technological advancement. Based on population growth, the economy doubled every , years from the Paleolithic era until the Neolithic Revolution. The new agricultural economy doubled every years, a remarkable increase. If the rise of superhuman intelligence causes a similar revolution, argues Robin Hanson, one would expect the economy to double at least quarterly and possibly on a weekly basis. The term "technological singularity" reflects the idea that such change may happen suddenly, and that it is difficult to predict how the resulting new world would operate.
While the technological singularity is usually seen as a sudden event, some scholars argue the current speed of change already fits this description. In addition, some argue that we are already in the midst of a major evolutionary transition that merges technology, biology, and society. Digital technology has infiltrated the fabric of human society to a degree of indisputable and often life-sustaining dependence. We spend most of our waking time communicating through digitally mediated channels With one in three marriages in America beginning online, digital algorithms are also taking a role in human pair bonding and reproduction".
The article further argues that from the perspective of the evolution , several previous Major Transitions in Evolution have transformed life through innovations in information storage and replication RNA , DNA , multicellularity , and culture and language. In the current stage of life's evolution, the carbon-based biosphere has generated a cognitive system humans capable of creating technology that will result in a comparable evolutionary transition.
The digital information created by humans has reached a similar magnitude to biological information in the biosphere. Since the s, "the quantity of digital information stored has doubled about every 2. In biological terms, there are 7. The digital realm stored times more information than this in see fgure. The total amount of DNA contained in all of the cells on Earth is estimated to be about 5.
This would represent a doubling of the amount of information stored in the biosphere across a total time period of just years". The goal was to discuss the potential impact of the hypothetical possibility that robots could become self-sufficient and able to make their own decisions. They discussed the extent to which computers and robots might be able to acquire autonomy , and to what degree they could use such abilities to pose threats or hazards.
Some machines are programmed with various forms of semi-autonomy, including the ability to locate their own power sources and choose targets to attack with weapons. Also, some computer viruses can evade elimination and, according to scientists in attendance, could therefore be said to have reached a "cockroach" stage of machine intelligence. The conference attendees noted that self-awareness as depicted in science-fiction is probably unlikely, but that other potential hazards and pitfalls exist.
Berglas claims that there is no direct evolutionary motivation for an AI to be friendly to humans. Evolution has no inherent tendency to produce outcomes valued by humans, and there is little reason to expect an arbitrary optimisation process to promote an outcome desired by mankind, rather than inadvertently leading to an AI behaving in a way not intended by its creators such as Nick Bostrom's whimsical example of an AI which was originally programmed with the goal of manufacturing paper clips, so that when it achieves superintelligence it decides to convert the entire planet into a paper clip manufacturing facility.
Anders Sandberg has also elaborated on this scenario, addressing various common counter-arguments. Bostrom discusses human extinction scenarios, and lists superintelligence as a possible cause:. When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal.
We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question. A significant problem is that unfriendly artificial intelligence is likely to be much easier to create than friendly AI. While both require large advances in recursive optimisation process design, friendly AI also requires the ability to make goal structures invariant under self-improvement or the AI could transform itself into something unfriendly and a goal structure that aligns with human values and does not automatically destroy the human race.
An unfriendly AI, on the other hand, can optimize for an arbitrary goal structure, which does not need to be invariant under self-modification. Eliezer Yudkowsky proposed that research be undertaken to produce friendly artificial intelligence in order to address the dangers. He noted that the first real AI would have a head start on self-improvement and, if friendly, could prevent unfriendly AIs from developing, as well as providing enormous benefits to mankind.
Bill Hibbard proposes an AI design that avoids several dangers including self-delusion,  unintended instrumental actions,   and corruption of the reward generator. It also proposed a simple design that was vulnerable to corruption of the reward generator. One hypothetical approach towards attempting to control an artificial intelligence is an AI box , where the artificial intelligence is kept constrained inside a simulated world and not allowed to affect the external world.
However, a sufficiently intelligent AI may simply be able to escape by outsmarting its less intelligent human captors. Stephen Hawking said in that "Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks. So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right? If a superior alien civilisation sent us a message saying, "We'll arrive in a few decades," would we just reply, "OK, call us when you get here — we'll leave the lights on"?
Probably not — but this is more or less what is happening with AI.