Douglas Hofstadter, pudding and pie,Several popular authors are using specious arguments to speculate wildly and irresponsibly about the human mind. Douglas Hofstadter has written popular speculative books such as Gödel, Escher, Bach: An Eternal Golden Braid and I Am a Strange Loop. He is fond of thought experiments that cannot be carried out in practice, such as “What if all the neurons in your brain would be replaced one by one by synthetic equivalents?” or “What if a teleportation device were to make an exact duplicate of you?”.1 He then purports to make deep and profound conclusions about consciousness and free will from such thought experiments. But they are fundamentally flawed: they are so far removed from our abilities and knowledge that it may very well be that they cannot be carried out or that carrying them out would require completely new understanding.2
Kissed the integers and made them cry.
But when the infinities came out to play,
Douglas Hofstadter ran away.
— Adapted by Peter Van Roy from Georgie Porgie.
John Searle's Chinese Room thought experiment has the same flaw and so does Raymond Kurzweil's breathless prediction of the Singularity. It is like jumping six inches in the air and then deducing from this what space travel might be like.3 In the year 1957, people told of the power of laptop computers in 2007 might very well think that they would be intelligent, which is far from the case. Google's search engine uses enormously more computing power than a laptop and can answer many profound questions, but it is clearly an idiot savant: it shows no glimmering of intelligence. The barrier between the fundamental equations of physics and human experience is so wide that we cannot make any firm conclusions about it without actually traversing it. Per Brand calls it an example of an infinity barrier.4. Glibly jumping over such a barrier as if it did not exist leads only to wild speculation, not to firm knowledge.
The outer limits of human intelligence are greatly underestimated. Consider a world-class mathematician or musician. This person starts when young with a promising innate ability and works hard over several decades to improve that ability. Over a long period of time, the conscious mind of the person is single-mindedly dedicated to this task. The final result of this effort is that the ability has progressed beyond the capacity of most people to imagine the level it has attained. Without some real understanding of how to quantify this level, it is ridiculous to pretend that it can easily be jumped over.
Concepts, Techniques, and Models of Computer Programming