Douglas Hofstadter, pudding and pie,Douglas Hofstadter has written popular speculative books such as Gödel, Escher, Bach: An Eternal Golden Braid and I Am a Strange Loop. He is fond of thought experiments that cannot be carried out in practice, such as “What if all the neurons in your brain would be replaced one by one by synthetic equivalents?” or “What if a teleportation device were to make an exact duplicate of you?”. He then purports to make deep and profound conclusions about consciousness and free will from such thought experiments. But they are fundamentally flawed: they are so far removed from our abilities and knowledge that it may very well be that they cannot be carried out or that carrying them out would require completely new understanding.
Kissed the integers and made them cry.
But when the infinities came out to play,
Douglas Hofstadter ran away.
— adapted by Peter Van Roy from Georgie Porgie.
John Searle's Chinese Room thought experiment has the same flaw and so does Raymond Kurzweil's breathless prediction of the Singularity. It is like jumping six inches in the air and then deducing from this what space travel might be like. In the year 1957, people told of the power of laptop computers in 2007 might very well think that they would be intelligent, which is far from the case. The barriers between, e.g., the fundamental equations of physics and human experience, are so wide that we cannot make any firm conclusions about them without actually traversing them. Per Brand calls them infinity barriers. Glibly jumping over these barriers as if they did not exist leads only to wild speculation, not to firm knowledge.