Bill Hibbard grapples with transhumanity and the social contract and raises some provocative challenges. I especially like his attempt to define what greater-than-human intelligence would look like:
The most familiar measure of intelligence is IQ, but it is difficult to understand what a machine IQ of a million or a billion would mean. As is often pointed out, intelligence cannot be measured by a single number. But one measure of a mind’s intelligence, relevant to power in the human world, is the number of humans the mind is capable of knowing well. This will become a A practical measure of intelligence, as we develop machines much more intelligent than humans., is the number of people that a mind can know well. Humans evolved an ability to know about 200 other people well, driven by the selective advantage of working in groups (Bownds 1999). Now Google is working hard to develop intelligence in its their enormous servers, which already keep records of the search histories of hundreds of millions of users. As these servers develop the ability to converse in human languages, these search histories will evolve into detailed simulation models of our minds. Ultimately, large servers will know billions of people well. This will give them enormous power to predict and influence economics and politics; rather than relying on population statistics, such a mind will know the political and economic behavior of almost everyone in detail.
I hadn't considered such a social definition of intelligence before.