Text extracted via OCR from the original document. May contain errors from the scanning process.
12.9 AGI Ethics As Related to Various Future Scenarios 237
pragmatic underlying ethical pedagogy and experiential grounding may prove to be even more
dangerous than our messy human mental ecologies.
If one assumes a hard takeoff AGI, then all this must be codified in the system at launch,
as once a potentially Singularitarian AGI is launched there is no way to know what time
period constitutes “before the singularity point.” This means developing theory of mind empathy
and logical ethics in code prior to giving the system unfettered access to hardware and self-
modification code. However, though nobody can predict if or when a Singularity will occur
after unrestricted launch, only a truly irresponsible AGI development team would attempt to
create an AGI without first experimenting with ethical training of the system in an intelligence-
capped form, by means of ethical instruction via human-AGI interaction both pedagogically
and experientially.
12.9.4 Global Brain Mindplex Scenarios
Another class of scenarios — overlapping some of the previous ones — involves the emergence
of a “Global Brain,” an emergent intelligence formed from global communication networks in-
corporating humans and software programs in a larger body of self-organizing dynamics. The
notion of the Global Brain is reviewed in [Hey07, Tur77] and its connection with advanced
AI is discussed in detail in Goertzel’s book Creating Internet Intelligence [Goe01], where three
possible phases of “Global Brain” development are articulated:
e Phase 1: computer and communication technologies as enhancers of human
interactions. This is what we have today: science and culture progress in ways that would
not be possible if not for the “digital nervous system” we’re spreading across the planet.
The network of idea and feeling sharing can become much richer and more productive than
it is today, just through incremental development, without any Metasystem transition.
e Phase 2: the intelligent Internet. At this point our computer and communication sys-
tems, through some combination of self-organizing evolution and human engineering, have
become a coherent mind on their own, or a set of coherent minds living in their own digital
environment.
e Phase 3: the full-on Singularity. A complete revision of the nature of intelligence, human
and otherwise, via technological and intellectual advancement totally beyond the scope of
our current comprehension. At this point our current psychological and cultural realities
are no more relevant than the psyche of a goose is to modern society.
The main concern of Creating Internet Intelligence is with
e how to get from Phase 1 to Phase 2 - i.e. how to build an AGI system that will effect or
encourage the transformation of the Internet into a coherent intelligent system
e how to ensure that the Phase 2, Internet-savvy, global-brain-centric AGI systems will be
oriented toward intelligence-improving self-modification (so they’ll propel themselves to
Phase 3), and also toward generally positive goals (as opposed to, say, world domination
and extermination of all other intelligent life forms besides themselves!)
One possibly useful concept in this context is that of a mindplex: an intelligence that is
composed largely of individual intelligences with their own self-models and global workspaces,
HOUSE_OVERSIGHT_013153