computer ethic

1 ==================================================================== The Coming T ech nolo gica l Si ngula rity:

How to Survi ve in th e Po st-H uma n Era Verno r Vi nge Departmen t o f Ma th ema tica l Sci ence s San Diego St ate U nive rsity (c) 1993 by Ve rnor Vi nge (Verbatim copyin g/tra nsl atio n a nd d ist rib utio n o f th is entire article is p ermi tte d in a ny me diu m, p ro vided th is notice i s pre se rve d.) This article w as fo r th e VI SION-2 1 Symp osi um sponsored b y NASA Lew is Rese arch C ente r and the Ohio A ero sp ace In st it u te , Ma rch 3 0-3 1, 1 993.

It is also retrieva ble fro m th e N ASA te ch nica l re port s server as p art o f N ASA CP-1 0129.

A slightly chan ged ve rsion a ppeare d in th e Winter 1993 issu e o f _ W hole Ea rth R evi ew _.

Ab stra ct Within thirty y ears, w e w ill h ave th e te ch nolo gica l means to create su perh uma n in te llig ence . Sh ort ly afte r, the human era w ill b e e nded.

Is such progr ess avo id able ? If n ot to b e a vo id ed, ca n events be guide d so th at w e ma y su rvive ? T hese q uest io ns are investigated . So me p ossi ble a nsw ers (and so me fu rth er dangers) are pre sente d.

_What is The Singu la rity? _ The acceleration o f te ch nolo gica l p ro gre ss has been th e ce ntr a l feature of this centu ry. I a rg ue in th is paper th at w e a re o n th e e dge of change comparab le to th e ri se o f h uma n lif e o n Ea rth. T he p re ci se cause of this change is th e immi nent cre atio n b y te ch nolo gy of entities with greater th an h uma n in te llig ence . T here a re se vera l m eans by which science m ay ach ie ve th is bre akt hro ugh (a nd th is is anoth er reason for having c onfid ence th at th e e ve nt w ill o ccu r):

o The developme nt o f co mp ute rs th at a re " a w ake " a nd 2 superhumanly in te llig ent. (T o d ate , mo st co ntro ve rsy in th e area of AI relates to w heth er w e ca n cre ate h uma n e quiva le nce in a machine. Bu t if th e a nsw er is "ye s, w e ca n", th en th ere is little doubt that b ein gs mo re in te llig ent ca n b e co nst ru cted shortly thereafte r.

o Large computer netw orks (and th eir asso ciate d u se rs) may "w ake up" as a superhu manly in te llig ent e ntit y.

o Computer/human in te rf ace s ma y beco me so in tima te th at u se rs may reasonably b e co nsi dere d su perh uma nly in te llig ent.

o Biological science ma y fin d w ays to imp rove u pon th e n atu ra l human intellect. The first three po ssibilit ie s depend in la rg e p art o n improvements in co mpute r hard ware . Pro gre ss in co mp ute r h ard w are h as followed an amazin gly steady cu rve in th e la st fe w d eca des [1 6]. B ase d largely on this trend, I b elie ve th at th e cre atio n o f g re ate r th an human intelligence w ill o ccu r durin g th e n ext th irt y ye ars. ( C harle s Platt [19] has pointed o ut th e AI e nth usi ast s have b een ma king cl aim s like this for the last t hirt y ye ars. Ju st so I'm not g uilt y of a relative-time ambig uit y, le t me mo re sp eci fic: I'l l b e su rprise d if this event occurs be fo re 2 005 o r afte r 2030.) What are the co nse quence s of th is eve nt? W hen g re ate r-th an-h um an intelligence drives pro gre ss, th at p ro gre ss will b e mu ch mo re r a pid .

In fact, there seems no re aso n w hy pro gre ss it se lf w ould n ot in vo lve the creation of still mo re in te llig ent e ntit ie s -- on a st ill-sh ort er time scale. The best a nalo gy th at I se e is wit h th e e vo lu tio nary past :

Animals can adapt to p ro ble ms and ma ke in ve ntio ns, b ut o fte n n o fa st er than natural selectio n ca n d o it s work -- th e w orld a ct s as it s ow n simulator in the case o f n atu ra l se le ct io n. W e h uma ns have th e a bilit y to internalize the worl d a nd co nduct " w hat if 's" in o ur heads; w e ca n solve many problem s th ousa nds of time s fa st er th an n atu ra l se le ct io n.

Now, by creating th e me ans to e xe cute th ose si mu la tio ns at m uch h ig her speeds, we are ente rin g a re gime a s ra dica lly dif fe re nt fro m o ur h um an past as we humans are fro m th e lo w er anima ls.

From the human p oin t o f vi ew th is ch ange w ill b e a th ro win g a w ay of all the previous ru le s, p erh aps in th e b lin k of a n e ye , a n exponential runawa y beyo nd a ny hope o f co ntro l. D eve lo pm ents th at before were though t mi ght o nly happen in " a mi llio n ye ars" ( if e ve r) will likely happen in th e n ext ce ntu ry. (I n [4 ], G re g Be ar pain ts a picture of the major ch anges happenin g in a ma tte r of h ours. ) 3 I think it's fair to ca ll th is eve nt a si ngula rity ("th e Singularity" for the p urp ose s of th is paper). It is a p oin t w here o ur models must be disca rded a nd a n ew re alit y ru le s. As we m ove cl ose r and closer to this poi nt, it w ill lo om vaster and va ster ove r hum an affairs till the notion beco me s a co mmo npla ce . Ye t w hen it fin ally happens it may still b e a g re at su rprise a nd a g re ate r unkn ow n. In the 1950s there we re ve ry fe w w ho sa w it : St an U la m [2 7] p ara phra se d John von Neumann as sa ying:

One conversatio n ce nte re d o n th e e ve r acce le ra tin g p ro gre ss of technology and changes in th e mo de o f h uma n lif e , w hich g ive s th e appearance of a ppro ach in g so me e sse ntia l si ngula rity in th e history of the rac e b eyo nd w hich h uma n a ffa irs, a s we kn ow th em , could not continu e.

Von Neumann e ven u se s th e te rm singula rity, th ough it a ppears he is still thinking of no rmal p ro gre ss, n ot th e cre atio n o f su perh um an intellect. (For me, th e su perh uma nit y is th e e sse nce o f th e Singularity. Without th at w e w ould g et a g lu t o f te ch nica l ri ch es, never properly absorb ed (se e [2 4]). ) In the 1960s the re w as re co gnit io n o f so me o f th e imp lica tio ns of superhuman intellig ence . I. J. G ood w ro te [1 0]:

Let an ultraintell ig ent ma chin e b e d efin ed a s a ma chin e that can far surp ass all th e in te lle ct ual a ct ivi tie s of a ny any man howeve r cl eve r. Si nce th e d esi gn o f ma chin es is one o f these intellectua l a ct ivi tie s, a n u lt ra in te llig ent ma chin e co uld design even bet te r ma chin es; th ere w ould th en u nquest io nably be an "intelligen ce e xp lo si on," a nd th e in te llig ence o f ma n would be left far behin d. T hus th e first u lt ra in te llig ent machine is the _ la st _ in ve ntio n th at ma n n eed e ve r ma ke, provided that the ma chin e is doci le e nough to te ll u s how to keep it under co ntro l.

... It is more probab le th an n ot th at, w it h in th e tw entie th ce ntu ry, an ultraintelligen t ma chin e w ill b e b uilt a nd th at it w ill b e the last invention th at ma n n eed ma ke.

Good has capture d th e e sse nce o f th e ru naw ay, b ut d oes not p ursu e its most disturbing co nse quence s. An y in te llig ent ma chin e o f th e so rt he describes would n ot b e h uma nki nd's "to ol" -- any mo re th an h um ans are the tools of rabb it s or ro bin s or ch imp anze es.

4 Through the '60s and '7 0s and '8 0s, re co gnit io n o f th e ca ta cl ysm spread [28] [1] [30] [ 4]. Pe rhaps it w as th e sci ence -fict io n w rit e rs who felt the first concre te imp act . Af te r all, th e " h ard " science-fiction writers are th e o nes who try to w rite sp eci fic storie s about all that techno lo gy ma y do fo r us. Mo re a nd mo re, th ese w rit e rs felt an opaque wall acro ss th e fu tu re . O nce , th ey co uld p ut su ch fantasies millions o f ye ars in th e fu tu re [2 3]. N ow th ey sa w th at their most diligent e xtra pola tio ns re su lt e d in th e u nkn ow able ...

soon. Once, galacti c emp ire s mi ght h ave se eme d a Po st-H um an d om ain .

Now, sadly, even in te rp la neta ry ones are .

What about the '9 0s and th e '0 0s and th e '1 0s, a s we sl id e to w ard the edge? How will t he a ppro ach o f th e Si ngula rity sp read a cr oss th e human world view? F or a w hile ye t, th e g enera l cri tics of m ach in e sapience will have go od p re ss. Af te r all, till w e h ave h ard ware a s powerful as a human b ra in it is pro bably fo olish to th in k we'l l b e able to create huma n e quiva le nt (o r gre ate r) in te llig ence . (T here is the far-fetched possi bilit y th at w e co uld ma ke a h uma n e quiva le nt o ut of less powerful har dw are , if w ere w illin g to g ive u p sp eed, if w e were willing to settle fo r an a rt if ici al b ein g w ho w as lit e ra lly slow [29]. But it's much m ore like ly th at d evi sing th e so ftw are w ill b e a tricky process, invol ving lo ts of fa lse st art s and e xp erime nta tio n. If so, then the arrival o f se lf -a ware ma chin es will n ot h appen till a fte r the development of h ard ware th at is su bst antia lly mo re p ow erfu l th an humans' natural equ ip me nt.) But as time pass es, w e sh ould se e mo re symp to ms. T he d ile m ma fe lt by science fiction wri te rs will b e p erce ive d in o th er cre ative endeavors. (I have h eard th oughtfu l co mic book write rs worry about how to have spectacu la r effe ct s when e ve ryt hin g vi sible ca n b e produced by the tech nica lly co mmo npla ce .) W e w ill se e a uto m atio n replacing higher an d h ig her le ve l jo bs. W e h ave to ols rig ht n ow (symbolic math prog rams, ca d/ca m) th at re le ase u s fro m most lo w -le ve l drudgery. Or put ano th er w ay: T he w ork th at is tru ly pro duct ive is th e domain of a steadily smalle r and mo re e lit e fra ctio n o f h um anit y. In the coming of the S in gula rity, w e a re se ein g th e p re dict io ns of _ tr u e_ technological unemp lo yme nt fin ally co me tru e.

Another symptom o f p ro gre ss to w ard th e Si ngula rity: id eas themselves should sp read e ve r fa st er, a nd e ve n th e mo st r a dica l w ill quickly become commo npla ce . W hen I b egan w ritin g, it se em ed ve ry easy to come up with idea s th at to ok deca des to p erco la te in to th e cu lt u ra l 5 consciousness; now th e le ad time se ems more like e ig hte en m onth s. ( O f course, this could ju st b e me lo si ng my ima gin atio n a s I g et o ld , b ut I see the effect in ot hers to o.) Like th e sh ock in a co mp ressi ble flow, the Singularity mo ves close r as we a cce le ra te th ro ugh th e critical speed. And what of the arri va l o f th e Si ngula rity it se lf ? W hat ca n b e said of its actual ap peara nce ? Si nce it in vo lve s an in te lle ct ual runaway, it will prob ably occu r fa st er th an a ny te ch nica l re vo lu tio n seen so far. The pr eci pit a tin g e ve nt w ill like ly be u nexp ect ed - - perhaps even to the re se arch ers in vo lve d. (" Bu t a ll o ur pre vious models were catato nic! W e w ere ju st tw eaki ng so me p ara mete rs. ..." ) If networking is widesp read e nough (i n to u biq uit o us emb edded syst em s), it may seem as if our a rt if a ct s as a w hole h ad su ddenly wake ned.

And what happe ns a mo nth o r tw o (o r a d ay or tw o) afte r th at? I have only analogies to p oin t to : T he ri se o f h uma nki nd. W e w ill b e in the Post-Human era. An d fo r all my ramp ant te ch nolo gica l o ptim ism , sometimes I think I'd b e mo re co mf ort able if I w ere re gard in g th ese transcendental eve nts fro m one th ousa nd ye ars remo ve ... in st ead o f twenty. _Can the Singularity be Avo id ed?_ Well, maybe it wo n't h appen a t a ll: So me time s I try to im agin e the symptoms that w e sh ould e xp ect to se e if th e Si ngula rity is not to develop. There are th e w id ely re sp ect ed a rg ume nts of Pe nro se [1 8] a nd Searle [21] against t he p ra ctica lit y of ma chin e sa pie nce . In A ugust of 1992, Thinking Ma chin es Corp ora tio n h eld a w orksh op to in ve stig ate the question "How W e W ill Bu ild a Ma chin e th at T hin ks" [T hearlin g]. A s you might guess fro m th e w orksh op's tit le , th e p art ici pants were n ot especially supportiv e o f th e a rg ume nts again st ma chin e in te llig ence .

In fact, there was ge nera l a gre eme nt th at mi nds ca n e xi st o n nonbiological subst rate s and th at a lg orith ms are o f ce ntra l im porta nce to the existence of mi nds. H ow eve r, th ere w as mu ch d ebate a bout th e raw hardware powe r th at is pre se nt in o rg anic bra in s. A mi norit y fe lt that the largest 199 2 co mp ute rs were w it h in th re e o rd ers of m agnit u de of the power of the h uma n b ra in . T he ma jo rity of th e p art ici pants agreed with Morave c's est ima te [1 6] th at w e a re te n to fo rt y ye ars away from hardwar e p arity. An d ye t th ere w as anoth er mi norit y who pointed to [6] [20], a nd co nje ct ure d th at th e co mp uta tio nal co mpete nce of single neurons ma y be fa r hig her th an g enera lly belie ve d. If so , our present compute r hard ware mi ght b e a s mu ch a s _te n_ o rd ers of 6 magnitude short of th e e quip me nt w e ca rry aro und in o ur heads. If th is is true (or for that m atte r, if th e Pe nro se o r Se arle cri tiq ue is valid), we might nev er se e a Si ngula rity. In st ead, in th e e arly '0 0s we would find our h ard ware p erf orma nce cu rve s begin to le ve l o ff - - this caused by our in abilit y to a uto ma te th e co mp le xi ty of th e d esi gn work necessary to su pport th e h ard ware tre nd cu rve s. W e'd e nd u p w it h some _very_ powerf ul h ard ware , b ut w it h out th e a bilit y to p ush it further. Commercial d ig it a l si gnal p ro ce ssi ng mi ght b e a w eso me, giving an analog ap peara nce e ve n to d ig it a l o pera tio ns, b ut n oth in g would ever "wake up " a nd th ere w ould n eve r be th e in te lle ct ual r u naw ay which is the essenc e o f th e Si ngula rity. It w ould like ly be se en a s a golden age ... and i t w ould a lso b e a n e nd o f p ro gre ss. T his is ve ry like the future predi cted b y Gunth er St ent. In fa ct , o n p age 1 37 o f [24], Stent explicitly ci te s th e d eve lo pme nt o f tra nsh uma n intelligence as a suf fici ent co ndit io n to b re ak his pro je ct io ns.

But if the techno lo gica l Si ngula rity ca n h appen, it w ill. Ev en if all the government s of th e w orld w ere to u nderst and th e " th re at" and be in deadly fe ar of it , p ro gre ss to w ard th e g oal w ould co ntin ue.

In fiction, there hav e b een st orie s of la w s passe d fo rb id din g th e construction of "a ma chin e in th e fo rm of th e mi nd o f ma n" [1 2]. In fact, the competitive adva nta ge -- eco nomi c, mi lit a ry, e ve n a rtist ic -- of every advance in a uto ma tio n is so co mp ellin g th at p assi ng la w s, or having customs, t hat fo rb id su ch th in gs me rely assu res th at so meone else will get them first .

Eric Drexler [7] ha s pro vided sp ect acu la r in si ght a bout h ow fa r technical improvem ent ma y go. H e a gre es th at su perh uma n in te llig ence s will be available in th e n ear fu tu re -- and th at su ch e ntit ie s pose a threat to the human statu s quo. Bu t D re xler arg ues th at w e ca n e m bed such transhuman d evi ce s in ru le s or physi cal co nfin eme nt su ch th at their results can be exa mined a nd u se d sa fe ly. T his is I. J. G ood's ultraintelligent mach in e, w it h a d ose o f ca utio n. I a rg ue th at confinement is intrinsi cally imp ractica l. F or th e ca se o f p hysi cal confinement: Imagin e yo urse lf co nfin ed to yo ur house w it h o nly lim it e d data access to the o utsi de, to yo ur ma sters. If th ose ma sters th ought at a rate -- say -- one mi llio n time s slow er th an yo u, th ere is lit tle doubt that over a pe rio d o f ye ars (yo ur time ) yo u co uld co me u p w it h "helpful advice" tha t w ould in ci denta lly se t yo u fre e. (I ca ll th is "fast thinking" form o f su perin te llig ence " w eak su perh uma nit y" . S uch a "weakly superhuma n" e ntit y would p ro bably burn o ut in a fe w w eeks of outside time. "Stron g su perh uma nit y" w ould b e mo re th an cr anki ng u p the clock speed on a h uma n-e quiva le nt mi nd. It's hard to sa y 7 precisely what "stro ng su perh uma nit y" w ould b e like , b ut th e difference appears t o b e p ro fo und. Ima gin e ru nnin g a d og m in d a t ve ry high speed. Would a th ousa nd ye ars of d oggy livi ng a dd u p to a ny hum an insight? (Now if the d og mi nd w ere cl eve rly re wire d a nd _ th en_ r u n a t high speed, we mig ht se e so me th in g d if fe re nt....) Mo st sp ecu la tio ns about superintellige nce se em to b e b ase d o n th e w eakl y su perh um an model. I believe tha t o ur best g uesse s about th e p ost -Si ngula rit y world can be obtaine d b y th in ki ng o n th e n atu re o f st ro ng superhumanity. I wil l re tu rn to th is poin t la te r in th e p aper. ) The other appro ach to D re xleria n co nfin eme nt is to b uild _ ru le s_ into the mind of the create d su perh uma n e ntit y (Asi mov's Law s) . I think that performan ce ru le s strict e nough to b e sa fe w ould a lso produce a device w hose a bilit y was clearly in fe rio r to th e u nfe tte re d versions (and so huma n co mp etit io n w ould fa vo r th e d eve lo pm ent o f th e those more dangero us mo dels). St ill, th e Asi mov dre am is a w onderfu l one: Imagine a willin g sl ave , w ho h as 1000 time s yo ur ca pabilit ie s in every way. Imagine a cre atu re w ho co uld sa tisf y yo ur eve ry sa fe w ish (whatever that mea ns) and st ill h ave 9 9.9 % o f it s time fre e fo r o th er activities. There wo uld b e a n ew u nive rse w e n eve r re ally underst ood, but filled with benev ole nt g ods (though o ne o f _ my_ w ish es m ig ht b e to become one of them). If the Singularity ca n n ot b e p re ve nte d o r co nfin ed, ju st h ow b ad could the Post-Huma n e ra b e? W ell ... p re tty bad. T he p hysi cal extinction of the hum an ra ce is one p ossi bilit y. (O r as Eri c Dre xl er put it of nanotechnol ogy: G ive n a ll th at su ch te ch nolo gy ca n d o, perhaps governmen ts would si mp ly deci de th at th ey no lo nger n eed citizens!). Yet physi cal e xt in ct io n ma y not b e th e sca rie st possibility. Again, a nalo gie s: T hin k of th e d if fe re nt w ays we r e la te to animals. Some o f th e cru de p hysi cal a buse s are imp la usi ble , ye t....

In a Post-Human wo rld th ere w ould st ill b e p le nty of n ich es where human equivalent a uto ma tio n w ould b e d esi ra ble : e mb edded syst em s in autonomous devices, se lf -a ware d aemo ns in th e lo w er fu nct io nin g o f larger sentients. (A st rongly su perh uma n in te llig ence w ould like ly be a Society of Mind [1 5] w it h so me ve ry comp ete nt co mp onents. ) S om e of these human equ iva le nts mi ght b e u se d fo r noth in g mo re th an d ig it a l signal processing. Th ey would b e mo re like w hale s th an h um ans. O th ers might be very human -like , ye t w it h a o ne-si dedness, a _ dedica tio n_ that would put them in a me nta l h osp it a l in o ur era . T hough n one o f these creatures migh t b e fle sh -and-b lo od h uma ns, th ey mi ght b e th e closest things in the new e nvi ro me nt to w hat w e ca ll h uma n n ow . ( I. J.

Good had something to sa y about th is, th ough a t th is la te d ate th e 8 advice may be moo t: G ood [1 1] p ro pose d a " Me ta -G old en R ule ", which might be para phra se d a s "T re at yo ur in fe rio rs as yo u w ould b e treated by your sup erio rs. " It's a w onderf ul, p ara doxi ca l id ea ( a nd most of my friends d on't b elie ve it ) si nce th e g ame -theore tic payo ff is so hard to articulat e. Ye t if w e w ere a ble to fo llo w it , in so me sense that might say some th in g a bout th e p la usi bilit y of su ch ki ndness in this universe.) I have argued abo ve th at w e ca nnot p re ve nt th e Si ngula rit y, that its coming is an in evi ta ble co nse quence o f th e h uma ns' n atu ra l competitiveness and th e p ossi bilit ie s in here nt in te ch nolo gy. A nd ye t ... we are the initiato rs. Eve n th e la rg est a va la nch e is tri g gere d b y small things. We ha ve th e fre edom to e st ablish in it ia l co ndit io ns, make things happe n in w ays th at a re le ss in imi ca l th an o th ers. O f course (as with start in g a va la nch es), it ma y not b e cl ear w hat th e right guiding nudge re ally is:

_Other Paths to the Si ngula rity: In te llig ence Amp lif ica tio n_ When people sp eak of cre atin g su perh uma nly in te llig ent b ein gs, they are usually imag in in g a n AI p ro je ct . Bu t a s I n ote d a t th e beginning of this pap er, th ere a re o th er path s to su perh uma nit y.

Computer networks a nd h uma n-co mpute r in te rf ace s se em more m undane th an AI, and yet they cou ld le ad to th e Si ngula rity. I ca ll th is contrasting approac h In te llig ence Amp lif ica tio n (I A). IA is so meth in g that is proceeding v ery natu ra lly, in mo st ca ses not e ve n re co gnize d by its developers for what it is. Bu t e ve ry time o ur abilit y to a cce ss information and to co mmunica te it to o th ers is imp rove d, in so me se nse we have achieved an in cre ase o ve r natu ra l in te llig ence . Ev en n ow , th e team of a PhD hum an a nd g ood co mp ute r w orkst atio n (e ve n a n o ff- n et workstation!) could p robably ma x any writte n in te llig ence te st in existence. And it's very likel y th at IA is a mu ch e asi er ro ad to th e achievement of sup erh uma nit y th an p ure AI . In h uma ns, th e h ard est development problems have a lre ady been so lve d. Bu ild in g u p fr o m w it h in ourselves ought to b e e asi er th an fig urin g o ut first w hat w e r e ally are and then building ma chin es th at a re a ll o f th at. An d th ere is at least conjectural pre cedent fo r th is appro ach . C airn s-Smi th [5 ] h as speculated that biol ogica l lif e ma y have b egun a s an a dju nct to st ill more primitive life b ase d o n cryst allin e g ro wth . L yn n Ma rgulis [1 4] has made strong arg ume nts fo r th e vi ew th at mu tu alism is th e g re at 9 driving force in evol utio n.

Note that I am n ot p ro posi ng th at AI re se arch b e ig nore d o r le ss funded. What goes o n w it h AI w ill o fte n h ave a pplica tio ns in IA , a nd vice versa. I am sug gest in g th at w e re co gnize th at in n etw ork and interface research t here is so me th in g a s pro fo und (a nd p ote ntia l w ild ) as Artificial Intelligen ce. W it h th at in si ght, w e ma y se e p ro je ct s that are not as direct ly applica ble a s co nve ntio nal in te rf ace a nd network design wor k, b ut w hich se rve to a dva nce u s to w ard th e Singularity along the IA path .

Here are some p ossi ble p ro je ct s th at ta ke o n sp eci al significance, given th e IA poin t o f vi ew :

o Human/comput er te am auto ma tio n: T ake p ro ble ms th at a re n orm ally considered for pu rely ma chin e so lu tio n (l ike h ill-cl imb in g problems), and de sign p ro gra ms and in te rf ace s th at ta ke a advantage of huma ns' in tu it io n a nd a va ila ble co mp ute r hard w are .

Considering all t he b iza rre ness of h ig her dime nsi onal hill-climbing probl ems (and th e n eat a lg orith ms th at h ave b een devised for their so lu tio n), th ere co uld b e so me ve ry in te re st in g displays and cont rol to ols pro vided to th e h uma n te am mem ber.

o Develop human /co mp ute r symb io si s in a rt : C omb in e th e g ra phic generation capa bilit y of mo dern ma chin es and th e e st hetic sensibility of huma ns. O f co urse , th ere h as been a n e norm ous amount of research in d esi gnin g co mp ute r aid s fo r art ist s, a s labor saving too ls. I'm suggest in g th at w e e xp lici tly aim fo r a greater merging o f co mp ete nce , th at w e e xp lici tly re co gnize th e cooperative appro ach th at is possi ble . Ka rl Si ms [2 2] h as done wonderful work i n th is dire ctio n.

o Allow human/co mpute r te ams at ch ess to urn ame nts. W e a lr e ady have programs th at ca n p la y bette r th an a lmo st a ll h uma ns. B ut how much work h as been d one o n h ow th is pow er co uld b e u se d b y a human, to get some th in g e ve n b ette r? If su ch te ams were a llo w ed in at least some ch ess to urn ame nts, it co uld h ave th e p osi tive effect on IA rese arch th at a llo w in g co mp ute rs in to urn am ents had for the correspond in g n ich e in AI .

o Develop interfac es th at a llo w co mp ute r and n etw ork acce ss wit h out requiring the huma n to b e tie d to o ne sp ot, si ttin g in fro nt o f a computer. (This is an a sp ect o f IA th at fit s so w ell w it h kn ow n economic advant ages th at lo ts of e ffo rt is alre ady bein g sp ent o n it.) o Develop more symme tri ca l d eci sion su pport syst ems. A p opula r research/produc t a re a in re ce nt ye ars has been d eci sion su pport 10 systems. This is a fo rm of IA, b ut ma y be to o fo cu sse d o n systems that are ora cu la r. As much a s th e p ro gra m givi ng th e u se r information, there mu st b e th e id ea o f th e u se r givi ng th e program guidanc e.

o Use local area n ets to ma ke h uma n te ams th at re ally work (ie , are more effective th an th eir co mp onent me mbers). T his is generally the area o f " g ro upw are ", a lre ady a ve ry popula r commercial pursu it . T he ch ange in vi ew poin t h ere w ould b e to regard the group a ct ivi ty as a co mb in atio n o rg anism. In o ne sense, this sugge stio n mi ght b e re gard ed a s th e g oal o f in ve ntin g a "Rules of Order" fo r su ch co mb in atio n o pera tio ns. F or in st ance , group focus mig ht b e mo re e asi ly ma in ta in ed th an in cl assi cal meetings. Expert ise o f in divi dual h uma n me mbers could b e iso la te d from ego issues such th at th e co ntri b utio n o f d if fe re nt m em bers is focussed on th e te am pro je ct . An d o f co urse sh are d d ata b ase s could be used mu ch mo re co nve nie ntly th an in co nve ntio nal committee opera tio ns. (N ote th at th is su ggest io n is aime d a t te am operations rathe r th an p olit ica l me etin gs. In a p olit ica l setting, the automa tio n d escri bed a bove w ould si mp ly enfo rce th e power of the perso ns ma king th e ru le s! ) o Exploit the worl dw id e In te rn et a s a co mb in atio n h uma n/m ach in e tool. Of all the items on th e list , p ro gre ss in th is is proceeding the fa stest a nd ma y ru n u s in to th e Si ngula rit y befo re anything else. Th e p ow er and in flu ence o f e ve n th e p re se nt- d ay Internet is vastly u ndere stima te d. F or in st ance , I th in k our contemporary co mpute r syst ems would b re ak under th e w eig ht o f their own comple xity if it w ere n't fo r th e e dge th at th e U SEN ET "group mind" giv es th e syst em admi nist ra tio n a nd su pport p eople !) The very anarchy of th e w orld w id e n et d eve lo pme nt is evi dence o f its potential. As connect ivi ty and b andw id th a nd a rch ive si ze a nd computer speed all in cre ase , w e a re se ein g so me th in g like L yn n Margulis' [14] vis io n o f th e b io sp here a s data p ro ce sso r recapitulated, bu t a t a mi llio n time s gre ate r sp eed a nd w it h millions of huma nly in te llig ent a gents (o urse lve s).

The above exam ple s illu st ra te re se arch th at ca n b e d one w it h in the context of contemp ora ry comp ute r sci ence d epart me nts. T here a re other paradigms. For exa mp le , mu ch o f th e w ork in Art if ici al Intelligence and neura l n ets would b enefit fro m a cl ose r co nnect io n with biological life. I nst ead o f si mp ly tryi ng to mo del a nd u nderst and biological life with c omp ute rs, re se arch co uld b e d ire cted to w ard th e creation of composi te syst ems th at re ly on b io lo gica l lif e fo r guidance or for the p rovidin g fe atu re s we d on't u nderst and w ell e nough 11 yet to implement in h ard ware . A lo ng-t ime d re am of sci ence -fict io n h as been direct brain to comp ute r in te rf ace s [2 ] [2 8]. In fa ct , th ere is concrete work that can b e d one (a nd h as been d one) in th is are a:

o Limb prosthetic s is a to pic of d ire ct co mme rcial a pplica bilit y.

Nerve to silicon tra nsd uce rs can b e ma de [1 3]. T his is an exciting, near-te rm step to w ard d ire ct co mmu nca tio n.

o Similar direct lin ks in to b ra in s ma y be fe asi ble , if th e b it rate is low: given h uma n le arn in g fle xi bilit y, th e a ct ual brain neuron tar gets mi ght n ot h ave to b e p re cise ly se le ct ed.

Even 100 bits pe r se cond w ould b e o f g re at u se to st ro ke victims who wou ld o th erw ise b e co nfin ed to me nu-d rive n interfaces. o Plugging in to the o ptic tru nk has th e p ote ntia l fo r bandw id th s of 1 Mbit/second o r so . Bu t fo r th is, w e n eed to kn ow th e fine-scale archite cture o f vi sion, a nd w e n eed to p la ce a n enormous web o f e le ct ro des wit h e xq uisi te p re cision. If w e w ant our high bandwid th co nnect io n to b e _ in a ddit io n_ to w hat p ath s are already pres ent in th e b ra in , th e p ro ble m beco me s va stly m ore intractable. Just st icki ng a g rid o f h ig h-b andw id th re ce ive rs into a brain certa in ly won't d o it . Bu t su ppose th at th e high-bandwidth g rid w ere p re se nt w hile th e b ra in st ru cture w as actually setting u p, a s th e e mb ryo d eve lo ps. T hat su ggest s:

o Animal embryo exp erime nts. I w ould n't e xp ect a ny IA su cce ss in the first years o f su ch re se arch , b ut g ivi ng d eve lo pin g b ra in s access to compl ex simu la te d n eura l st ru cture s mi ght b e ve ry interesting to the people w ho st udy how th e e mb ryonic bra in develops. In the lo ng ru n, su ch e xp erime nts mi ght p ro duce animals with addi tio nal se nse p ath s and in te re stin g in te lle ct ual abilities. Originally, I had h oped th at th is discu ssion o f IA would yi eld some clearly safer ap pro ach es to th e Si ngula rity. (Af te r all, IA allows our participa tio n in a ki nd o f tra nsce ndance .) Al as, lo oki ng back over these IA p roposa ls, a bout a ll I a m sure o f is th at th ey should be considered , th at th ey ma y give u s mo re o ptio ns. B ut a s fo r safety ... well, some o f th e su ggest io ns are a lit tle sca rey on th eir face. One of my informa l re view ers poin te d o ut th at IA fo r in divi dual humans creates a r ath er si nist er elit e . W e h uma ns have mi llio ns of years of evolutionary baggage th at ma kes us re gard co mp etit io n in a deadly light. Much of th at d eadlin ess ma y not b e n ece ssa ry in to day' s world, one where lose rs ta ke o n th e w in ners' tri cks and a re co opte d into the winners' en te rp rise s. A cre atu re th at w as built _ de n ovo _ might possibly be a much mo re b enig n e ntit y th an o ne w it h a ke rn el 12 based on fang and t alo n. An d e ve n th e e galit a ria n vi ew o f a n In te rn et that wakes up along wit h a ll ma nki nd ca n b e vi ew ed a s a n ig htm are [25]. The problem is n ot th at th e Si ngula rity re pre se nts simp ly th e passing of humanki nd fro m cente r st ange, b ut th at it co ntra dict s so me of our most deeply h eld n otio ns of b ein g. I th in k a cl ose r lo ok at th e notion of strong supe rhuma nit y ca n sh ow w hy th at is.

_Strong Superhuma nit y and th e Be st W e C an Ask fo r_ Suppose we cou ld ta ilo r th e Si ngula rity. Su ppose w e co uld a tta in our most extravaga nt h opes. W hat th en w ould w e a sk fo r:

That humans themse lve s would b eco me th eir ow n su cce ssors, th at whatever injustice o ccurs would b e te mp ere d b y our kn ow le dge o f o ur roots. For those who re ma in ed u nalt e re d, th e g oal w ould b e b enig n treatment (perhaps eve n g ivi ng th e st ay-b ehin ds th e a ppeara nce o f being masters of god like sl ave s). It co uld b e a g old en a ge th at a lso involved progress ( ove rle apin g St ent's barri er). Immo rtalit y (o r a t least a lifetime as lon g a s we ca n ma ke th e u nive rse su rvive [9 ] [3]) would be achieva ble .

But in this bright est a nd ki ndest w orld , th e p hilo so phica l problems themselve s beco me in timi datin g. A mi nd th at st ays at th e sa me capacity cannot live fo re ve r; a fte r a fe w th ousa nd ye ars it w ould lo ok more like a repeatin g ta pe lo op th an a p erso n. (T he mo st ch illin g picture I have seen of th is is in [1 7].) T o live in defin it e ly lo ng, the mind itself must g ro w ... a nd w hen it b eco me s gre at e nough, a nd looks back ... what fe llo w -feelin g ca n it h ave w it h th e so ul th at it was originally? Cer ta in ly th e la te r bein g w ould b e e ve ryt hin g th e original was, but so much va stly mo re. An d so e ve n fo r th e in divi dual, the Cairns-Smith (or Lyn n Ma rgulis) notio n o f n ew lif e g ro win g incrementally out of th e o ld mu st st ill b e va lid .

This "problem" a bout immo rtalit y co me s up in mu ch mo re d ir e ct ways. The notion of e go a nd se lf -a ware ness has been th e b edro ck of the hardheaded rat io nalism of th e la st fe w ce ntu rie s. Ye t n ow th e notion of self-awaren ess is under atta ck fro m th e Art if ici al Intelligence people (" self -a ware ness and o th er delu si ons" ).

Intelligence Amplifica tio n u ndercu ts th e imp ort ance o f e go fr o m another direction. T he p ost -Si ngula rity world w ill in vo lve e xt re m ely high-bandwidth netw orki ng. A ce ntra l fe atu re o f st ro ngly su perh um an 13 entities will likely be th eir abilit y to co mmu nica te a t va ria ble bandwidths, includin g o nes fa r hig her th an sp eech o r w ritte n m essa ges.

What happens when p ie ce s of e go ca n b e co pie d a nd me rged, w hen th e size of a selfawaren ess can g ro w o r sh rin k to fit th e n atu re o f th e problems under consi dera tio n? T hese a re e sse ntia l fe atu re s of st ro ng superhumanity and th e Si ngula rity. T hin ki ng a bout th em, o ne b egin s to feel how essentially stra nge a nd d if fe re nt th e Po st-H uma n e ra w ill b e -- _no matter how c le ve rly and b enig nly it is bro ught to b e_.

From one angle, th e vi sion fit s ma ny of o ur happie st d re am s:

a place unending, w here w e ca n tru ly kn ow o ne a noth er and u nderst and the deepest mysteri es. F ro m anoth er angle , it 's a lo t like th e w orst case scenario I imag in ed e arlie r in th is paper.

Which is the valid vi ew poin t? In fa ct , I th in k th e n ew e ra is simply too different t o fit in to th e cl assi cal fra me o f g ood a nd evil. That frame is ba sed o n th e id ea o f iso la te d, immu ta ble m in ds connected by tenuou s, lo w -b andw it h lin ks. Bu t th e p ost -Si ngula rit y world _does_ fit wit h th e la rg er tra dit io n o f ch ange a nd co opera tio n that started long ag o (p erh aps eve n b efo re th e ri se o f b io lo gica l life). I think there _a re_ n otio ns of e th ics th at w ould a pply in su ch an era. Research int o IA and h ig h-b andw id th co mmu nica tio ns sh ould improve this underst andin g. I se e ju st th e g limme rin gs of th is now , in Good's Meta-Gold en R ule , p erh aps in ru le s fo r dist in guish in g se lf from others on the b asi s of b andw id th o f co nnect io n. An d w hile m in d and self will be vast ly mo re la bile th an in th e p ast , mu ch o f w hat w e value (knowledge, me mory, th ought) need n eve r be lo st . I th in k Freeman Dyson ha s it ri g ht w hen h e sa ys [8 ]: " G od is what m in d b eco mes when it has passed beyo nd th e sca le o f o ur co mp rehensi on." [I wish to thank John C arro ll o f Sa n D ie go St ate U nive rsity and H ow ard Davidson of Sun Mi crosyst ems fo r discu ssing th e d ra ft ve rsio n o f th is paper with me.] _Annotated Sources [a nd a n o cca sional p le a fo r bib lio gra phica l h elp ]_ 14 [1] Alfvén, Hannes, wr it in g a s Olo f Jo hanneso n, _ T he En d o f M an?_, Award Books, 196 9 e arlie r publish ed a s "T he T ale o f th e B ig Computer", Cowa rd-McC ann, tra nsl ate d fro m a b ook co pyr ig ht 1 966 Albert Bonniers Fo rla g AB wit h En glish tra nsl atio n co pyri ght 1 966 by Victor Gollanz, L td .

[2] Anderson, Poul, "Ki ngs W ho D ie ", _ If_ , Ma rch 1 962, p 8-3 6.

Reprinted in _Seve n C onquest s_ , Po ul An derso n, Ma cMilla n C o., 1 969.

[3] Barrow, John D. an d F ra nk J. T ip le r, _ T he An th ro pic Cosm olo gica l Principle_, Oxford U nive rsity Pre ss, 1 986.

[4] Bear, Greg, "Blood Mu sic" , _ An alo g Sci ence F ict io n-Sci ence F act _, June, 1983. Expa nded in to th e n ove l _ Bl ood Mu sic_ , Mo rro w , 1 985 [5] Cairns-Smith, A. G ., _ Se ven C lu es to th e O rig in o f L if e _, C am brid ge University Press, 1 985.

[6] Conrad, Michael _et a l. _ , " T ow ard s an Art if ici al Bra in ", _BioSystems_, vo l2 3, p p175-2 18, 1 989.

[7] Drexler, K. Eric, _E ngin es of C re atio n_, An chor Pre ss/Double day, 1 986.

[8] Dyson, Freeman, _I nfin it e in Al l D ire ctio ns_ , H arp er && Row , 1 988.

[9] Dyson, Freeman, "Ph ysics and Bi olo gy in a n O pen U nive rse", _ R evi ew of Modern Physics_ , vo l 5 1, p p447-4 60, 1 979.

[10] Good, I. J., "Specu la tio ns Conce rnin g th e F irst U lt ra in te llig ent Machine", in _Adv ance s in C omp ute rs_ , vo l 6 , F ra nz L. A lt a nd Morris Rubinoff, ed s, p p31-8 8, 1 965, Aca demi c Pre ss.

[11] Good, I. J., [Help! I ca n't fin d th e so urce o f G ood's Me ta -G old en Rule, though I ha ve th e cl ear re co lle ct io n o f h earin g a bout it sometime in the 19 60s. T hro ugh th e h elp o f th e n et, I h ave fo und pointers to a numb er of re la te d it e ms. G . H arry Stin e a nd A ndre w Haley have written a bout me ta la w a s it mi ght re la te to extraterrestrials: G . H arry Stin e, " H ow to G et a lo ng w it h Extraterrestrials ... o r Yo ur N eig hbor" , _ An alo g Sci ence F act - Science Fiction_, F ebru ary, 1 980, p 39-4 7.] [12] Herbert, Frank, _D une_, Be rkley Bo oks, 1 985. H ow eve r, th is nove l w as serialized in _Anal og Sci ence F ict io n-Sci ence F act _ in th e 1 960s.

15 [13] Kovacs, G. T. A. _e t a l. _ , " R egenera tio n Mi cro ele ct ro de A rra y fo r Peripheral Nerve R eco rdin g a nd St imu la tio n", _ IEEE Tra nsa ctio ns on Biomedical Eng in eerin g_, v 39, n 9 , p p 8 93-9 02.

[14] Margulis, Lynn and D orio n Sa gan, _ Mi cro cosmo s, F our B illio n Y ears of Evolution from Ou r Mi cro bia l An cestors_ , Su mmi t Bo oks, 1 986.

[15] Minsky, Marvin, _S oci ety of Mi nd_, Si mo n a nd Sch ust er, 1 985.

[16] Moravec, Hans, _Mi nd C hild re n_, H arva rd U nive rsity Pre ss, 1 988.

[17] Niven, Larry, "The Et hics of Ma dness" , _ If_ , Ap ril 1 967, p p82-1 08.

Reprinted in _Neut ron St ar_ , L arry Nive n, Ba lla ntin e Bo oks, 1 968.

[18] Penrose, R., _The Emp ero r's New Mi nd_, O xford U nive rsit y Pre ss, 1 989.

[19] Platt, Charles, Priv ate C ommu nica tio n.

[20] Rasmussen, S. _et a l. _ , " C omp uta tio nal C onnect io nism wit h in N euro ns:

a Model of Cytosk ele ta l Au to ma ta Su bse rving N eura l N etw orks" , in _Emergent Compu ta tio n_, St ephanie F orre st, e d., p 428-4 49, M IT Press, 1991. [21] Searle, John R., "Mi nds, Bra in s, a nd Pro gra ms" , in _ T he B ehavi ora l a nd Brain Sciences_, v.3, C amb rid ge U nive rsity Pre ss, 1 980. T he essay is reprinted in _ T he Mi nd's I_ , e dit e d b y Dougla s R.

Hofstadter and Da nie l C . D ennett, Ba sic Bo oks, 1 981. T his reprinting contain s an e xce lle nt cri tiq ue o f th e Se arle e ssa y.

[22] Sims, Karl, "Interact ive Evo lu tio n o f D yn ami ca l Syst ems" , T hin ki ng Machines Corpora tio n, T ech nica l R eport Se rie s (p ublish ed in _ T ow ard a Practice of Auto nomo us Syst ems: Pro ceedin gs of th e F ir st E uro pean Cnference on Arti fici al L if e _, Pa ris, MI T Pre ss, D ece mb er 1 991.

[23] Stapledon, Olaf, _T he St arma ker_, Be rkley Bo oks, 1 961 ( b ut fr o m the forward probab ly writte n b efo re 1 937).

[24] Stent, Gunther S., _T he C omi ng o f th e G old en Ag e: A View o f th e E nd of Progress_, The Natu ra l H ist ory Pre ss, 1 969.

[25] Swanwick Michae l, _ Va cuum Flo w ers_ , se ria lize d in _ Isa ac Asi m ov' s Science Fiction Ma gazi ne_, D ece mb er(? ) 1986 - Febru ary 1987.

16 Republished by A ce Bo oks, 1 988.

[26] Thearling, Kurt, "H ow W e W ill Bu ild a Ma chin e th at T hin ks" , a w orksh op at Thinking Mach in es Corp ora tio n. Pe rso nal C ommu nica tio n.

[27] Ulam, S., Tribute t o Jo hn vo n N euma nn, _ Bu lle tin o f th e A m erica n Mathematical Soc ie ty_ , vo l 6 4, n r 3, p art 2 , Ma y, 1 958, p 1-4 9.

[28] Vinge, Vernor, "Boo kworm, R un!" , _ An alo g_, Ma rch 1 966, p p8-4 0.

Reprinted in _Tru e N ame s and O th er D angers_ , Ve rnor V in ge, B aen Books, 1987. [29] Vinge, Vernor, "Tru e N ame s", _ Bi nary Star N umb er 5_, D ell, 1 981.

Reprinted in _Tru e N ame s and O th er D angers_ , Ve rnor V in ge, B aen Books, 1987. [30] Vinge, Vernor, First W ord , _ O mn i_ , Ja nuary 1983, p 10.