computer ethic
1 Ethical Issu es in A dvan ced A rti ficia l In te llig en ce Nick Bostro m Oxford U nive rsity Philo so phy Facu lt y 10 Me rton St reet Oxford O X1 4 JJ Unit e d Ki ngdom Home page:
http ://w ww.n ickb ost ro m. com [This is a slightly revise d ve rsion o f a p aper publish ed in Cognit iv e, E m otiv e a n d E th ic al Aspects of Decision Ma kin g in H um an s a n d in A rti ficia l In te llig en ce , V ol. 2 , e d. I. S m it e t al., Int. Institute of Advan ced St udie s in Syst ems Rese arch a nd C yb ern etics, 2 003, p p. 1 2-1 7] ABST RACT The ethical issues related to th e p ossi ble fu tu re cre atio n o f ma chin es wit h g enera l in te lle ct ual capabilities far outstrippin g th ose o f h uma ns are q uit e d ist in ct fro m a ny eth ica l p ro ble m s arisi ng in current automation and in fo rma tio n syst ems. Su ch su perin te llig ence w ould n ot b e ju st another technological dev elo pme nt; it w ould b e th e mo st imp ort ant in ve ntio n e ve r m ade, a nd would lead to explosive pro gre ss in a ll sci entif ic and te ch nolo gica l fie ld s, a s th e superintelligence would c onduct re se arch w it h su perh uma n e ffici ency. T o th e e xt ent th at e th ics is a cognitive pursuit, a s uperin te llig ence co uld a lso e asi ly su rpass hum ans in th e q ualit y of it s moral thinking. However, it w ould b e u p to th e d esi gners of th e su perin te llig ence to sp eci fy it s original motivations. Since th e su perin te llig ence ma y beco me u nst oppably pow erfu l b eca use o f its intellectual superiority and th e te ch nolo gie s it co uld d eve lo p, it is cr uci al th at it b e p ro vi ded with human-friendly motiv atio ns. T his paper su rve ys some o f th e u niq ue e th ica l issu es in creating superintelligence , a nd d iscu sses what mo tiva tio ns we o ught to g ive a su perin te llig ence , and introduces some cost -benefit co nsi dera tio ns re la tin g to w heth er th e d eve lo pm ent o f superintelligent machines ought to b e a cce le ra te d o r re ta rd ed.
KEYWORDS: Artificial int ellig ence , e th ics, u plo adin g, su perin te llig ence , g lo bal se curit y, cost-benefit analysis 1. INTRODUCTION A superintelligence is any in te lle ct th at is va stly outp erf orms th e b est h um an b ra in s in p ra ct ica lly every field, including scie ntif ic cre ativi ty, g enera l w isd om, a nd so cial ski lls.
[1 ] T his defin it io n leaves open how the supe rin te llig ence is imp le me nte d – it co uld b e in a d ig it a l co mpute r, a n ensemble of networked co mpute rs, cu lt u re d co rtica l tissu e, o r so meth in g e lse .
On this definition, Deep Bl ue is not a su perin te llig ence , si nce it is only sm art w it h in o ne n arro w domain (chess), and even th ere it is not va stly su perio r to th e b est h um ans. E ntit ie s su ch a s corporations or the scient if ic co mmu nit y are n ot su perin te llig ence s eit h er. A lt h ough th ey ca n perform a number of inte lle ct ual fe ats of w hich n o in divi dual h um an is ca pable , th ey are n ot 2 sufficiently integrated to c ount a s “in te lle ct s”, a nd th ere a re ma ny fie ld s in w hich th ey perfo rm much worse than single h uma ns. F or exa mp le , yo u ca nnot h ave a r e al- tim e co nve rsa tio n w it h “the scientific community ”.
While the possibility of doma in -sp eci fic “su perin te llig ence s” is also w orth e xp lo rin g, th is paper focuses on issues arising fro m th e p ro sp ect o f g enera l su perin te llig ence . S pace co nst ra in ts prevent us from attemptin g a nyt hin g co mp rehensi ve o r deta ile d. A ca rto onish ske tch o f a fe w selected ideas is the mos t w e ca n a im fo r in th e fo llo w in g fe w p ages.
Several authors have arg ued th at th ere is a su bst antia l ch ance th at su perin te llig ence m ay be created within a few decad es, p erh aps as a re su lt o f g ro win g h ard w are p erfo rm ance a nd increased ability to imple ment a lg orith ms and a rch it e ct ure s simi la r to th ose u se d b y hum an brains. [2] It might turn out to ta ke mu ch lo nger, b ut th ere se ems curre ntly to b e n o g ood g ro und for assigning a negligible p ro babilit y to th e h yp oth esi s th at su perin te llig ence w ill b e cr eate d within the lifespan of som e p eople a live to day. G ive n th e e normi ty of th e co nse quence s of superintelligence, it would make se nse to g ive th is pro sp ect so me se rio us co nsi dera tio n e ve n if one thought that there were o nly a sma ll p ro babilit y of it h appenin g a ny tim e so on.
2. SUPERINTELLIGENCE IS DIF FER ENT A prerequisite for having a me anin gfu l d iscu ssion o f su perin te llig ence is th e r e aliza tio n th at superintelligence is not ju st a noth er te ch nolo gy, a noth er to ol th at w ill a dd in cr em enta lly to human capabilities. Superi nte llig ence is ra dica lly dif fe re nt. T his poin t b ears em phasi zing, fo r anthropomorphizing superi nte llig ence is a mo st fe cu nd so urce o f m isco nce ptio ns.
Let us consider some of t he u nusu al a sp ect s of th e cre atio n o f su perin te llig ence :
· Superintelligence may b e th e la st in ve ntio n h uma ns eve r need to m ake . Given a superintelligence ’s in te lle ct ual su perio rity, it w ould b e mu ch b ette r a t d oin g sci entif ic research and technologica l d eve lo pme nt th an a ny huma n, a nd p ossi bly bette r e ve n th an a ll humans taken together. On e imme dia te co nse quence o f th is fa ct is th at:
· Technological progres s in a ll o th er fie ld s will b e a cce le ra te d b y th e a rriva l o f a dva nce d artificial intelligence. It is likely that any techno lo gy th at w e ca n cu rre ntly fo re se e w ill b e sp eedily deve lo ped b y th e first superintelligence, no doubt a lo ng w it h ma ny oth er te ch nolo gie s of w hich w e a re a s ye t clueless. The foreseeable te ch nolo gie s th at a su perin te llig ence is like ly to d eve lo p in cl ude mature molecular manufa cturin g, w hose a pplica tio ns are w id e-ra ngin g:
[3 ] a) very powerful comput ers b) advanced weaponry, p robably ca pable o f sa fe ly disa rmi ng a n ucl ear p ow er c) space travel and von N euma nn p ro bes (se lf -re pro duci ng in te rst ella r p ro bes) d) elimination of aging an d d ise ase e) fine-grained control o f h uma n mo od, e mo tio n, a nd mo tiva tio n f) uploading (neural or s ub-n eura l sca nnin g o f a p art icu la r bra in a nd im ple m enta tio n o f th e same algorithmic structure s on a co mp ute r in a w ay th at p erse vere s m em ory and p erso nalit y) g) reanimation of cryoni cs patie nts h) fully realistic virtual rea lit y · Superintelligence will le ad to mo re a dva nce d su perin te llig ence .
This results both from the imp rove d h ard ware th at a su perin te llig ence co uld cr eate , a nd a lso from improvements it coul d ma ke to it s ow n so urce co de.
3 · Artificial minds can be easi ly co pie d.
Since artificial intelligences are so ftw are , th ey ca n e asi ly and q uickl y be co pie d, so lo ng a s th ere is hardware available to s to re th em. T he sa me h old s fo r huma n u plo ads. H ard w are a si de, th e marginal cost of creating a n a ddit io nal co py of a n u plo ad o r an a rtif ici al in te llig ence a fte r th e fir st one has been built is nea r ze ro. Art if ici al mi nds co uld th ere fo re q uickl y co me to e xi st in g re at numbers, although it is possi ble th at e ffici ency would fa vo r co nce ntr a tin g co mputa tio nal resources in a single supe r-inte lle ct .
· Emergence of superin te llig ence ma y be su dden.
It appears much harder to g et fro m where w e a re n ow to h uma n-le ve l a rtif ici al in te llig ence th an to get from there to super in te llig ence . W hile it ma y th us ta ke q uit e a w hile b efo re w e g et superintelligence, the final st age ma y happen sw if tly. T hat is, th e tr a nsi tio n fr o m a st ate w here we have a roughly human -le ve l a rt if ici al in te llig ence to a st ate w here w e h ave fu ll- b lo w n superintelligence, with revo lu tio nary applica tio ns, ma y be ve ry rapid , p erh aps a m atte r o f d ays rather than years. This possi bilit y of a su dden e me rgence o f su perin te llig ence is re fe rre d to a s the singularity hypothesis . [4 ] · Artificial intellects are pote ntia lly auto nomo us agents . A superintelligence shoul d n ot n ece ssa rily be co nce ptu alize d a s a m ere to ol. W hile sp eci alize d superintelligences that ca n th in k only about a re strict ed se t o f p ro ble m s m ay be fe asi ble , general superintelligence w ould b e ca pable o f in dependent in it ia tive a nd o f m aki ng it s ow n plans, and may therefore b e mo re a ppro pria te ly th ought o f a s an a uto nom ous agent.
· Artificial intellects nee d n ot h ave h uma nlike mo tive s.
Human are rarely willing sl ave s, b ut th ere is noth in g imp la usi ble a bout th e id ea o f a superintelligence having as it s su perg oal to se rve h uma nit y or so me p articu la r h um an, w it h n o desire whatsoever to revol t o r to “l ib era te ” it se lf . It a lso se ems perfe ct ly possi ble to h ave a superintelligence whose so le g oal is so me th in g co mp le te ly arb it r a ry, su ch a s to m anufa ct ure a s many paperclips as poss ib le , a nd w ho w ould re sist w it h a ll it s mi ght a ny atte m pt to a lt e r th is goal. For better or worse, a rt if ici al in te lle ct s need n ot sh are o ur hum an m otiva tio nal te ndenci es.
· Artificial intellects may n ot h ave h uma nlike p sych es.
The cognitive architecture of a n a rt if ici al in te lle ct ma y also b e q uit e u nlike th at o f h um ans.
Artificial intellects may find it e asy to g uard a gain st so me ki nds of h um an e rro r a nd b ia s, w hile a t the same time being at incre ase d ri sk of o th er ki nds of mi stake th at n ot e ve n th e m ost h aple ss human would make. Sub je ct ive ly, th e in ner co nsci ous lif e o f a n a rtif ici al in te lle ct , if it h as one, may also be quite different fro m ours.
For all of these reasons, one sh ould b e w ary of a ssu ming th at th e e m erg ence o f superintelligence can be pre dict ed b y ext ra pola tin g th e h ist ory of o th er te ch nolo gica l breakthroughs, or that the n atu re a nd b ehavi ors of a rt if ici al in te lle ct s would n ece ssa rily resemble those of human or oth er anima l mi nds.
3. SUPERINTELLIGENT MORAL T H IN KIN G To the extent that ethics is a co gnit ive p ursu it , a su perin te llig ence co uld d o it b ette r th an h um an thinkers. This means that q uest io ns about e th ics, in so fa r as th ey have co rre ct a nsw ers th at can be arrived at by reaso nin g a nd w eig htin g u p o f e vi dence , co uld b e m ore a ccu ra te ly answered by a superintel lig ence th an b y huma ns. T he sa me h old s fo r q uest io ns of p olicy and long-term planning; when it co me s to u nderst andin g w hich p olici es would le ad to w hich r e su lt s, 4 and which means would be mo st e ffe ct ive in a tta in in g g ive n a ims , a su perin te llig ence w ould outperform humans. There are therefore many quest io ns th at w e w ould n ot n eed to a nsw er o urse lve s if w e h ad o r were about to get superint ellig ence ; w e co uld d ele gate ma ny in ve stig atio ns and d eci sions to th e superintelligence. For exa mple , if w e a re u nce rtain h ow to e va lu ate p ossi ble o utco mes, w e could ask the superintelli gence to e st ima te h ow w e w ould h ave e va lu ate d th ese o utco mes if w e had thought about them f or a ve ry lo ng time , d elib era te d ca refu lly, h ad h ad m ore m em ory and better intelligence, and so fo rt h. W hen fo rmu la tin g a g oal fo r th e su perin te llig ence , it w ould n ot always be necessary to g ive a d eta ile d, e xp lici t d efin it io n o f th is goal. W e co uld e nlist th e superintelligence to help us dete rmi ne th e re al in te ntio n o f o ur re quest , th us decr easi ng th e r isk that infelicitous wording or confu si on a bout w hat w e w ant to a ch ie ve w ould le ad to o utco mes that we would disapprove o f in re tro sp ect .
4. IMPORTANCE OF INI TIA L MO TIVA TIO NS The option to defer many d eci sions to th e su perin te llig ence d oes not m ean th at w e ca n a ffo rd to be complacent in how we const ru ct th e su perin te llig ence . O n th e co ntr a ry, th e se ttin g u p o f initial conditions, and in pa rticu la r th e se le ct io n o f a to p-le ve l g oal fo r th e su perin te llig ence , is of the utmost importance. O ur entire fu tu re ma y hin ge o n h ow w e so lve th ese p ro ble m s.
Both because of its supe rio r pla nnin g a bilit y and b eca use o f th e te ch nolo gie s it co uld d eve lo p, it is plausible to suppose tha t th e first su perin te llig ence w ould b e ve ry pow erfu l. Q uit e p ossi bly, it would be unrivalled: it wou ld b e a ble to b rin g a bout a lmo st a ny possi ble o utco me a nd to th w art any attempt to prevent th e imp le me nta tio n o f it s to p g oal. It co uld ki ll o ff a ll o th er a gents, persuade them to change th eir behavi or, o r blo ck th eir atte mp ts at in te rfe re nce . E ve n a “ fe tte re d superintelligence” that was runnin g o n a n iso la te d co mp ute r, a ble to in te ra ct w it h th e r e st o f th e world only via text interface , mi ght b e a ble to b re ak out o f it s co nfin em ent b y persu adin g it s handlers to release it. The re is eve n so me p re limi nary exp erime nta l e vi dence th at th is would b e the case. [5] It seems that the best wa y to e nsu re th at a su perin te llig ence w ill h ave a b enefici al im pact o n th e world is to endow it with p hila nth ro pic va lu es. Its to p g oal sh ould b e fr ie ndlin ess.
[6 ] How e xa ctly friendliness should be un derst ood a nd h ow it sh ould b e imp le me nte d, a nd h ow th e a m it y sh ould be apportioned between d if fe re nt p eople a nd n onhuma n cre atu re s is a m atte r th at m erit s fu rth er consideration. I would argu e th at a t le ast a ll h uma ns, a nd p ro bably m any oth er se ntie nt creatures on earth should g et a si gnif ica nt sh are in th e su perin te llig ence ’s benefice nce . If th e benefits that the superint ellig ence co uld b est ow a re e normo usl y va st, th en it m ay be le ss important to haggle over th e d eta ile d d ist rib utio n p atte rn a nd mo re im porta nt to se ek to e nsu re that everybody gets at lea st so me si gnif ica nt sh are , si nce o n th is su pposi tio n, e ve n a tin y sh are would be enough to guara nte e a ve ry lo ng a nd ve ry good lif e . O ne r isk th at m ust b e g uard ed against is that those who deve lo p th e su perin te llig ence w ould n ot m ake it g enerica lly philanthropic but would inst ead g ive it th e mo re limi te d g oal o f se rvi ng o nly so me sm all g ro up, such as its own creators or th ose w ho co mmi ssioned it .
If a superintelligence start s out w it h a fri e ndly to p g oal, h ow eve r, th en it ca n b e r e lie d o n to st ay friendly, or at least not to delib era te ly rid it se lf o f it s fri e ndlin ess. T his poin t is ele m enta ry. A “friend” who seeks to tran sform himse lf in to so me body who w ants to h urt yo u, is not yo ur fr ie nd.
A true friend, one who re ally ca res about yo u, a lso se eks th e co ntin uatio n o f h is ca rin g fo r yo u.
5 Or to put it in a different w ay, if yo ur to p g oal is X, a nd if yo u th in k th at b y ch angin g yo urse lf in to someone who instead wa nts Y yo u w ould ma ke it le ss like ly th at X w ill b e a ch ie ve d, th en yo u will not rationally transfor m yourse lf in to so me one w ho w ants Y . T he se t o f o ptio ns at e ach p oin t in time is evaluated on th e b asi s of th eir co nse quence s fo r re aliza tio n o f th e g oals held a t th at time, and generally it will be irra tio nal to d elib era te ly ch ange o ne’s ow n to p g oal, si nce th at would make it less likely t hat th e cu rre nt g oals will b e a tta in ed.
In humans, with our comp lica te d e vo lve d me nta l e co lo gy of st ate -d ependent co mpetin g d rive s, desires, plans, and ideals , th ere is ofte n n o o bvi ous way to id entif y what o ur to p g oal is; w e might not even have one. So fo r us, th e a bove re aso nin g n eed n ot a pply. B ut a superintelligence may be st ru cture d d if fe re ntly. If a su perin te llig ence h as a d efin it e , d ecl ara tive goal-structure with a clearl y id entif ie d to p g oal, th en th e a bove a rg um ent a pplie s. A nd th is is a good reason for us to bui ld th e su perin te llig ence w it h su ch a n e xp lici t m otiva tio nal a rch it e ct ure .
5. SHOULD DEVELOPMEN T B E DEL AYED O R A CCEL ER ATE D ?
It is hard to think of any pro ble m th at a su perin te llig ence co uld n ot e it h er so lve o r a t le ast h elp us solve. Disease, poverty, e nvi ro nme nta l d est ru ctio n, u nnece ssa ry su ffe rin g o f a ll ki nds: th ese are things that a superinte llig ence e quip ped w it h a dva nce d n anote ch nolo gy would b e ca pable o f eliminating. Additionally, a su perin te llig ence co uld g ive u s in defin it e lif e sp an, e it h er b y stoppin g and reversing the aging p roce ss th ro ugh th e u se o f n anome dici ne [7 ] , o r b y offe rin g u s th e option to upload ourselves. A su perin te llig ence co uld a lso cre ate o pportu nit ie s fo r u s to va stly increase our own intellect ual a nd e mo tio nal ca pabilit ie s, a nd it co uld a ssi st u s in cr eatin g a highly appealing experient ia l w orld in w hich w e co uld live live s devo te d to in jo yf ul game-playing, relating to e ach o th er, e xp erie nci ng, p erso nal g ro wth , a nd to livi ng cl ose r to o ur ideals. The risks in developing s uperin te llig ence in cl ude th e ri sk of fa ilu re to g ive it th e su perg oal o f philanthropy. One way in w hich th is co uld h appen is th at th e cre ato rs of th e su perin te llig ence decide to build it so that it se rve s only th is se le ct g ro up o f h uma ns, r a th er th an h um anit y in general. Another way for i t to h appen is th at a w ell-me anin g te am o f p ro gra m mers m ake a b ig mistake in designing its g oal syst em. T his co uld re su lt , to re tu rn to th e e arlie r e xa mple , in a superintelligence whose t op g oal is th e ma nufa ct urin g o f p apercl ip s, w it h th e co nse quence th at it starts transforming first a ll o f e art h a nd th en in cre asi ng p ort io ns of sp ace in to p apercl ip manufacturing facilities. M ore su btly, it co uld re su lt in a su perin te llig ence r e alizi ng a st ate o f affairs that we might now ju dge a s desi ra ble b ut w hich in fa ct tu rn s out to b e a fa lse u to pia , in which things essential to h uma n flo urish in g h ave b een irre versibly lo st . W e n eed to b e ca re fu l about what we wish for fro m a su perin te llig ence , b eca use w e mi ght g et it .
One consideration that sh ould b e ta ke n in to a cco unt w hen d eci din g w heth er to p ro m ote th e development of superintel lig ence is th at if su perin te llig ence is fe asi ble , it w ill like ly be d eve lo ped sooner or later. Therefore, w e w ill p ro bably one d ay have to ta ke th e g am ble o f superintelligence no matt er w hat. Bu t o nce in e xi st ence , a su perin te llig ence co uld h elp u s reduce or eliminate other exi st entia l ri sks [8 ] , su ch a s th e ri sk th at a dva nce d n anote ch nolo gy will be used by humans in wa rfare o r te rro rism, a se rio us th re at to th e lo ng-te rm su rvi va l o f intelligent life on earth. If we g et to su perin te llig ence first , w e ma y avo id th is risk fr o m nanotechnology and man y oth ers. If, o n th e o th er hand, w e g et n anote ch nolo gy fir st , w e w ill have to face both the risk s fro m nanote ch nolo gy and, if th ese ri sks are su rvi ve d, a lso th e r isks 6 from superintelligence. Th e o ve rall ri sk seems to b e mi nimi ze d b y im ple m entin g superintelligence, with gre at ca re, a s so on a s possi ble .
REFERENCES Bostrom, N. (1998). "How Long Be fo re Su perin te llig ence ?" In te rn atio nal Jo urn al o f F utu re s Studies , 2. http://www.nickbost ro m. com/ superin te llig ence .h tml Bostrom, N. (2002). "Exist entia l R isks: An alyzi ng H uma n Ext in ct io n S ce nario s and R ela te d Hazards." Journal of Evolu tio n a nd T ech nolo gy , 9 .
http://www.nickbostrom.co m/exi st entia l/ ri sks. html Drexler, K. E. Engines of Cre atio n: T he C omi ng Era o f N anote ch nolo gy . ( A nch or B ooks: N ew York, 1986). http://www.fore sight.o rg /EO C/in dex. html Freitas Jr., R. A. Nanomedici ne, Vo lu me 1 : Ba sic Capabilit ie s . (L andes Bio sci ence :
Georgetown, TX, 1999). http ://w ww.n anome dici ne.co m Hanson, R., et al. (1998). " A Critica l D iscu ssion o f Vi nge's Si ngula rit y Conce pt." Ext ro py Onlin e . http://www.extropy.org/eo /a rt icl es/ vi.h tml Kurzweil, R. The Age of Spiri tu al Ma chin es: W hen C omp ute rs Ex ceed H um an In te llig ence . (Viking: New York, 1999) .
Moravec, H. Robot: Mere Ma chin e to T ra nsce ndent Mi nd . (O xford U nive rsi ty Pre ss: N ew Y ork, 1999). Vinge, V. (1993). "The Co ming T ech nolo gica l Si ngula rity. " Whole E arth R evi ew , W in te r issu e.
Yudkowsky, E. (2002). "Th e AI Bo x Exp erime nt." Webpage . http://sysopmind.com/essa ys/aib ox. html Yudkowsky, E. (2003). Cre atin g F rie ndly AI 1 .0 .
http ://w ww.si ngin st .o rg /C FA I/in dex. htm l [1] (Bostrom, 1998) [2] (Bostrom 1998; Kurzwe il 1 999; Mo rave c 1999) [3] (Drexler 1986) [4] (Vinge 1993; Hanson et a l. 1 998) [5] (Yudkowsky 2002) [6] (Yudkowsky 2003) [7] (Freitas Jr. 1999) [8] (Bostrom 2002)
