This is a Python assignment with 2 questions that use Python data science skills to perform a few analyses on the topics of equity and algorithmic bias.

P ro ble m 1 In t h is pro b le m , yo u w ill c re ate a vi su aliz a tio n o f g en d er r e p re se nta tio n in a rtw ork in t h e Ta te A rt M use um (h ttp s: //g it h ub .c o m /ta te g alle ry/ co lle ctio n) . lo t o f in fo rm atio n t h at I' ve r e m ove d in t h e d ata p re p ara tio n b elo w , in clu d in g t h e n am e o f t h e a rtist , t h eir b ir th a n d d eath d ate s, a n d va rio us deta ils ab out e a ch p ie ce. Y o u m ay wish t o e xp lo re t h e f u ll d ata se ts la te r, b ut f o r n o w , I t h o ug ht yo u'd p re fe r t o b e a b le t o f o cu s on o nly th e c o lu m ns need ed f o r t o d ay.

In [1]:

import pandas as pd artwork = pd .read_csv ('https://raw.githubusercontent.com/rfordatascience/tidytuesda y/master/data/2021/2021-01-12/artwork.csv' ) artists = pd .read_csv ("https://github.com/tategallery/collection/raw/master/artist_ data.csv" ) artwork .to_csv ("artwork.csv" , index = False ) artists .to_csv ("artists.csv" , index = False ) 30/11/2021, 17:14 HW7 localhost:8888/lab 2/12 In [2]: import pandas as pd artwork = pd .read_csv ('https://raw.githubusercontent.com/rfordatascience/tidytuesda y/master/data/2021/2021-01-12/artwork.csv' ) artists = pd .read_csv ("https://github.com/tategallery/collection/raw/master/artist_ data.csv" ) artwork ["id" ] = artwork ["artistId" ] artwork = artwork [[ "id" , "year" , "acquisitionYear" , "title" , "medium" ]] artists = artists [[ "id" , "gender" ]] df = pd .merge (artwork , artists ) def dimension (med_string ): """ Assign a dimension to a given piece of artwork based on the description of the medium, supplied as a string.

Media that include the words "paper", "canvas", "oil", or "paint" are assumed 2D.

Media that are not 2d and include the words "bronze", "stone", or "ceramic" are assumed 3D.

Otherwise, the media is "Other/Unknown" @param med_string: str, the original medium @return dim: one of "2D", "3D", or "Other/Unknown" according to the rules abov e.

""" if type (med_string ) != str : med_string = str (med_string ) med_string = med_string .lower () if any ([ w in med_string for w in ["paper" , "canvas" , "oil" , "paint" ]]): return "2D" elif any ([ w in med_string for w in ["bronze" , "stone" , "ceramic" ]]): return "3D" else : return "Other/Unknown" df ["dimension" ] = [dimension (m) for m in df ["medium" ]] df = df [[ "title" ,"acquisitionYear" , "gender" , "dimension" ]] The title c o lu m n g ive s th e t it le o f e ach p ie ce. T he acquisitionYear st ate s th e ye ar in w hic h t h e a rtw ork w as acq uir e d b y th e T a te . T he gender c o lu m n g ive s th e g en d er o f t h e a rtist . The dimension c o lu m n st ate s wheth er t h e p ie ce is tw o-d im en si onal (lik e a d ra w in g o r a p ain tin g ) o r th re e-d im en si onal (lik e a sc ulp tu re o r c era m ic ). T his is dete rm in ed f r o m a m ore t h o ro ug h d esc rip tio n o f t h e m ed iu m u si ng t h e si m ple dimension() f u nctio n f r o m a b ove , a lt h o ug h a m ore c a re fu l c la ssi b e b en e In [ ]:

# use this block to inspect the data if you'd like 30/11/2021, 17:14 HW7 localhost:8888/lab 3/12 What Y o u S ho uld D o Cre ate a p lo t t o a n swe r t h e f o llo w in g q uest io n: H ow h as th e a m ount o f a rtw ork b y f e m ale a rtis ts in cre ase d w it h t im e, a s a f r a ctio n o f a ll a rtw ork o w ned b y th e T a te ? A re w om en b ette r r e p re se nte d in t h e T a te t h ro ug h c e rta in f o rm s of artist ic e xp re ssi on t h an o th ers? T o a n swe r t h is quest io n, c re ate t h e f o llo w in g p lo t:

T he ve rtic al a xis is th e p erc en ta g e o f a ll a rtw ork c re ate d b y fe m ale a rtist s whic h w as acq uir e d o n o r b efo re t h e st ate d d ate . Y o u m ay assu me t h at a rtw ork , o nce a c q uir e d , r e m ain s perm an en tly wit h t h e T a te (i. e . it is no t lo st o r so ld ).

S pecs There a re m ult ip le g ood a p pro ach es. A so lu tio n u si ng a for - o r while - lo op c an r e ceive p artia l c re d it . F or f u ll c re d it , n o e xp lic it lo op s!

It is no t n ecessa ry fo r yo ur o utp ut t o e xa ctly m atc h m in e - - f e el f r e e t o c h an g e c o lo rs, m od if y th e la b els, e tc . H ow eve r, yo u sh ould e n su re t h at yo u in clu d e a xis la b els an d t h e le g en d .

C om men ts an d d ocst rin g s are n o t n ecess ary in t h is pro b le m .

Yo u a re f r e e t o u se a n y Pyt ho n t o ols yo u "W hat if m y p lo t lo oks d i Yo ur th at le ad yo ur p lo t t o lo ok sl ig htly di quan tit a tive ly sim ila r a n d h as th e sa me q ualit a tive in te rp re ta tio n.

H in ts np.cumsum() . Y o u'll n eed t o a p pro p ria te ly so rt df 30/11/2021, 17:14 HW7 localhost:8888/lab 4/12 In [ ]:

# your solution here P ro ble m 2 T his pro b le m is base d o n t h e a rtic le O berm eye r, Z ., P ow ers, B ., V o geli, C ., & M ulla in ath an , S . (2 019). D isse ctin g r a c ia l b ia s in a n alg orit h m u se d t o m an ag e t h e h ealt h o f p op ula tio ns. Scie n ce , 3 66(6 464), 4 47-4 53. In t h is artic le , t h e a u th o rs use p atie n t m ed ic al r e c o rd s, d em og ra p hic s, a n d in su ra n ce c la im s to st ud y bia s in a m ach in e le arn in g m od el u se d t o p re d ic t p atie n t r isk . T his m od el h as been u se d t o m ake r e c o m m en d atio ns ab out w hic h p atie n ts sh ould b e a d m it te d t o m ore in te n si ve c a re p ro gra m s on t h e b asi s of t h eir h ealt h .

In t h is pro b le m , yo u w ill r e p lic ate se vera l o f t h e q ualit a tive The r e su lt s pre se nte d in t h is artic le w ere d isc usse d b y Dr. R uha B en ja m in in t h e vi deo " A re W e A uto m atin g R acism (h ttp s: //w ww.yo utu b e.c o m /w atc h ?v= Ok5 sK LX qyn Q) ," w hic h w as one o f t h e vi deo s we w atc h ed a s part o f o ur d isc ussi on o f a lg orit h m ic b ia s in W eek 8 . Y o u a re f r e e t o c o nsu lt e it h er t h e a rtic le o r t h e vi deo w hen co m ple tin g t h is assi gnm en t. W hile d oin g so m ay be in te re st in g , it is no t lik e ly to c o ncre te ly help yo u in t h e p ro b le m s belo w .

D ata A ccess In o rd er t o p ro te ct p atie n t p riva cy, t h e a u th o rs did n o t sh are t h e " re a l" d ata u se d in t h eir st ud y. In st ead , t h ey cre ate d a r a n d om iz e d ve rsi on o f t h e d ata t h at p re se rve s m an y of t h e sa me p atte rn s an d t re n d s. R un t h e c ell b elo w t o a ccess th e d ata . I h ave a lso u p lo ad ed t h e C SV U RL.

In [ ]:

import pandas as pd url = "https://gitlab.com/labsysmed/dissecting-bias/-/raw/master/data/data_new.csv? inline=false" df = pd .read_csv (url ) T here a re 4 8,7 84 p atie n ts re p re se nte d a s r o w s in t h e d ata , a n d 1 60 p ie ces of in fo rm atio n a b out e ach p atie n t re p re se nte d a s co lu m ns. R un t h e c o d e b elo w t o c h eck t h is:

In [ ]:

df .shape 30/11/2021, 17:14 HW7 localhost:8888/lab 5/12 A f e w o f t h e c o lu m ns are g oin g t o b e e sp ecia lly im porta n t in o ur a n alysi s: risk_score_t is th e a lg orit h m 's risk sc ore a ssi gned t o a g ive n p atie n t. cost_t is th e p atie n t's m ed ic al c o st s in t h e st ud y p erio d . race is th e p atie n t's se lf - re p orte d r a ce. T he a u th o rs white a n d black patie n ts.

gagne_sum_t is th e t o ta l n um ber o f c h ro nic illn esse s pre se nte d b y th e p atie n t d urin g t h e st ud y perio d . dem_female is a p atie n t se x in d ic ato r, w it h 1 in d ic atin g f e m ale p atie n ts an d 0 in d ic atin g m ale p atie n ts. R un t h e c o d e b elo w t o t a ke a lo ok a t t h ese c o lu m ns.

In [ ]:

cols = ['risk_score_t' , 'cost_t' , 'gagne_sum_t' ,'race' , 'dem_female' ] df [cols ].head () 30/11/2021, 17:14 HW7 localhost:8888/lab 6/12 P art A H ere 's ho w t h e a lg orit h m w as use d in t h e m ed ic al se ttin g : P atie n ts wit h h ig her sc ore s fr o m t h e a lg orit h m w ere m ore lik e ly to b e e n ro lle d in “ h ig h-risk c are m an ag em en t” p ro gra m s. A h ig h-risk c are m an ag em en t p ro gra m o h ealt h care p ro vi ders to h elp t h em m an ag e c o m ple x h ea lt h n ee d s. In o th er w ord s, If yo u a re ve ry sick, g ettin g a h ig h sc ore o n t h e a lg orit h m c an h elp yo u r e ce ive more m ed ic al atte n tio n. O ne o f t h e m ajo r p atie n ts, e ve n w hen t h o se B la ck p atie n ts w ere e q ually sick a s W hit e p atie n ts. In t h is part, yo u w ill r e p lic ate t h is T o d o s o , c re ate t h e f o llo w in g p lo t:

K ey P oin ts T he v e rtic al a xis g iv e s t h e p erc e n tile r is k o f p atie n ts a ssig ned b y t h e a lg orit h m , r o und ed t o t h e neare st p erc en ta g e p oin t . A p atie n t in t h e 8 5th p erc en tile , f o r e xa m ple , r e ceive d a r isk sc ore f r o m t h e alg orit h m h ig her t h an 8 4% of a ll p atie n ts an d lo w er t h an 1 5% of a ll p atie n ts. T he r a w r isk sc ore (n o t t h e p erc en tile ) o f e ach p atie n t is co nta in ed in t h e risk_score_t c o lu m n. T he h o riz o nta l a xis g iv e s t h e a ve ra g e n u m ber o f c h ro nic illn esse s p re se n te d b y p atie n ts in t h e co rre sp ond in g r is k p erc en tile . F or e xa m ple , W hit e m en in t h e 8 0th r isk sc ore p erc en tile p re se nte d , o n ave ra g e, a p pro xim ate ly tw o c h ro nic illn esse s. T he n um ber o f c h ro nic illn esse s pre se nte d b y a p atie n t is c o nta in ed in t h e "gagne_sum_t" c o lu m n o f t h e d ata . D i race ). T w o p an els dist in g uish b etw een m ale a n d fe m ale p atie n ts. T he dem_female c o lu m n g ive s th e se x o f e ach p atie n t, w it h 0 r e p re se ntin g m ale a n d 1 re p re se ntin g f e m ale . S pecs Ple ase a tte n d t o t h e f o llo w in g d eta ils: 30/11/2021, 17:14 HW7 localhost:8888/lab 7/12 It is im porta n t t h at yo u ro und t h e p erc en tile r is k s c o re s t o t h e n eare st p erc e n tile , a n d c o m pute t h e ave ra g e n um ber o f c o nd it io ns wit h in e ach r o und ed p erc en tile . T his m ean s th at, f o r e xa m ple , t h ere sh ould b e 1 01 d ata p oin ts (p erc en tile s 0%-1 00%) c o rre sp ond in g t o B la ck w om en , 1 01 o th er d ata p oin ts c o rre sp ond in g t o W hit e w om en , e tc . F ailu re t o r o und a n d c o m pute t h e m ean w ill r e su lt in yo ur p lo t co nta in in g a n u nre ad ab le n um ber o f p oin ts.

T he h o riz o nta l a xis, ve rtic al a xis, a n d a xis tit le s are a ll a p pro p ria te ly c a p it a lize d : t h e w ord is cap it a liz e d .

The le g en d t it le , a s well a s th e le g en d e n tr ie s, a re a lso a p pro p ria te ly ca p it a liz e d .

B eyo nd t h ese sp ecs, yo u a re f r e e t o m od if y th e c o lo rs, t r a n sp are n cy, e tc , a n d g et c re a tive w it h t h e t e xt. Y o u are n o t r e q uir e d t o r e p lic ate t h e e xa ct si ze o r a sp ect r a tio o f t h e H in ts T he o nly co lu m ns yo u n eed t o w ork w it h in t h is pro b le m a re : risk_score_t , race , gagne_sum_t (c o nta in in g t h e n um ber o f c h ro nic illn esse s per p atie n t), a n d dem_female The p lo ttin g a sp ect p ro b le m c an b e so lve d u si ng e it h er st an d ard matplotlib o r seaborn . C orre ct ap pro ach es usi ng e it h er se t o f t o ols will r e c eive f u ll c re d it .

T he p erc en tile s of a n d ata f r a m e c o lu m n df["x"] c an b e c o m pute d b y df["x"].rank()/len(df) . The r e su lt s will b e va lu es betw een 0 a n d 1 . O ne sh ould t h en m ult ip ly by 100 a n d round() t h e r e su lt s to o b ta in t h e p erc en tile s as in te g ers betw een 0 a n d 1 00.

To c o m pute t h e m ean n um ber o f c h ro nic illn esse s per p erc en tile , g ro up b y th e in te g er p erc e n tile s (a s well a s ra ce a n d se x) a n d t h en c o m pute t h e mean o f t h e gagne_sum_t c o lu m n. T he p lo tte d p oin ts sh ould c o rre sp ond t o ave ra g e n um ber o f c h ro nic c o nd it io ns, gro up ed b y p erc en tile s an d d em ogra p hic va ria b le s. In [ ]:

# feel free to use this cell for "scratchwork" (e.g. data exploration).

# Place your actual solution in the cell below In [ ]:

# your solution P art B In n o m ore t h an f o ur se nte n ces, d esc rib e t h e m ean in g o f t h e p lo t yo u p ro d uced in P art A . F or e xa m ple , su ppose th at P atie n t A is Bla ck, t h at P atie n t B is W hit e , a n d t h at b oth P atie n t A a n d P atie n t B h ave e xa ctly th e sa me ch ro nic illn esse s. A re P atie n t A a n d P atie n t B e q ually lik e ly to b e r e fe rre d t o t h e h ig h-risk c are m an ag em en t p ro gra m ?

[Yo ur d is c u ssio n h ere ] 30/11/2021, 17:14 HW7 localhost:8888/lab 8/12 P art C N ext, yo u'll p erfo rm a n a n alysi s to id en tif y th e so urc e o f t h is disp arit y in B la ck a n d W hit e p atie n ts. Y o u m ig ht im ag in e t h at t h e m od el w as tr a in ed t o b ase it s risk sc ore s on a n " o ve ra ll le ve l o f h ea lt h " in t h e t r a in in g d ata .

H ow eve r, it is ve ry di For t h is re aso n, t h e a lg orit h m st ud ie d w as t r a in ed in st ead u si ng to ta l m ed ic al c o sts a s th e t a rg et va ria b le . T hat is: T he r isk sc ore a n a g en t r e ceive s is a f u nctio n o f t h e m od el' s pre d ic tio n o f t h e t o ta l m ed ic al c o st s whic h w ill b e in cu rre d b y th at in d ivi dual. T his is a su per co st s are r e g ula rly re co rd ed in in su ra n ce c la im s data .

In t h is pro b le m , yo u'll u se lin ear r e g re ssi on t o e st im ate t h e d i an d B la ck p atie n ts in t h is data se t, a n d c o m men t o n t h is re su lt in t h e c o nte xt o f P art A a n d B .

W hat Y o u S ho uld D o 1. If yo u m od i df in a n y way, yo u sh ould r e -ru n t h e c o d e in w hic h yo u lo ad t h e d ata fr a m e. 2. R un t h e su pplie d c ell in o rd er t o lim it t h e c o lu m ns in t h e d ata f r a m e t o t h e o nes yo u w ill u se in t h is an alysi s. 3. T he race c o lu m n o f t h e d ata is cu rre n tly a st rin g . E nco d e it u si ng in te g er la b els. 4. P artit io n t h e d ata in to a t a rg et d ata y c o nsi st in g o f t h e cost_t c o lu m n o f df . L et t h e p re d ic to r d ata X co nta in a ll o th er c o lu m ns, e xc lu d in g cost_t . 5. P erfo rm a t r a in -te st sp lit o f X a n d y , u si ng 2 0% of t h e d ata a s te st d ata . P le a se p ass th e a rg um en t random_state = 2021 t o yo ur sp lit f u nctio n in o rd er t o e n su re r e p ro d ucib ilit y . I m porta n t : yo u sh ould d o t h is usi ng o nly one f u nctio n c all. 6. C re ate a lin ear r e g re ssi on m od el a n d score o f t h e m od el o n t h e tr a in in g a n d t e st in g d ata . H ere a re t h e sc ore s th at I g o t - - it 's ok if yo urs are a lit tle d i Tra in in g sc ore : 0.12629789734544883 T e st in g sc ore : 0.12415443228313183 7. B ase d t h ese r e su lt s, c o m men t o n w heth er yo u a re c o ncern ed a b out o ve r Note : t h ese a re n o t "a ccu ra cy" sc ore s but r a th er " c o e on st atist ic al t a sk s are c o m mon in m ed ic al a n d b io lo gic al a p plic atio ns. 8. E xa m in e t h e coef_ a ttr ib ute o f t h e race c o lu m n is th e d ata f r a m e. T his m ean s th at t h e ve ry coef_ a rra y give s th e m od el' s est im ate o f t h e d i H ere 's what I g ot - - it 's ok if yo ur a n swe r is a lit tle d i C oe race : 579.9031747777375 . 9. B la ck p atie n ts in t h e U S t e n d t o g en era te lo w er m ed ic al c o st s th an t h eir e q ually- sick W hit e c o unte rp arts, d ue t o lo ng -st an d in g d isp arit ie s in a ccess to m ed ic al r e so urc es. U sing yo ur r e su lt f r o m S te p 8 : S ta te yo ur e st im ate o f t h e d i D esc rib e in n o m ore t h an 4 se nte n ces ho w yo ur r e su lt w ould e xp la in t h e d isp arit ie s in r isk sc ore s fr o m P art A . 30/11/2021, 17:14 HW7 localhost:8888/lab 9/12 Note : T he e stim ate d c o st d is p arit y in t h e p ub lis h ed p ap er is h ig her, o ve r t w ic e t h e r e su lt g iv e n h ere . T his m ay re S te p 1 If yo u m od i df in a n y way in P art A , yo u sh ould r u n t h e c o d e b elo w t o r e lo ad t h e d ata fr a m e.

In [ ]:

# Step 1: run, do not modify df = pd .read_csv (url ) S te p 2 R un t h is cell in o rd er t o lim it t h e c o lu m ns in t h e d ata f r a m e t o t h e o nes yo u w ill u se in t h is an alysi s. 30/11/2021, 17:14 HW7 localhost:8888/lab 10/12 In [ ]:

# Step 2: run, do not modify cols = ['cost_t' , 'race' , 'dem_female' , 'dem_age_band_18-24_tm1' , 'dem_age_band_25-34_tm1' , 'dem_age_band_35-44_tm1' , 'dem_age_band_45-54_tm1' , 'dem_age_band_55-64_tm1' , 'dem_age_band_65-74_tm1' , 'dem_age_band_75+_tm1' , 'alcohol_elixhauser_tm1' , 'anemia_elixhauser_tm1' , 'arrhythmia_elixhauser_tm1' , 'arthritis_elixhauser_tm1' , 'bloodlossanemia_elixhauser_tm1' , 'coagulopathy_elixhauser_tm1' , 'compdiabetes_elixhauser_tm1' , 'depression_elixhauser_tm1' , 'drugabuse_elixhauser_tm1' , 'electrolytes_elixhauser_tm1' , 'hypertension_elixhauser_tm1' , 'hypothyroid_elixhauser_tm1' , 'liver_elixhauser_tm1' , 'neurodegen_elixhauser_tm1' , 'obesity_elixhauser_tm1' , 'paralysis_elixhauser_tm1' , 'psychosis_elixhauser_tm1' , 'pulmcirc_elixhauser_tm1' , 'pvd_elixhauser_tm1' , 'renal_elixhauser_tm1' , 'uncompdiabetes_elixhauser_tm1' , 'valvulardz_elixhauser_tm1' , 'wtloss_elixhauser_tm1' , 'cerebrovasculardz_romano_tm1' , 'chf_romano_tm1' , 'dementia_romano_tm1' , 'hemiplegia_romano_tm1' , 'hivaids_romano_tm1' , 'metastatic_romano_tm1' , 'myocardialinfarct_romano_tm1' , 'pulmonarydz_romano_tm1' , 'tumor_romano_tm1' , 'ulcer_romano_tm1' ] df = df [cols ] S te p 3 T he race c o lu m n o f t h e d ata is cu rre n tly a st rin g . E nco d e it u si ng in te g er la b els. 30/11/2021, 17:14 HW7 localhost:8888/lab 11/12 In [ ]:

# Step 3: your code here S te p 4 P artit io n t h e d ata in to a t a rg et d ata y c o nsi st in g o f t h e cost_t c o lu m n o f df . L et t h e p re d ic to r d ata X co nta in a ll o th er c o lu m ns, e xc lu d in g cost_t . In [ ]:

# Step 4: your code here S te p 5 P erfo rm a t r a in -te st sp lit o f X a n d y , u si ng 2 0% of t h e d ata a s te st d ata . P le a se p ass th e a rg um en t random_state = 2021 t o yo ur sp lit f u nctio n in o rd er t o e n su re r e p ro d ucib ilit y . Im porta n t : yo u sh ould d o t h is usi ng o nly one f u nctio n c a ll. In [ ]:

# Step 5: your code here S te p 6 C re ate a lin ear r e g re ssi on m od el a n d score o f t h e m od el o n t h e t r a in in g an d t e st in g d ata . H ere a re t h e sc ore s th at I g o t - - it 's ok if yo urs are a lit tle d i Tra in in g sc ore : 0.12629789734544883 T e st in g sc ore : 0.12415443228313183 In [ ]:

# Step 6: your code here S te p 7 B ase d t h e r e su lt s ab ove , c o m men t o n w heth er yo u a re c o ncern ed a b out o ve r Note : t h ese a re n o t " a ccu ra cy" sc ore s but r a th er " c o e b ut lo w sc ore s on st atist ic al t a sk s are c o m mon in m ed ic al a n d b io lo gic al a p plic atio ns.

[Yo ur c o m men t o n o ve r ] 30/11/2021, 17:14 HW7 localhost:8888/lab 12/12 Ste p 8 E xa m in e t h e coef_ a ttr ib ute o f t h e race c o lu m n is th e p re d ic to r d ata f r a m e. T his m ean s th at t h e ve ry coef_ a rra y give s th e m od el' s est im ate o f t h e d i H ere 's what I g ot - - it 's ok if yo ur a n swe r is a lit tle d i C oe race : 579.9031747777375 . In [ ]:

# Step 8: your code here S te p 9 B la ck p atie n ts in t h e U S t e n d t o g en era te lo w er m ed ic al c o st s th an t h eir e q ually- sick W hit e c o unte rp arts, d ue t o lo ng -st an d in g d isp arit ie s in a ccess to m ed ic al r e so urc es. U sing yo ur r e su lt f r o m S te p 8 : S ta te yo ur e st im ate o f t h e d i D esc rib e in n o m ore t h an 4 se nte n ces ho w yo ur r e su lt w ould e xp la in t h e d isp arit ie s in r isk sc ore s fr o m P art A . [yo ur d is c u ssio n o f y o ur r e su lt s h ere ! ]