Use the file names to identify the corresponding questions. For instance,Question1.yourname. For all the questions below, use degrees of freedom N -...

• Use the file names to identify the corresponding questions. For instance,Question1.yourname.sas'.

• For all the questions below, use degrees of freedom N - 1.

1. (32 points) Given a dataset, (file data.online.scores.txt) which includes the records of students' exam scores (sample from the population) for the past few years of an online course. The first column students' id, the second column is the mid-term scores, and the third column is the final scores, and data are splitted by tab. Based on the dataset, give out the following statistical description of data. If the result is not integer, then round it to 3 decimal places. Give out the basic statistical description about mid-term scores.

a. (8 p) Max, min

b. (12 p) First quartile Q1, median, third quartile Q3.

c. (4 p) The mean score.

d. (4 p) The mode score.

e. (4 p) Empirical Variance.

2. (25 points) Based on the data of students' score (file data.online.scores.txt). Please normalize the mid-term score using z-score normalization (divided by the empirical standard deviation).

a. (10 p) Compare the empirical variance before and after normalization.

b. (5 p) Given original score of 90, what is the corresponding score after normalization?

c. (5 p) Pearson's correlation coefficient between midterm scores and final scores is:

d. (5 p) Covariance between midterm scores and final scores is:

data.online.score.txt

09795

18399

27981

37284

49284

57891

67989

76867

87876

97781

107583

119099

128584

1377100

148986

158687

166587

176794

186776

199393

208391

2197100

228689

236143

248577

259690

267998

276168

289393

299897

308487

315497

328875

338695

347793

359199

3695100

378083

386278

396481

408089

4194100

429079

438278

447466

455981

467189

479293

487999

499497

506178

517694

529487

538397

543982

558093

565371

5710098

587891

599796

606993

615593

629083

637597

645882

659399

668198

676888

687792

697373

709198

719287

727577

736593

747496

757795

766162

778086

785261

795357

808493

819471

829099

836478

845587

855881

869186

877597

889192

898798

9090100

918897

926893

936180

949597

955576

967790

979198

988486

997992

1007788

1019092

1028295

1038391

1047790

1057485

1068281

1078092

1086490

1096691

1106388

1115064

1127981

1138897

1148784

1157682

1167499

117100100

1187684

1198894

1207782

1217587

1228793

1238290

1247277

1257274

1267892

1276897

1288298

1297897

1306059

1316977

13210095

1337092

1347383

1359295

1366680

13762100

1388395

13974100

1407975

1417289

1426995

1437186

1447796

1458391

1465776

1477182

1488188

1497296

1507687

1517891

1527566

1537496

1549095

1557082

1566586

1579093

1585561

1598090

1609686

1617792

1628595

1638598

1647582

1658280

1668592

1675074

1688199

1697286

1709794

1715474

1727481

1736578

1747687

1758080

1768595

1775883

1786076

1797193

1809395

1815667

1827999

1839597

1846496

1858392

1869190

1877476

1886796

1897661

1909797

1918689

19210096

1938089

1947592

1955459

1967190

1976261

1988197

1996185

2006761

2015872

2027394

2039391

2046356

2057597

2068694

2079195

2085989

2096994

2107099

2116290

2125866

2138992

2145286

2158573

2166389

2179195

2189793

2196868

2208192

2218190

2227787

2239090

2247598

2257796

2266574

2277692

2288388

2295384

2305882

2318392

2326284

2338692

2346189

2357293

2368487

2378382

2387095

2399899

2407496

2417688

2426893

2436885

2448788

2456677

2468590

2479077

2487386

2497496

25095100

2519799

2528783

2537399

2548271

2558396

2568497

2577485

2586988

2599293

2606284

2619297

2625867

2637885

2646896

2657483

2666789

2676377

2685990

26967100

2707496

2717099

2727567

2738092

2748497

2759494

2766695

2778696

2786376

2798083

2807171

2818087

2828388

2839183

28496100

2858699

2868593

2876199

2885182

2897792

2907286

2916890

2928392

2938086

2946179

2959996

2969675

2977480

2987067

2997389

3005664

3016488

3027899

3036497

3047379

30589100

3069797

3078988

3088085

3098597

3106980

3117284

3127787

3137085

3148495

3156989

3168393

3174567

3187596

3197365

3208993

3213851

3225065

3237577

3248985

3258589

3268277

3277892

3288894

3296696

3309499

3317778

3327071

3338976

3345159

3359385

3365885

3378085

3388393

33910098

3409595

3417993

3426572

3437492

3447296

3459088

3466582

3475789

34887100

3497771

3506076

35190100

3528382

3535875

3543751

3558079

3567892

3578795

3586986

3598494

3607369

3619391

3629585

3636699

3648397

3657273

3669895

3678096

3688598

3697792

3706680

37187100

3725892

3737486

3747990

3757684

3766764

3776876

3785095

3797474

3807295

3818088

3829471

3837688

3848594

3854485

3865888

3875664

3888284

3897776

3908093

3919697

3926888

3935285

3946476

39592100

3968384

3975894

3989187

3997693

4008178

4019998

4029595

4036395

4047080

4058196

4065774

4079285

4089189

4095286

4108083

4118091

4127074

4139191

4148999

4158177

4167583

4178598

4188278

4198397

4204443

4218799

4228984

4238988

4245685

4258696

4267494

4277699

4288888

4298495

4305878

4317196

4328686

4334873

4348386

4357495

4367085

4379387

4385888

4394574

4409484

4418897

4427777

4439792

4448297

4459697

4467072

4478195

4486196

4498291

4507877

4518287

4526067

45396100

4545696

4556269

4568891

45710092

4589196

4597191

4607087

4617483

4628377

4637271

464100100

4655791

4669986

4678098

4688084

4698194

4706263

4716681

4726399

4738593

4746783

4756497

47687100

4778385

4788289

4797690

4805167

4815762

4828498

4837387

4847187

4859484

4868194

4877398

4888998

4899299

4904568

49110094

49289100

4937694

4949087

4959699

4968181

4979198

4985881

4999693

5006599

5016995

5026886

5035877

5047699

5056779

5066488

5078499

5086576

50994100

5105279

5118490

5128985

5139199

5148593

5155589

5165887

5179299

5189495

5197783

5206492

5219397

5227981

5236676

5247996

5257660

5268890

5278067

5285071

5296685

5308197

5319697

5327487

5336982

5349483

5358276

5368795

5375063

5387188

5397087

5407994

5418993

5426285

5435850

5448299

5457484

5466181

54710096

5487086

5497088

5509496

55169100

5526389

55382100

5546775

5558590

5567178

5576795

5587493

5597193

5606576

56185100

56291100

5639492

5649076

56590100

5666174

5678983

5685477

5697898

5708598

5719296

57291100

5739996

5749096

5755470

57610099

5776272

5786681

5799197

5807185

5817691

5827699

5839390

5847174

5858396

58684100

5876082

5889984

5898797

5909298

5919089

5929598

5938792

5948484

5958492

5968692

5977873

5989079

5998899

6004360

6018885

6028497

6038169

6048897

6056583

6065779

6078086

6087884

6098986

6106995

6117792

61297100

61395100

6147589

6158999

6167584

6177296

6189295

6199986

6209592

6219493

6228995

6237798

6247589

6257678

6265786

62759100

6289783

6299198

6306575

6315867

6327195

6336654

6345735

6358786

6369185

6379096

6387398

6398193

6405851

6417271

6426753

6437288

6449289

6458286

6467992

6479293

6489482

6497378

6507588

6517697

65272100

6538380

6547494

6559999

65666100

6574971

6587983

6597294

6608382

6615786

6627987

6639198

6647372

6655694

6666785

6676499

6688169

6699195

6707880

6715262

6726987

6735779

6747293

6758095

6768391

6776193

6787696

67985100

6806696

6816679

6828191

6838697

6849697

6857688

6866994

6875668

6889193

6895870

6905080

6915996

6925687

6937587

6945167

6956077

6968799

6979291

6987189

6999889

7007066

7018399

7027194

7036988

7049397

7058895

7067979

7078879

7087398

7098494

7107889

7117095

7126283

7137083

7148098

7158190

7168190

7178892

7185277

7197196

7206457

7216796

7227288

7235886

7245887

7258696

72610097

7276583

7288498

7296973

7309688

7319594

7329784

7336988

7345469

7354951

7367998

73797100

7387594

7397581

7409396

7419198

7428698

7437585

7448497

7458987

7468788

7477982

74894100

7499780

7507396

7518488

7529686

7537572

7549694

7558482

7568488

7577797

7584870

7597491

76099100

7619777

7626871

7639597

7649991

7658499

7667788

7675265

7687185

7697194

7706583

7717570

7727585

7737998

7746469

7759397

7767182

7778982

7787685

7797187

7808986

7818086

7827288

7839483

7849496

7857894

7867478

7877587

7886882

7896088

7907899

7915397

7927886

7938177

7947382

7957498

7968097

79785100

79896100

7998197

8009292

8017396

8029495

8039086

80479100

8059592

8068986

8076184

8087179

8093848

8107688

8118393

8128393

8136472

8146487

8155354

8167170

8177472

8188385

8197461

8205473

8216081

8228099

8239390

8248989

8259486

8268199

8276386

828100100

8299784

8307579

8317568

83286100

83384100

8347283

8359494

8369096

8376083

8384281

8397879

8406692

8413735

8428384

8434162

8449799

8457993

84670100

8478596

8487799

8498397

8505697

8517692

85210088

8536897

8548085

8556393

8567794

8576497

8589692

8597774

8606992

8617796

8628393

8636784

8645985

8659490

8667293

8677374

86882100

8694973

8706066

8718797

8727772

8735956

8748890

8757483

8768897

8777885

8786690

8796977

8805055

8817484

8828579

8839796

8849697

88590100

8866689

8877584

8887587

8897799

8907691

8919898

8926265

8938598

8948097

8956683

8965353

8979199

8988795

8997695

9007190

9018698

9025275

9038898

9047179

9059597

9067797

9077795

9089297

9099189

9106480

91190100

9126699

9135768

9145661

9156694

9166382

9178396

9188592

9195883

9207693

9218383

9225694

9238995

9248998

9257293

9268094

9279092

9288085

9299195

9306258

93175100

9328995

9336164

9347298

9355878

9366579

9379796

9389899

9399093

9406086

94187100

9428086

9437289

9447985

9457893

9468498

9476193

9488192

9498390

9509597

9516593

9524670

9535672

9548680

9558490

9567693

9576894

9587792

9599986

9606969

9618978

9629596

9639990

9645161

9657795

9667481

9678590

9686772

9696599

9709195

9718699

9727783

97368100

9746097

9755480

9766998

9777594

9789298

9795881

9807774

9818175

9828489

9839399

9845777

9858186

9864470

9879685

9887493

9898199

9908381

9917889

9926383

9935162

9946674

9959798

9968296

9978591

9988499

9995687

3. (38 points) Given the inventories of two libraries Citadel's Maester Library (CML) and Castle Black's library(CBL), compare the similarity between this two libraries by using the different proximity measures. if the result is not integer, then round it to 3 decimal places.

a. (5 p) Given 200 books, the following table summarizes how many books are supplied by corresponding library in Table 1. In Table 1, for CBL = 0, CML = 0, it corresponds the number of items among the 200 items that are served neither by CBL nor CML. For CBL = 1, CML = 0, it corresponds the number of items among the 200 items that are served by CBL but not CML. So on and so forth. Based on Table 1, calculate the Jaccard coefficient of Citadel's Maester Library (CML) and Castle Black's library(CBL).

Citadel's Maester Library (CML)

Castle Black's library(CBL) 0 1

0 20 120

1 2 58

b. (15 p) For each kind of books, we have multiple copies. Based on all books (treat the counts of the 100 books as a feature vector of the two libraries), (file data.libraries.inventories.txt), calculate the minkowski distance of the two vectors with regard to different h values:

1. h = 1

2. h=2

3. h = ∞

c. (9 p) The Cosine similarity between Citadel's Maester Library (CML) and Castle Black's with regard to the feature vector. (file data.libraries.inventories.txt).

d. (9 p) Kullbac-Leibler divergence between Citadel's Maester Library (CML) and Castle Black's library(CBL) with regard to the feature vector. We denote that there are i_1 of book1 in Citadel's Maester Library (CML), and j_1 of book1 in Castle Black's library(CBL). Assume that someone will pick up a book randomly, the probability of this person to pick up book 1 in Citadel's Maester Library (CML) is i_1 / (i_1 + ... + i_100). Based on this probability distribution, calculate the Kullback-Leibler divergence of these two libraries P(CML || CBL).

DATA Library inventories:

librarybook_1book_2book_3book_4book_5book_6book_7book_8book_9book_10book_11book_12book_13book_14book_15book_16book_17book_18 book_19book_20book_21book_22book_23book_24book_25book_26book_27book_28book_29book_30book_31book_32book_33book_34book_35 book_36book_37book_38book_39book_40book_41book_42book_43book_44book_45book_46book_47book_48book_49book_50book_51book_52 book_53book_54book_55book_56book_57book_58book_59book_60book_61book_62book_63book_64book_65book_66book_67book_68book_69 book_70book_71book_72book_73book_74book_75book_76book_77book_78book_79book_80book_81book_82book_83book_84book_85book_86 book_87book_88book_89book_90book_91book_92book_93book_94book_95book_96book_97book_98book_99book_100

CML5313690103369319551192148793048802427717013741146116186101173110501491393011986972563173130601481296916783 12128133173149105641341372117911635119601401506814519780731283180961119512213264181165129796372187991434913184179 21511276645014446110163183107129162173

CBL1038013216714417210889103636558141116152122129831751161736761114195166854287183217125109719371772091345999147 75501811021731601479790571511001021811181631411461491492501281551481231898018680114102316634592131042513175164214 537911875169192921391221231236113590125118

4. (5 points) The Table 2 is a summary about customers' purchase history of diapers and beer. Calculate the chi-square correlation value. If the result is not integer, then round it to 3 decimal places.

Table 2: Purchase history.

Buy diaper Do not buy diaper

Buy beer 150 40

Do not buy beer 15 3300