Thesis Update - Fuzzy Clustering

This week I performed a completed fuzzy clustering set upon data provided by Dr Poon. I firstly checked which number of clusters was best, using the clValid function, and then performed a fuzzy clustering analysis upon the data using this number of clusters. The results are as follows:

Clustering Determination using clValid:

Clustering Determination using clValid:

Cluster sizes:
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

Validation Measures:
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

fanny Connectivity 17.6429 15.6361 26.2500 38.6222 44.8575 47.1246 52.2718 86.2238 80.3901 83.1063 75.4885 99.8266 79.9262 88.6492 103.7821 141.2881 94.4425 108.0456 88.9873 109.7246 104.8306 128.4329 116.3290 96.4512 118.6901 129.3504 100.5310 137.2397 87.9849 120.8452 103.8091 131.6369 124.2206 137.7921 150.8726 140.9762 169.8508 156.1194 165.8202 179.5583 188.7667 184.4540 196.6766 194.9726 204.8901 192.5278 222.9175 219.1512 234.1079 239.3738 250.2071 255.2821 244.8940 243.3845 242.0552 251.2544 245.1659 252.9909 263.9631 285.9647 300.1361 303.7901 304.6492 310.1956 342.0210 338.8845 350.2933 360.7131 360.5933 361.4758 347.8706 357.0214 366.3710 361.6758 375.6111 355.5508 363.5639 369.8798 376.3706 380.6302 384.4921 391.0083 396.2425 400.8155 409.2960 411.4611 419.0857 422.3060 419.7639 424.0175 428.4333 419.9694 428.1302 429.0349 434.1635 423.6893 429.7310 432.2143 441.9460 451.9274 464.2226 467.7210 470.2028 474.5651 476.5817 463.7944 466.9492 471.0075 476.4484 470.1337 476.7099 477.4099 486.1218 476.2429 482.7587 487.9103 489.0440 491.6155 495.2456 497.9123 496.1159 496.8548 501.0881 497.7444 497.8548 503.6226 506.5488 509.6853 513.5020
Dunn 0.0122 0.0248 0.0280 0.0271 0.0221 0.0241 0.0250 0.0107 0.0169 0.0227 0.0201 0.0280 0.0266 0.0424 0.0180 0.0222 0.0264 0.0464 0.0521 0.0497 0.0470 0.0205 0.0392 0.0394 0.0518 0.0359 0.0389 0.0251 0.0389 0.0234 0.0389 0.0374 0.0241 0.0433 0.0209 0.0469 0.0466 0.0469 0.0433 0.0209 0.0469 0.0277 0.0389 0.0414 0.0456 0.0569 0.0418 0.0570 0.0365 0.0277 0.0363 0.0299 0.0299 0.0299 0.0639 0.0561 0.0560 0.0757 0.0547 0.0889 0.0375 0.0375 0.0375 0.0375 0.0525 0.0657 0.0298 0.0263 0.0263 0.0263 0.0525 0.0525 0.0531 0.0531 0.0531 0.0500 0.0638 0.0510 0.0633 0.0906 0.0794 0.0626 0.0626 0.0626 0.0626 0.0626 0.0309 0.0248 0.0275 0.0275 0.0257 0.0257 0.0257 0.0257 0.0259 0.0420 0.0444 0.0457 0.0457 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0277 0.0313 0.0313 0.0357 0.0393 0.0528 0.0528 0.0565 0.0589 0.0589 0.0682 0.0682 0.0427 0.0427 0.0427 0.0427
Silhouette 0.4849 0.5735 0.4836 0.4278 0.3433 0.3887 0.3835 0.2603 0.3265 0.2677 0.2010 0.1594 0.1656 0.2525 0.2271 0.2583 0.3578 0.3457 0.3973 0.3630 0.3293 0.2901 0.3419 0.3399 0.2816 0.2861 0.3980 0.2971 0.4109 0.3093 0.3387 0.3537 0.2285 0.3295 0.3190 0.3041 0.3226 0.3264 0.3200 0.3461 0.3114 0.3143 0.2946 0.2865 0.2761 0.2809 0.2621 0.3140 0.3073 0.3025 0.2784 0.2374 0.3115 0.2992 0.3334 0.2961 0.3200 0.3064 0.3040 0.3350 0.3667 0.3669 0.3589 0.3601 0.2290 0.2255 0.2234 0.2512 0.2477 0.2503 0.2861 0.2897 0.3071 0.3324 0.3213 0.3611 0.3522 0.3544 0.3566 0.3589 0.3557 0.3503 0.3460 0.3425 0.3363 0.3387 0.1350 0.1511 0.1839 0.1924 0.1592 0.1608 0.1769 0.1797 0.1723 0.2039 0.2042 0.2087 0.2017 0.1973 0.1897 0.1964 0.1876 0.1842 0.1806 0.2261 0.2210 0.2245 0.2239 0.2377 0.2269 0.2208 0.2157 0.2408 0.2396 0.2453 0.2535 0.2478 0.2572 0.2528 0.2627 0.2542 0.2561 0.2758 0.2765 0.2736 0.2681 0.2727 0.2608

Optimal Scores:

Score Method Clusters
Connectivity 15.6361 fanny 3
Dunn 0.0906 fanny 81
Silhouette 0.5735 fanny 3

For Connectivity and Silhouette (3)

Fuzzy Clustering Results, using 3 clusters:

Fuzzy Clustering object of class 'fanny' :
m.ship.expon. 2
objective 1944560
tolerance 1e-15
iterations 32
converged 1
maxit 500
n 263
Membership coefficients (in %, rounded):
[,1] [,2] [,3]
[1,] 64 20 15
[2,] 66 19 15
[3,] 69 17 14
[4,] 70 17 13
[5,] 58 23 19
[6,] 63 20 16
[7,] 66 19 15
[8,] 68 18 15
[9,] 69 17 14
[10,] 70 17 13
[11,] 72 16 11
[12,] 73 16 11
[13,] 74 15 11
[14,] 72 17 11
[15,] 70 18 13
[16,] 74 15 12
[17,] 71 16 13
[18,] 76 14 10
[19,] 73 15 12
[20,] 75 15 11
[21,] 73 15 12
[22,] 12 84 4
[23,] 10 86 4
[24,] 10 86 4
[25,] 10 86 3
[26,] 11 85 4
[27,] 12 84 4
[28,] 12 84 4
[29,] 14 81 5
[30,] 15 81 5
[31,] 17 78 5
[32,] 18 76 6
[33,] 19 74 7
[34,] 19 75 6
[35,] 19 74 7
[36,] 21 72 7
[37,] 23 69 8
[38,] 23 69 8
[39,] 24 68 8
[40,] 24 68 8
[41,] 25 65 9
[42,] 28 62 10
[43,] 28 62 10
[44,] 18 13 69
[45,] 18 13 69
[46,] 12 8 80
[47,] 11 7 81
[48,] 12 8 80
[49,] 12 8 80
[50,] 12 8 80
[51,] 14 10 76
[52,] 16 12 72
[53,] 14 10 76
[54,] 13 9 79
[55,] 13 8 79
[56,] 14 9 78
[57,] 14 9 78
[58,] 14 9 77
[59,] 15 10 75
[60,] 16 10 74
[61,] 16 11 73
[62,] 17 11 72
[63,] 18 12 70
[64,] 20 13 67
[65,] 21 14 64
[66,] 9 88 3
[67,] 9 88 3
[68,] 9 88 3
[69,] 9 88 3
[70,] 8 89 3
[71,] 10 87 3
[72,] 11 85 4
[73,] 11 85 4
[74,] 11 85 4
[75,] 13 82 5
[76,] 14 81 5
[77,] 14 81 5
[78,] 14 81 5
[79,] 16 79 6
[80,] 14 80 5
[81,] 16 78 6
[82,] 16 78 6
[83,] 15 79 6
[84,] 15 79 6
[85,] 16 78 6
[86,] 21 71 8
[87,] 18 75 7
[88,] 74 14 12
[89,] 74 14 12
[90,] 80 12 8
[91,] 81 12 7
[92,] 78 13 9
[93,] 75 14 11
[94,] 72 14 13
[95,] 69 15 16
[96,] 56 18 26
[97,] 50 18 32
[98,] 60 17 23
[99,] 72 14 14
[100,] 65 16 18
[101,] 60 17 23
[102,] 52 18 30
[103,] 52 18 30
[104,] 52 18 30
[105,] 48 18 34
[106,] 41 17 41
[107,] 31 15 53
[108,] 36 16 48
[109,] 33 16 51
[110,] 74 19 7
[111,] 77 17 7
[112,] 72 21 7
[113,] 71 22 7
[114,] 78 16 6
[115,] 80 14 6
[116,] 80 14 6
[117,] 82 12 6
[118,] 81 12 7
[119,] 78 13 9
[120,] 78 13 9
[121,] 80 12 8
[122,] 80 12 8
[123,] 77 13 10
[124,] 78 13 9
[125,] 75 14 11
[126,] 74 16 11
[127,] 72 16 12
[128,] 70 17 13
[129,] 66 18 16
[130,] 67 19 15
[131,] 65 19 15
[132,] 33 17 50
[133,] 35 17 49
[134,] 34 17 49
[135,] 34 17 49
[136,] 32 16 53
[137,] 26 14 60
[138,] 25 14 62
[139,] 21 12 67
[140,] 18 11 71
[141,] 16 10 74
[142,] 17 10 73
[143,] 17 10 73
[144,] 16 10 74
[145,] 16 10 74
[146,] 14 9 78
[147,] 13 8 79
[148,] 13 8 80
[149,] 12 8 80
[150,] 12 8 80
[151,] 13 8 79
[152,] 13 8 79
[153,] 14 9 77
[154,] 12 84 4
[155,] 14 82 4
[156,] 16 79 5
[157,] 18 77 5
[158,] 21 73 6
[159,] 21 74 6
[160,] 25 68 6
[161,] 31 61 7
[162,] 36 56 8
[163,] 40 52 8
[164,] 44 48 9
[165,] 48 43 9
[166,] 46 46 9
[167,] 51 41 9
[168,] 58 34 9
[169,] 60 31 9
[170,] 64 28 8
[171,] 65 27 8
[172,] 68 25 8
[173,] 74 19 7
[174,] 77 16 7
[175,] 76 17 7
[176,] 63 29 8
[177,] 65 28 8
[178,] 65 28 8
[179,] 66 26 8
[180,] 68 25 7
[181,] 73 20 7
[182,] 76 18 7
[183,] 77 17 6
[184,] 77 16 6
[185,] 80 14 6
[186,] 81 13 6
[187,] 79 14 6
[188,] 77 16 7
[189,] 79 15 6
[190,] 79 14 7
[191,] 79 13 7
[192,] 77 15 8
[193,] 78 14 8
[194,] 76 15 9
[195,] 75 15 10
[196,] 71 17 12
[197,] 70 18 12
[198,] 13 83 4
[199,] 12 84 4
[200,] 12 84 4
[201,] 12 85 4
[202,] 11 85 4
[203,] 11 85 4
[204,] 10 86 3
[205,] 10 87 3
[206,] 9 88 3
[207,] 9 88 3
[208,] 9 88 3
[209,] 8 89 3
[210,] 9 88 3
[211,] 8 89 3
[212,] 9 88 3
[213,] 12 85 3
[214,] 13 83 4
[215,] 12 84 4
[216,] 15 81 4
[217,] 18 77 5
[218,] 21 73 6
[219,] 18 76 5
[220,] 48 44 8
[221,] 49 43 8
[222,] 50 42 8
[223,] 51 41 8
[224,] 38 54 8
[225,] 49 43 8
[226,] 53 38 8
[227,] 62 30 9
[228,] 64 27 9
[229,] 68 23 9
[230,] 66 25 9
[231,] 62 29 10
[232,] 57 33 10
[233,] 57 33 10
[234,] 57 32 11
[235,] 58 31 11
[236,] 58 31 11
[237,] 56 32 12
[238,] 55 31 14
[239,] 54 31 15
[240,] 54 31 15
[241,] 53 30 16
[242,] 14 81 5
[243,] 13 82 5
[244,] 13 83 5
[245,] 12 84 4
[246,] 12 84 4
[247,] 11 86 4
[248,] 10 86 3
[249,] 10 87 3
[250,] 10 87 3
[251,] 10 87 3
[252,] 9 88 3
[253,] 10 87 3
[254,] 9 88 3
[255,] 10 87 3
[256,] 12 84 4
[257,] 12 84 4
[258,] 13 83 4
[259,] 14 82 4
[260,] 15 81 4
[261,] 16 80 4
[262,] 16 79 4
[263,] 19 76 5
Fuzzyness coefficients:
dunn_coeff normalized
0.5928260 0.3892389
Closest hard clustering:
[1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[60] 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 1 1 1 1 1 1 1 1 1
[119] 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 1 2 1 1 1 1 1 1 1 1 1 1 1
[178] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1
[237] 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

Available components:
[1] "membership" "coeff" "memb.exp" "clustering" "k.crisp" "objective" "convergence" "diss"
[9] "call" "silinfo" "data"

[,1] [,2] [,3]
[1,] 0.64308606 0.20316617 0.15374776
[2,] 0.65740785 0.19464854 0.14794361
[3,] 0.68964138 0.17457229 0.13578633
[4,] 0.70191629 0.17154831 0.12653540
[5,] 0.58358282 0.22519072 0.19122646
[6,] 0.63167295 0.20439811 0.16392893
[7,] 0.66309979 0.18735918 0.14954103
[8,] 0.67539285 0.17845279 0.14615436
[9,] 0.68797690 0.17321485 0.13880825
[10,] 0.69953625 0.16574425 0.13471950
[11,] 0.72156444 0.16383609 0.11459947
[12,] 0.73161044 0.15862833 0.10976122
[13,] 0.73862621 0.15436670 0.10700709
[14,] 0.72366989 0.16651925 0.10981087
[15,] 0.69741765 0.17613134 0.12645101
[16,] 0.73565879 0.14930836 0.11503285
[17,] 0.71150445 0.15686307 0.13163248
[18,] 0.75831888 0.13985422 0.10182690
[19,] 0.73245812 0.14727960 0.12026229
[20,] 0.74698637 0.14557968 0.10743396
[21,] 0.72698149 0.15226343 0.12075508
[22,] 0.11629744 0.84232658 0.04137598
[23,] 0.10308109 0.86133368 0.03558524
[24,] 0.10157861 0.86291686 0.03550454
[25,] 0.10079328 0.86429341 0.03491331
[26,] 0.11365397 0.84687319 0.03947284
[27,] 0.12060269 0.83864432 0.04075298
[28,] 0.11849516 0.84103643 0.04046841
[29,] 0.13941929 0.81317712 0.04740359
[30,] 0.14529510 0.80591935 0.04878556
[31,] 0.16533660 0.78033004 0.05433336
[32,] 0.17628606 0.76437152 0.05934243
[33,] 0.19127064 0.74268905 0.06604031
[34,] 0.18641653 0.74913292 0.06445054
[35,] 0.19173206 0.74317196 0.06509598
[36,] 0.21030396 0.71723893 0.07245711
[37,] 0.23131762 0.68801667 0.08066571
[38,] 0.23195018 0.68731709 0.08073273
[39,] 0.24002253 0.67557395 0.08440353
[40,] 0.23846552 0.67756832 0.08396616
[41,] 0.25472246 0.65297482 0.09230273
[42,] 0.27901078 0.61674919 0.10424003
[43,] 0.27790511 0.61864206 0.10345283
[44,] 0.18167949 0.13058518 0.68773534
[45,] 0.17798295 0.12770224 0.69431481
[46,] 0.11920734 0.08034544 0.80044722
[47,] 0.11145960 0.07417952 0.81436088
[48,] 0.11739738 0.07926588 0.80333674
[49,] 0.11668412 0.07889987 0.80441601
[50,] 0.11877977 0.08053675 0.80068348
[51,] 0.14155474 0.09846595 0.75997931
[52,] 0.16443135 0.11687942 0.71868923
[53,] 0.14422864 0.10024070 0.75553066
[54,] 0.12729069 0.08617904 0.78653027
[55,] 0.12722387 0.08219027 0.79058586
[56,] 0.13560085 0.08686367 0.77753548
[57,] 0.13555491 0.08836465 0.77608044
[58,] 0.13706577 0.09138560 0.77154863
[59,] 0.15042665 0.09933736 0.75023599
[60,] 0.15763545 0.10382862 0.73853593
[61,] 0.16368192 0.10904375 0.72727433
[62,] 0.16825765 0.11262758 0.71911477
[63,] 0.18036661 0.12003546 0.69959792
[64,] 0.19567254 0.13055114 0.67377631
[65,] 0.21439219 0.14254268 0.64306513
[66,] 0.08715453 0.88295648 0.02988899
[67,] 0.08848089 0.88096001 0.03055911
[68,] 0.08645427 0.88430505 0.02924068
[69,] 0.09065097 0.87875719 0.03059184
[70,] 0.08133971 0.89123771 0.02742258
[71,] 0.09584413 0.87192822 0.03222765
[72,] 0.11092123 0.85087567 0.03820310
[73,] 0.11209909 0.84887277 0.03902813
[74,] 0.10818970 0.85401114 0.03779916
[75,] 0.13301014 0.81961055 0.04737931
[76,] 0.13683924 0.81384467 0.04931609
[77,] 0.13626314 0.81485039 0.04888647
[78,] 0.13933557 0.80993644 0.05072799
[79,] 0.15542585 0.78730249 0.05727166
[80,] 0.14473976 0.80199420 0.05326604
[81,] 0.15933194 0.78105112 0.05961695
[82,] 0.16085993 0.77771960 0.06142047
[83,] 0.14977784 0.79345922 0.05676293
[84,] 0.15490113 0.78650311 0.05859576
[85,] 0.15640402 0.78439286 0.05920312
[86,] 0.20650675 0.71323304 0.08026022
[87,] 0.18145480 0.74951980 0.06902540
[88,] 0.73703609 0.14386046 0.11910345
[89,] 0.74003506 0.14207153 0.11789341
[90,] 0.80116671 0.12377021 0.07506308
[91,] 0.80676073 0.12434550 0.06889377
[92,] 0.77975766 0.12748702 0.09275532
[93,] 0.75038509 0.13665745 0.11295747
[94,] 0.72477530 0.14395266 0.13127204
[95,] 0.68916099 0.15414325 0.15669576
[96,] 0.56279542 0.17678054 0.26042403
[97,] 0.50200654 0.17848797 0.31950549
[98,] 0.59675109 0.17206779 0.23118112
[99,] 0.72017167 0.14429854 0.13552979
[100,] 0.65316313 0.16219022 0.18464665
[101,] 0.59988136 0.17195477 0.22816387
[102,] 0.52275531 0.17827169 0.29897301
[103,] 0.51720598 0.17842106 0.30437296
[104,] 0.52498218 0.17880317 0.29621465
[105,] 0.47707065 0.17954689 0.34338246
[106,] 0.41369904 0.17460815 0.41169281
[107,] 0.31367463 0.15229538 0.53402999
[108,] 0.35768951 0.16151866 0.48079183
[109,] 0.33163061 0.15638283 0.51198656
[110,] 0.73960091 0.19089034 0.06950875
[111,] 0.76543339 0.16864606 0.06592054
[112,] 0.71881521 0.21048002 0.07070477
[113,] 0.70979980 0.21919479 0.07100541
[114,] 0.77912254 0.15784356 0.06303390
[115,] 0.80010221 0.13958790 0.06030989
[116,] 0.80091532 0.13791249 0.06117219
[117,] 0.82104652 0.11927401 0.05967947
[118,] 0.81193341 0.11701828 0.07104831
[119,] 0.78061010 0.12894793 0.09044197
[120,] 0.77793417 0.13199866 0.09006717
[121,] 0.80048250 0.12420835 0.07530915
[122,] 0.79988650 0.12384277 0.07627074
[123,] 0.77008120 0.13251231 0.09740650
[124,] 0.78135862 0.12970701 0.08893437
[125,] 0.75227953 0.14252068 0.10519979
[126,] 0.73508606 0.15614796 0.10876599
[127,] 0.72116942 0.16031797 0.11851261
[128,] 0.69732378 0.16949769 0.13317853
[129,] 0.65737611 0.18118377 0.16144012
[130,] 0.66707669 0.18519897 0.14772435
[131,] 0.65172457 0.19409434 0.15418108
[132,] 0.33009791 0.16557643 0.50432565
[133,] 0.34513018 0.16807636 0.48679346
[134,] 0.34382536 0.16713172 0.48904292
[135,] 0.34353438 0.16611501 0.49035061
[136,] 0.31698687 0.15776079 0.52525234
[137,] 0.25988096 0.13909361 0.60102543
[138,] 0.24885132 0.13513171 0.61601697
[139,] 0.20848195 0.11883509 0.67268296
[140,] 0.18097801 0.10641348 0.71260852
[141,] 0.16086899 0.09721563 0.74191537
[142,] 0.16867499 0.10023912 0.73108588
[143,] 0.17051616 0.10035049 0.72913335
[144,] 0.16260654 0.09667386 0.74071960
[145,] 0.16068184 0.09526815 0.74405001
[146,] 0.13679913 0.08602475 0.77717612
[147,] 0.13092360 0.08286750 0.78620890
[148,] 0.12539432 0.07906505 0.79554063
[149,] 0.12308446 0.07763445 0.79928109
[150,] 0.12060186 0.07772703 0.80167111
[151,] 0.12655846 0.08369958 0.78974196
[152,] 0.12562248 0.08308802 0.79128950
[153,] 0.13803499 0.09350853 0.76845648
[154,] 0.12457690 0.83638630 0.03903680
[155,] 0.13857533 0.81944019 0.04198448
[156,] 0.15922659 0.79407403 0.04669938
[157,] 0.18212460 0.76631372 0.05156168
[158,] 0.20974021 0.73294897 0.05731082
[159,] 0.20708132 0.73652176 0.05639692
[160,] 0.25414907 0.68111754 0.06473339
[161,] 0.31442047 0.61197345 0.07360608
[162,] 0.35877186 0.56274151 0.07848662
[163,] 0.40229917 0.51552156 0.08217927
[164,] 0.43887497 0.47612275 0.08500228
[165,] 0.48363667 0.42889612 0.08746721
[166,] 0.45522215 0.45891588 0.08586198
[167,] 0.50585359 0.40696334 0.08718307
[168,] 0.57557601 0.33822583 0.08619816
[169,] 0.60223289 0.31206333 0.08570378
[170,] 0.63960291 0.27699175 0.08340534
[171,] 0.65029768 0.26732362 0.08237870
[172,] 0.67532433 0.24587842 0.07879725
[173,] 0.73507655 0.19098357 0.07393988
[174,] 0.76880457 0.15667407 0.07452136
[175,] 0.76003222 0.16947431 0.07049347
[176,] 0.62560022 0.29396297 0.08043681
[177,] 0.64658111 0.27519109 0.07822780
[178,] 0.64626344 0.27544304 0.07829352
[179,] 0.66394946 0.25980279 0.07624775
[180,] 0.67996769 0.24602229 0.07401002
[181,] 0.73147118 0.20025500 0.06827382
[182,] 0.75857013 0.17625701 0.06517286
[183,] 0.76717200 0.16826076 0.06456723
[184,] 0.77426432 0.16169221 0.06404347
[185,] 0.80373698 0.13565977 0.06060325
[186,] 0.80518916 0.13447402 0.06033682
[187,] 0.79411302 0.14371294 0.06217405
[188,] 0.77089375 0.16377598 0.06533027
[189,] 0.78880563 0.14689928 0.06429509
[190,] 0.79155725 0.13941857 0.06902419
[191,] 0.79070715 0.13473693 0.07455593
[192,] 0.76922280 0.14675389 0.08402331
[193,] 0.77637425 0.14220477 0.08142098
[194,] 0.76436928 0.14527665 0.09035407
[195,] 0.74664828 0.15484509 0.09850663
[196,] 0.71123462 0.16916439 0.11960099
[197,] 0.69873599 0.17903303 0.12223098
[198,] 0.12817758 0.82773107 0.04409135
[199,] 0.12250023 0.83538714 0.04211263
[200,] 0.12001228 0.83839791 0.04158980
[201,] 0.11551494 0.84521163 0.03927342
[202,] 0.11346319 0.84822938 0.03830744
[203,] 0.10992590 0.85346748 0.03660662
[204,] 0.10145701 0.86471023 0.03383276
[205,] 0.09691101 0.87041884 0.03267016
[206,] 0.09314032 0.87595152 0.03090816
[207,] 0.08933258 0.88148935 0.02917807
[208,] 0.08960222 0.88138141 0.02901637
[209,] 0.08487447 0.88709391 0.02803162
[210,] 0.08790979 0.88229766 0.02979255
[211,] 0.08219806 0.89097270 0.02682924
[212,] 0.09349764 0.87751393 0.02898844
[213,] 0.11505652 0.85063492 0.03430857
[214,] 0.13098290 0.83092361 0.03809349
[215,] 0.12179686 0.84100450 0.03719865
[216,] 0.14592286 0.81019968 0.04387747
[217,] 0.18238860 0.76550624 0.05210516
[218,] 0.20867436 0.73297186 0.05835378
[219,] 0.18292691 0.76348763 0.05358546
[220,] 0.47793633 0.43956367 0.08250000
[221,] 0.48838602 0.42939311 0.08222087
[222,] 0.49939626 0.41854159 0.08206215
[223,] 0.51152121 0.40648592 0.08199287
[224,] 0.38140750 0.54077595 0.07781655
[225,] 0.48841809 0.42846120 0.08312071
[226,] 0.53256893 0.38273790 0.08469317
[227,] 0.61700503 0.29621139 0.08678358
[228,] 0.64340563 0.26871432 0.08788006
[229,] 0.67546623 0.23336461 0.09116916
[230,] 0.66102720 0.24514291 0.09382988
[231,] 0.61595888 0.28876132 0.09527980
[232,] 0.56826769 0.33202378 0.09970853
[233,] 0.57182423 0.32699107 0.10118470
[234,] 0.57022069 0.32398504 0.10579427
[235,] 0.58414216 0.30637271 0.10948513
[236,] 0.57693201 0.31006817 0.11299982
[237,] 0.55841837 0.32152166 0.12005996
[238,] 0.55428214 0.31029983 0.13541804
[239,] 0.54189660 0.31091570 0.14718771
[240,] 0.54063954 0.30591957 0.15344089
[241,] 0.53226979 0.30335155 0.16437867
[242,] 0.13825362 0.81072212 0.05102426
[243,] 0.13082769 0.82147739 0.04769492
[244,] 0.12567786 0.82899773 0.04532441
[245,] 0.11858326 0.83920074 0.04221600
[246,] 0.11537116 0.84361878 0.04101006
[247,] 0.10523988 0.85843654 0.03632357
[248,] 0.10238668 0.86263557 0.03497775
[249,] 0.09777133 0.86951819 0.03271048
[250,] 0.09516655 0.87347713 0.03135632
[251,] 0.09822792 0.87022897 0.03154310
[252,] 0.09277850 0.87676727 0.03045423
[253,] 0.09637454 0.87292209 0.03070337
[254,] 0.08893323 0.88210912 0.02895764
[255,] 0.09730386 0.87219650 0.03049964
[256,] 0.12363848 0.83980979 0.03655174
[257,] 0.11986239 0.84460158 0.03553603
[258,] 0.12883672 0.83349363 0.03766966
[259,] 0.13715618 0.82363547 0.03920835
[260,] 0.14901744 0.80911178 0.04187078
[261,] 0.16030202 0.79540713 0.04429085
[262,] 0.16330648 0.79205729 0.04463622
[263,] 0.18605354 0.76456116 0.04938530

dunn_coeff normalized
0.5928260 0.3892389

I also tried performing a switching regression test upon the data, but have so far been unable to do so. This week, however, I will attempt to cluster using 3 cycles, and determining the clusters for each cycle.

Partial Draft:
I am also working upon my partial draft, and expect to have the introduction, literature review, and data interpretation drafted by the due date.

Thesis Update

Thesis Update: Fuzzy Clustering Results:

This week I performed a fuzzy clustering analysis using the “fanny” function from the ‘cluster’ package discussed previously in this blog upon the Compensation of Employee (Chain Volume Measure) and Gross Value Added data from Dataset 5204.0 from the Australian Bureau of Statistic in order to get a feeling on how to perform the final analysis upon the final data set (which was provided by Dr Poon). The results are as follows:

Fuzzy Clustering object of class 'fanny' :
m.ship.expon. 2
objective 86706.06
tolerance 1e-15
iterations 35
converged 1
maxit 500
n 17
Membership coefficients (in %, rounded):
[,1] [,2] [,3] [,4]
[1,] 87 2 6 4
[2,] 88 2 6 4
[3,] 15 35 22 28
[4,] 80 3 10 7
[5,] 10 4 70 16
[6,] 6 3 76 15
[7,] 4 4 14 78
[8,] 72 4 15 9
[9,] 15 5 63 17
[10,] 91 1 4 3
[11,] 7 5 51 37
[12,] 1 95 2 2
[13,] 6 4 67 23
[14,] 4 4 14 79
[15,] 8 15 20 57
[16,] 86 2 7 5
[17,] 62 5 21 12
Fuzzyness coefficients:
dunn_coeff normalized
0.5945847 0.4594462
Closest hard clustering:
[1] 1 1 2 1 3 3 4 1 3 1 3 2 3 4 4 1 1

Silhouette plot information:
cluster neighbor sil_width
10 1 3 0.82938820
1 1 3 0.81032816
2 1 3 0.80477324
16 1 3 0.80118019
4 1 3 0.76509472
8 1 3 0.69297873
17 1 3 0.57146445
3 2 4 0.06730237
12 2 4 -0.17685372
6 3 4 0.61400342
5 3 4 0.60246272
9 3 1 0.54122724
13 3 4 0.49619115
11 3 4 0.26723303
15 4 3 0.52706369
7 4 3 0.41649447
14 4 3 0.38930883
Average silhouette width per cluster:
[1] 0.75360110 -0.05477567 0.50422351 0.44428900
Average silhouette width of total data set:
[1] 0.5305671

136 dissimilarities, summarized :
Min. 1st Qu. Median Mean 3rd Qu. Max.
5182.8 21552.0 55122.0 62729.0 85846.0 177810.0
Metric : euclidean
Number of objects : 17

However, this data, upon discussion with Dr Poon, was deemed incorrect, as I had set the number of clusters for the fuzzy clustering. Therefore, this week, I will attempt to perform this test again without setting the cluster parameter. The fanny function, however, will not work without having the cluster being set, and thusly, I will attempt to use the e1071 package in R, as discussed last week, to perform this test. Additionally, if time permits, this week I will also be using the micEcon package’s switching regression function on the dataset provided by Dr Poon.

I have also started writing up my partial draft, and expect this to be completed by Friday of this week.

Thesis Update:

This week my meeting with Dr Poon highlighted my work for the coming week. I will be attempting to cluster, using the e1071 package (or possibly the clValid package) in R, based upon the data from Dataset 5204.0 from the Australian Bureau of Statistics available at:

http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/5204.02005-06?OpenDocument

In particular, this clustering analysis will be performed upon the Compensation of Employees by current prices according to industry (Table 59), according to Dr Poons instructions, as this would provide preliminary results for part of this project.

. The results will be posted up next week.

Additionally, I have been requested to create folders for each component of work which I am currently undertaking in order to archive and manage sources better in case further work needs to be conducted on this topic. As such, I have created 7 folders, as defined by Dr Poon (one folder for introduction, one for literature review, one for methodology, one for data modeling, one for analysis, one for discussion, one for conclusions).

Additionally, review of papers by Muller, Bernard, and Frantzen (see below for references), have given me insight into why this study needs to be conducted. Drawing upon these papers, it can be seen that a more extensive analysis (especially from Mullers Paper) needs to be conducted in order to reach a further level of understanding in the convergence and divergence of industries, as Muller states that past studies have had contrary results, with some papers finding convergence amongst both manufacturing and service sectors of industries, and others finding no convergence at all (Muller, pp 6-7).

Furthermore, in relation to this, Frantzen argues that while overall in OECD
countries there is a clear trend of convergence, and confirmation of neo-classical models, the estimation of panel data upon a national level, when accounting for country-specific fixed effects, challenges the model, thus creating a level of uncertainty. It has also clearly be stated by Frantzen that “the empirical work on convergence through technological diffusion is still limited”, thus providing further incentive to carry out this study upon Australian data.

Likewise, Bernard’s article states that “the debate over convergence has lost its way…becoming mired in a debate about 2% per year convergence rates and their robustness or lack thereof” (Bernard, pp2). My study currently will aim to provide empirical data calculated based on statistics gathered by the Australian Bureau of Statistics to further give grounds for the argument of “robustness, or lack thereof” (Bernard, pp2).


References:
Muller, G, 2000, A Glimpse on Sectoral Convergence of Productivity Levels, Halle Institute for Economic Research , Discussion Paper Nr 133

Bernard, A.B, Jones, C.I, 1996, Technology and Convergence, The Economic Journal, Vol 106. No 437.

Frantzen, D, 2004, Technological Diffusion and Productivity Convergence: A Study for Manufacturing in the OECD, Southern Economic Journal, 71(2), 352-376

Thesis

The purpose of the previous post is to illustrate the project plan which was created in the first week of this semester. So far, progress has been going according to plan. This plan, however, also provides for contingencies should they arise during the course of the project.

Thesis Update

This week I have outlined a few approaches to use in analysing the data once a data set has been decided upon. Firstly, I will use Fuzzy Clustering, and then Switching Regression, as outlined by Dr Poon, in order to see how I.T. correlates to Industry Structures. A second method I will be attempting, if time permits, would be to perform Switching Regression first, and then perform Fuzzy Clustering, in the method outlined by Hathaway and Bezdek in their 1993 paper titled Switching Regression and Fuzzy Clustering published in IEEE Transactions on Fuzzy Systems, Vol 1, No 3, August 1993,

I will attempt these two methods using firstly the "cor" and "fanny" functions for R which I have in literature on the internet (I have not tested these two functions as of yet, but will do so this week). However, this may only provide limited support for regression and clustering, and I may have to write the scripts to perform these functions in R, as I wish to do both Fuzzy C-Means, and Fuzzy C-Regression for the Fuzzy Clustering part of this project.

Additionally, we (Dr Poon's thesis group), have attempted to compile a data set in the past week, which we will be refining and adding to in the coming week.