Thesis Update

I have completed the final version of the thesis and have also had it proof read. The final copy of the thesis includes relevant results which were obtained throughout the duration of the thesis.

The problem which this paper attempts to analyse is that of the impact of IT upon industry structure. This problem is of particular importance to the Australian Economy as a whole, as Australia becomes more and more IT dependent. This problem has not been addressed sufficiently in previously literature, and as such, this paper will contribute significantly to the work of researchers and policymakers in this field of study.

This paper has analysed the changes which IT has brought about upon organizations, and on an aggregate level, industry. This paper has employed generally accepted methods in order to carry out this study, using regressions, structural breakpoint tests, and fuzzy c-means clustering in order to achieve this goal.

This paper has proven Langdons’ (2003) conclusion that the Software aspect of IT is rapidly becoming more dominant than the hardware aspect of IT, and thusly, this confirmation will allow decision makers to design policies which augment and support this finding in order to promote additional growth in the Australian Economy.

The results obtained for the study indicate that there is a shift in industry structure in the form of coalescence. Based on this, policies have been recommended in order to take advantage of the changes which IT has brought about. Additionally, the resulting structural change may be of significance in predicting the future structure of industry.

In summation, the author of this paper wishes to thank Dr Simon Poon for his ongoing support and help throughout the entire process. Without his support and advice, this paper would not have been possible. It is his patience and understanding that has been of the greatest help to me in the writing of this paper.

Additionally, this paper wishes to thank Dr Rafael Calvo for the administrative work which was performed during the execution of this paper.

This paper also wishes to acknowledge the Australian Bureau of Statistics, MIT GmBH and the Open R Archive for their advice and assistance in the provision of resources used in the provision of this project.

Thesis Update

Have finished the write up of the thesis based on the latest results which were discussed with Dr Poon, which were obtained for two cycles which were based on a structural breakpoint test.

The results indicate that IT has had an effect upon the shift in industry, as can be seen from the regression results obtained for the regression with dummy variables. The regression results indicate that of the IT aspects (Hardware and Software), industry has been increasingly dependent upon Software, and this has had the most effect upon Value added. Comparing this with the clustering results also indicates that Software has had the most impact, since there is coaelscence across the cycles.

One should note that the effects of IT, as discussed by David (1990), and various papers by Brynjolfsson, were able to be illustrated by the results obtained, as the time-lag effect was clearly shown, and disproves the productivity paradox.



Reference:

David, P, 1990, "The Dynamo and The Computer: An historical perspective on the modern productivity paradox", The American Economic Review, Vol. 80, No. 2, Papers and Proceedings of the Hundred and Second Annual Meeting of the American Economic Association. (May, 1990), pp. 355-361.

Thesis Update

Have completed all the major components of the analysis required, and am now in the process of writing up.

The analysis components included performing a Quandt Test, a Chow Test, a Switching Regression, and a cluster analysis, using both fixed and industry effects to account for the different industries and their effects upon the overall function which this thesis project uses. The Chow test was selected in order to determine breakpoints, which varied from the original thesis scope in that the original breakpoints for the cycles to perform the analysis were set arbitrarily in order to evenly distribute the data across the cycles. Furthermore, the clustering analysis, when performed upon 19090-1995, 1996-1999, and 2000-2006, reveals that there are originally 3 clusters, which diverges into 5 during 1996-1999, and then settles down to 4 clusters. The clustering analysis, done via setting the fuzziness exponenet m to 2, using the partition coefficient method 1/c, therefore indicates that there is a divergence, and then convergence, across the industries during the cycles. The divergence between 1990-1995 and 1996-1999 can be explained by the fact that the middle cycle, which was determined by a Chow test, can be called a transitional period (such as that in Dr Poon's paper), and also reveals intra- and inter- differences between the clusters.

The other major component, that of switching regression, indicates that the model used is an endogenous model, and not exogenous, as the rhos for both cycles of the switching regression significantly differ from 0, which indicates the aforementioned (Bertschek et al, 2005).

Thesis Update

The fuzzy clustering has been completed, with results showing that there has been consistently 4 clusters from 1990 - 2006. This is in line with Dr Poons findings in 2002, and thus, validates the results obtained by Dr Poon in 2002.

Additionally, the switching regression part of the analysis is near completion, with results obtained for switching between Cycle 1 (1990-95), Cycle 2 (1996-2000), and Cycle 3 (2001-2006). However, a problem was encountered for the switching regression between cycle2 and cycle3, with a null value produced after the 15th iteration of the switching regression function.

Using another method, I have also performed linear regressions on all three cycles, and performed the Goldfeld-Quandt test, as well as compared the coefficient of IT in each cycle, in order to determine the effect of IT upon VA as a measure of impact on industry. I have attempted to add fixed effects of industry into the model, but so far, the model rejects fixed effects, which I shall attempt to repair before concluding.

Thesis Update

This week in my meeting with Dr Poon, we encountered a problem about the validity of the use of switching regression in the analysis of the impact of IT upon industry structure, as industries, based on cluster analysis, do not elect to move from cluster to cluster. It was therefore decided in the meeting that there were several options to pursue for this case, listed as below:

1) Continue with Switching Regression
2) Perform a Chou Test
3) Implement Random Coefficient Analysis

Furthermore, a new, raw data set has been compiled for a final fuzzy clustering analysis, the results of which will be posted in the next two days. Additionally, the switching regression problem, if implemented, will use modified equations from Berstchek's 2005 paper analysing organisational change using switching regression. By following this, it is hoped that the results of the switching regression will show that the use of IT leads to a higher productivity, and hence, a change in industry structure caused by IT, which would be found be analysis the outcomes of the switching regression equations, which uses a TRUE/FALSE vector to determine which strategy is the best for each data point.

Thesis Update

For the last two weeks, I have been attempting to perform a switching regression analysis upon the data provided by Dr Poon. To this end, I have read various papers by Bezdek and Hathaway, Chau, as well as examples of switching regression. On Wednesday, I located an online tutorial which explained how to perform this analysis in R, and as such, I will attempt to do so over the weekend, and post the results up.

Additionally, fuzzy clustering is now being performed by data engine. The final results of this analysis will be posted up later. However, preliminary indications match that of Dr Poon's findings now, as compared to the ambiguous results derived from R.

Thesis Update

Last Week I completed the partial draft, which consisted of introduction, literary review, methodology and results. However, these parts were not totally completed, and require editing.

Additionally, I performed the fuzzy clustering analysis over three cycles using R, as stated before, and discussed the results with Dr Poon in our weekly meeting. It was decided that the use of R and the determination of clusters in this manner was not suitable, and as such, we will be switching to Data Engine (a fuzzy clustering tool) for this part of the project.

I will also be taking a look at the use and development of switching regression, and will also implement this in R on Dr Poons given data for now. I will also need to develop a presentation for the overall progress for this wednesdays meeting with Dr Poon, which I am currently in the process of doing.

I will post results up as soon as they become available.

Thesis Update - Fuzzy Clustering

This week I performed a completed fuzzy clustering set upon data provided by Dr Poon. I firstly checked which number of clusters was best, using the clValid function, and then performed a fuzzy clustering analysis upon the data using this number of clusters. The results are as follows:

Clustering Determination using clValid:

Clustering Determination using clValid:

Cluster sizes:
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

Validation Measures:
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

fanny Connectivity 17.6429 15.6361 26.2500 38.6222 44.8575 47.1246 52.2718 86.2238 80.3901 83.1063 75.4885 99.8266 79.9262 88.6492 103.7821 141.2881 94.4425 108.0456 88.9873 109.7246 104.8306 128.4329 116.3290 96.4512 118.6901 129.3504 100.5310 137.2397 87.9849 120.8452 103.8091 131.6369 124.2206 137.7921 150.8726 140.9762 169.8508 156.1194 165.8202 179.5583 188.7667 184.4540 196.6766 194.9726 204.8901 192.5278 222.9175 219.1512 234.1079 239.3738 250.2071 255.2821 244.8940 243.3845 242.0552 251.2544 245.1659 252.9909 263.9631 285.9647 300.1361 303.7901 304.6492 310.1956 342.0210 338.8845 350.2933 360.7131 360.5933 361.4758 347.8706 357.0214 366.3710 361.6758 375.6111 355.5508 363.5639 369.8798 376.3706 380.6302 384.4921 391.0083 396.2425 400.8155 409.2960 411.4611 419.0857 422.3060 419.7639 424.0175 428.4333 419.9694 428.1302 429.0349 434.1635 423.6893 429.7310 432.2143 441.9460 451.9274 464.2226 467.7210 470.2028 474.5651 476.5817 463.7944 466.9492 471.0075 476.4484 470.1337 476.7099 477.4099 486.1218 476.2429 482.7587 487.9103 489.0440 491.6155 495.2456 497.9123 496.1159 496.8548 501.0881 497.7444 497.8548 503.6226 506.5488 509.6853 513.5020
Dunn 0.0122 0.0248 0.0280 0.0271 0.0221 0.0241 0.0250 0.0107 0.0169 0.0227 0.0201 0.0280 0.0266 0.0424 0.0180 0.0222 0.0264 0.0464 0.0521 0.0497 0.0470 0.0205 0.0392 0.0394 0.0518 0.0359 0.0389 0.0251 0.0389 0.0234 0.0389 0.0374 0.0241 0.0433 0.0209 0.0469 0.0466 0.0469 0.0433 0.0209 0.0469 0.0277 0.0389 0.0414 0.0456 0.0569 0.0418 0.0570 0.0365 0.0277 0.0363 0.0299 0.0299 0.0299 0.0639 0.0561 0.0560 0.0757 0.0547 0.0889 0.0375 0.0375 0.0375 0.0375 0.0525 0.0657 0.0298 0.0263 0.0263 0.0263 0.0525 0.0525 0.0531 0.0531 0.0531 0.0500 0.0638 0.0510 0.0633 0.0906 0.0794 0.0626 0.0626 0.0626 0.0626 0.0626 0.0309 0.0248 0.0275 0.0275 0.0257 0.0257 0.0257 0.0257 0.0259 0.0420 0.0444 0.0457 0.0457 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0271 0.0277 0.0313 0.0313 0.0357 0.0393 0.0528 0.0528 0.0565 0.0589 0.0589 0.0682 0.0682 0.0427 0.0427 0.0427 0.0427
Silhouette 0.4849 0.5735 0.4836 0.4278 0.3433 0.3887 0.3835 0.2603 0.3265 0.2677 0.2010 0.1594 0.1656 0.2525 0.2271 0.2583 0.3578 0.3457 0.3973 0.3630 0.3293 0.2901 0.3419 0.3399 0.2816 0.2861 0.3980 0.2971 0.4109 0.3093 0.3387 0.3537 0.2285 0.3295 0.3190 0.3041 0.3226 0.3264 0.3200 0.3461 0.3114 0.3143 0.2946 0.2865 0.2761 0.2809 0.2621 0.3140 0.3073 0.3025 0.2784 0.2374 0.3115 0.2992 0.3334 0.2961 0.3200 0.3064 0.3040 0.3350 0.3667 0.3669 0.3589 0.3601 0.2290 0.2255 0.2234 0.2512 0.2477 0.2503 0.2861 0.2897 0.3071 0.3324 0.3213 0.3611 0.3522 0.3544 0.3566 0.3589 0.3557 0.3503 0.3460 0.3425 0.3363 0.3387 0.1350 0.1511 0.1839 0.1924 0.1592 0.1608 0.1769 0.1797 0.1723 0.2039 0.2042 0.2087 0.2017 0.1973 0.1897 0.1964 0.1876 0.1842 0.1806 0.2261 0.2210 0.2245 0.2239 0.2377 0.2269 0.2208 0.2157 0.2408 0.2396 0.2453 0.2535 0.2478 0.2572 0.2528 0.2627 0.2542 0.2561 0.2758 0.2765 0.2736 0.2681 0.2727 0.2608

Optimal Scores:

Score Method Clusters
Connectivity 15.6361 fanny 3
Dunn 0.0906 fanny 81
Silhouette 0.5735 fanny 3

For Connectivity and Silhouette (3)

Fuzzy Clustering Results, using 3 clusters:

Fuzzy Clustering object of class 'fanny' :
m.ship.expon. 2
objective 1944560
tolerance 1e-15
iterations 32
converged 1
maxit 500
n 263
Membership coefficients (in %, rounded):
[,1] [,2] [,3]
[1,] 64 20 15
[2,] 66 19 15
[3,] 69 17 14
[4,] 70 17 13
[5,] 58 23 19
[6,] 63 20 16
[7,] 66 19 15
[8,] 68 18 15
[9,] 69 17 14
[10,] 70 17 13
[11,] 72 16 11
[12,] 73 16 11
[13,] 74 15 11
[14,] 72 17 11
[15,] 70 18 13
[16,] 74 15 12
[17,] 71 16 13
[18,] 76 14 10
[19,] 73 15 12
[20,] 75 15 11
[21,] 73 15 12
[22,] 12 84 4
[23,] 10 86 4
[24,] 10 86 4
[25,] 10 86 3
[26,] 11 85 4
[27,] 12 84 4
[28,] 12 84 4
[29,] 14 81 5
[30,] 15 81 5
[31,] 17 78 5
[32,] 18 76 6
[33,] 19 74 7
[34,] 19 75 6
[35,] 19 74 7
[36,] 21 72 7
[37,] 23 69 8
[38,] 23 69 8
[39,] 24 68 8
[40,] 24 68 8
[41,] 25 65 9
[42,] 28 62 10
[43,] 28 62 10
[44,] 18 13 69
[45,] 18 13 69
[46,] 12 8 80
[47,] 11 7 81
[48,] 12 8 80
[49,] 12 8 80
[50,] 12 8 80
[51,] 14 10 76
[52,] 16 12 72
[53,] 14 10 76
[54,] 13 9 79
[55,] 13 8 79
[56,] 14 9 78
[57,] 14 9 78
[58,] 14 9 77
[59,] 15 10 75
[60,] 16 10 74
[61,] 16 11 73
[62,] 17 11 72
[63,] 18 12 70
[64,] 20 13 67
[65,] 21 14 64
[66,] 9 88 3
[67,] 9 88 3
[68,] 9 88 3
[69,] 9 88 3
[70,] 8 89 3
[71,] 10 87 3
[72,] 11 85 4
[73,] 11 85 4
[74,] 11 85 4
[75,] 13 82 5
[76,] 14 81 5
[77,] 14 81 5
[78,] 14 81 5
[79,] 16 79 6
[80,] 14 80 5
[81,] 16 78 6
[82,] 16 78 6
[83,] 15 79 6
[84,] 15 79 6
[85,] 16 78 6
[86,] 21 71 8
[87,] 18 75 7
[88,] 74 14 12
[89,] 74 14 12
[90,] 80 12 8
[91,] 81 12 7
[92,] 78 13 9
[93,] 75 14 11
[94,] 72 14 13
[95,] 69 15 16
[96,] 56 18 26
[97,] 50 18 32
[98,] 60 17 23
[99,] 72 14 14
[100,] 65 16 18
[101,] 60 17 23
[102,] 52 18 30
[103,] 52 18 30
[104,] 52 18 30
[105,] 48 18 34
[106,] 41 17 41
[107,] 31 15 53
[108,] 36 16 48
[109,] 33 16 51
[110,] 74 19 7
[111,] 77 17 7
[112,] 72 21 7
[113,] 71 22 7
[114,] 78 16 6
[115,] 80 14 6
[116,] 80 14 6
[117,] 82 12 6
[118,] 81 12 7
[119,] 78 13 9
[120,] 78 13 9
[121,] 80 12 8
[122,] 80 12 8
[123,] 77 13 10
[124,] 78 13 9
[125,] 75 14 11
[126,] 74 16 11
[127,] 72 16 12
[128,] 70 17 13
[129,] 66 18 16
[130,] 67 19 15
[131,] 65 19 15
[132,] 33 17 50
[133,] 35 17 49
[134,] 34 17 49
[135,] 34 17 49
[136,] 32 16 53
[137,] 26 14 60
[138,] 25 14 62
[139,] 21 12 67
[140,] 18 11 71
[141,] 16 10 74
[142,] 17 10 73
[143,] 17 10 73
[144,] 16 10 74
[145,] 16 10 74
[146,] 14 9 78
[147,] 13 8 79
[148,] 13 8 80
[149,] 12 8 80
[150,] 12 8 80
[151,] 13 8 79
[152,] 13 8 79
[153,] 14 9 77
[154,] 12 84 4
[155,] 14 82 4
[156,] 16 79 5
[157,] 18 77 5
[158,] 21 73 6
[159,] 21 74 6
[160,] 25 68 6
[161,] 31 61 7
[162,] 36 56 8
[163,] 40 52 8
[164,] 44 48 9
[165,] 48 43 9
[166,] 46 46 9
[167,] 51 41 9
[168,] 58 34 9
[169,] 60 31 9
[170,] 64 28 8
[171,] 65 27 8
[172,] 68 25 8
[173,] 74 19 7
[174,] 77 16 7
[175,] 76 17 7
[176,] 63 29 8
[177,] 65 28 8
[178,] 65 28 8
[179,] 66 26 8
[180,] 68 25 7
[181,] 73 20 7
[182,] 76 18 7
[183,] 77 17 6
[184,] 77 16 6
[185,] 80 14 6
[186,] 81 13 6
[187,] 79 14 6
[188,] 77 16 7
[189,] 79 15 6
[190,] 79 14 7
[191,] 79 13 7
[192,] 77 15 8
[193,] 78 14 8
[194,] 76 15 9
[195,] 75 15 10
[196,] 71 17 12
[197,] 70 18 12
[198,] 13 83 4
[199,] 12 84 4
[200,] 12 84 4
[201,] 12 85 4
[202,] 11 85 4
[203,] 11 85 4
[204,] 10 86 3
[205,] 10 87 3
[206,] 9 88 3
[207,] 9 88 3
[208,] 9 88 3
[209,] 8 89 3
[210,] 9 88 3
[211,] 8 89 3
[212,] 9 88 3
[213,] 12 85 3
[214,] 13 83 4
[215,] 12 84 4
[216,] 15 81 4
[217,] 18 77 5
[218,] 21 73 6
[219,] 18 76 5
[220,] 48 44 8
[221,] 49 43 8
[222,] 50 42 8
[223,] 51 41 8
[224,] 38 54 8
[225,] 49 43 8
[226,] 53 38 8
[227,] 62 30 9
[228,] 64 27 9
[229,] 68 23 9
[230,] 66 25 9
[231,] 62 29 10
[232,] 57 33 10
[233,] 57 33 10
[234,] 57 32 11
[235,] 58 31 11
[236,] 58 31 11
[237,] 56 32 12
[238,] 55 31 14
[239,] 54 31 15
[240,] 54 31 15
[241,] 53 30 16
[242,] 14 81 5
[243,] 13 82 5
[244,] 13 83 5
[245,] 12 84 4
[246,] 12 84 4
[247,] 11 86 4
[248,] 10 86 3
[249,] 10 87 3
[250,] 10 87 3
[251,] 10 87 3
[252,] 9 88 3
[253,] 10 87 3
[254,] 9 88 3
[255,] 10 87 3
[256,] 12 84 4
[257,] 12 84 4
[258,] 13 83 4
[259,] 14 82 4
[260,] 15 81 4
[261,] 16 80 4
[262,] 16 79 4
[263,] 19 76 5
Fuzzyness coefficients:
dunn_coeff normalized
0.5928260 0.3892389
Closest hard clustering:
[1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[60] 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 1 1 1 1 1 1 1 1 1
[119] 1 1 1 1 1 1 1 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 1 2 1 1 1 1 1 1 1 1 1 1 1
[178] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1
[237] 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

Available components:
[1] "membership" "coeff" "memb.exp" "clustering" "k.crisp" "objective" "convergence" "diss"
[9] "call" "silinfo" "data"

[,1] [,2] [,3]
[1,] 0.64308606 0.20316617 0.15374776
[2,] 0.65740785 0.19464854 0.14794361
[3,] 0.68964138 0.17457229 0.13578633
[4,] 0.70191629 0.17154831 0.12653540
[5,] 0.58358282 0.22519072 0.19122646
[6,] 0.63167295 0.20439811 0.16392893
[7,] 0.66309979 0.18735918 0.14954103
[8,] 0.67539285 0.17845279 0.14615436
[9,] 0.68797690 0.17321485 0.13880825
[10,] 0.69953625 0.16574425 0.13471950
[11,] 0.72156444 0.16383609 0.11459947
[12,] 0.73161044 0.15862833 0.10976122
[13,] 0.73862621 0.15436670 0.10700709
[14,] 0.72366989 0.16651925 0.10981087
[15,] 0.69741765 0.17613134 0.12645101
[16,] 0.73565879 0.14930836 0.11503285
[17,] 0.71150445 0.15686307 0.13163248
[18,] 0.75831888 0.13985422 0.10182690
[19,] 0.73245812 0.14727960 0.12026229
[20,] 0.74698637 0.14557968 0.10743396
[21,] 0.72698149 0.15226343 0.12075508
[22,] 0.11629744 0.84232658 0.04137598
[23,] 0.10308109 0.86133368 0.03558524
[24,] 0.10157861 0.86291686 0.03550454
[25,] 0.10079328 0.86429341 0.03491331
[26,] 0.11365397 0.84687319 0.03947284
[27,] 0.12060269 0.83864432 0.04075298
[28,] 0.11849516 0.84103643 0.04046841
[29,] 0.13941929 0.81317712 0.04740359
[30,] 0.14529510 0.80591935 0.04878556
[31,] 0.16533660 0.78033004 0.05433336
[32,] 0.17628606 0.76437152 0.05934243
[33,] 0.19127064 0.74268905 0.06604031
[34,] 0.18641653 0.74913292 0.06445054
[35,] 0.19173206 0.74317196 0.06509598
[36,] 0.21030396 0.71723893 0.07245711
[37,] 0.23131762 0.68801667 0.08066571
[38,] 0.23195018 0.68731709 0.08073273
[39,] 0.24002253 0.67557395 0.08440353
[40,] 0.23846552 0.67756832 0.08396616
[41,] 0.25472246 0.65297482 0.09230273
[42,] 0.27901078 0.61674919 0.10424003
[43,] 0.27790511 0.61864206 0.10345283
[44,] 0.18167949 0.13058518 0.68773534
[45,] 0.17798295 0.12770224 0.69431481
[46,] 0.11920734 0.08034544 0.80044722
[47,] 0.11145960 0.07417952 0.81436088
[48,] 0.11739738 0.07926588 0.80333674
[49,] 0.11668412 0.07889987 0.80441601
[50,] 0.11877977 0.08053675 0.80068348
[51,] 0.14155474 0.09846595 0.75997931
[52,] 0.16443135 0.11687942 0.71868923
[53,] 0.14422864 0.10024070 0.75553066
[54,] 0.12729069 0.08617904 0.78653027
[55,] 0.12722387 0.08219027 0.79058586
[56,] 0.13560085 0.08686367 0.77753548
[57,] 0.13555491 0.08836465 0.77608044
[58,] 0.13706577 0.09138560 0.77154863
[59,] 0.15042665 0.09933736 0.75023599
[60,] 0.15763545 0.10382862 0.73853593
[61,] 0.16368192 0.10904375 0.72727433
[62,] 0.16825765 0.11262758 0.71911477
[63,] 0.18036661 0.12003546 0.69959792
[64,] 0.19567254 0.13055114 0.67377631
[65,] 0.21439219 0.14254268 0.64306513
[66,] 0.08715453 0.88295648 0.02988899
[67,] 0.08848089 0.88096001 0.03055911
[68,] 0.08645427 0.88430505 0.02924068
[69,] 0.09065097 0.87875719 0.03059184
[70,] 0.08133971 0.89123771 0.02742258
[71,] 0.09584413 0.87192822 0.03222765
[72,] 0.11092123 0.85087567 0.03820310
[73,] 0.11209909 0.84887277 0.03902813
[74,] 0.10818970 0.85401114 0.03779916
[75,] 0.13301014 0.81961055 0.04737931
[76,] 0.13683924 0.81384467 0.04931609
[77,] 0.13626314 0.81485039 0.04888647
[78,] 0.13933557 0.80993644 0.05072799
[79,] 0.15542585 0.78730249 0.05727166
[80,] 0.14473976 0.80199420 0.05326604
[81,] 0.15933194 0.78105112 0.05961695
[82,] 0.16085993 0.77771960 0.06142047
[83,] 0.14977784 0.79345922 0.05676293
[84,] 0.15490113 0.78650311 0.05859576
[85,] 0.15640402 0.78439286 0.05920312
[86,] 0.20650675 0.71323304 0.08026022
[87,] 0.18145480 0.74951980 0.06902540
[88,] 0.73703609 0.14386046 0.11910345
[89,] 0.74003506 0.14207153 0.11789341
[90,] 0.80116671 0.12377021 0.07506308
[91,] 0.80676073 0.12434550 0.06889377
[92,] 0.77975766 0.12748702 0.09275532
[93,] 0.75038509 0.13665745 0.11295747
[94,] 0.72477530 0.14395266 0.13127204
[95,] 0.68916099 0.15414325 0.15669576
[96,] 0.56279542 0.17678054 0.26042403
[97,] 0.50200654 0.17848797 0.31950549
[98,] 0.59675109 0.17206779 0.23118112
[99,] 0.72017167 0.14429854 0.13552979
[100,] 0.65316313 0.16219022 0.18464665
[101,] 0.59988136 0.17195477 0.22816387
[102,] 0.52275531 0.17827169 0.29897301
[103,] 0.51720598 0.17842106 0.30437296
[104,] 0.52498218 0.17880317 0.29621465
[105,] 0.47707065 0.17954689 0.34338246
[106,] 0.41369904 0.17460815 0.41169281
[107,] 0.31367463 0.15229538 0.53402999
[108,] 0.35768951 0.16151866 0.48079183
[109,] 0.33163061 0.15638283 0.51198656
[110,] 0.73960091 0.19089034 0.06950875
[111,] 0.76543339 0.16864606 0.06592054
[112,] 0.71881521 0.21048002 0.07070477
[113,] 0.70979980 0.21919479 0.07100541
[114,] 0.77912254 0.15784356 0.06303390
[115,] 0.80010221 0.13958790 0.06030989
[116,] 0.80091532 0.13791249 0.06117219
[117,] 0.82104652 0.11927401 0.05967947
[118,] 0.81193341 0.11701828 0.07104831
[119,] 0.78061010 0.12894793 0.09044197
[120,] 0.77793417 0.13199866 0.09006717
[121,] 0.80048250 0.12420835 0.07530915
[122,] 0.79988650 0.12384277 0.07627074
[123,] 0.77008120 0.13251231 0.09740650
[124,] 0.78135862 0.12970701 0.08893437
[125,] 0.75227953 0.14252068 0.10519979
[126,] 0.73508606 0.15614796 0.10876599
[127,] 0.72116942 0.16031797 0.11851261
[128,] 0.69732378 0.16949769 0.13317853
[129,] 0.65737611 0.18118377 0.16144012
[130,] 0.66707669 0.18519897 0.14772435
[131,] 0.65172457 0.19409434 0.15418108
[132,] 0.33009791 0.16557643 0.50432565
[133,] 0.34513018 0.16807636 0.48679346
[134,] 0.34382536 0.16713172 0.48904292
[135,] 0.34353438 0.16611501 0.49035061
[136,] 0.31698687 0.15776079 0.52525234
[137,] 0.25988096 0.13909361 0.60102543
[138,] 0.24885132 0.13513171 0.61601697
[139,] 0.20848195 0.11883509 0.67268296
[140,] 0.18097801 0.10641348 0.71260852
[141,] 0.16086899 0.09721563 0.74191537
[142,] 0.16867499 0.10023912 0.73108588
[143,] 0.17051616 0.10035049 0.72913335
[144,] 0.16260654 0.09667386 0.74071960
[145,] 0.16068184 0.09526815 0.74405001
[146,] 0.13679913 0.08602475 0.77717612
[147,] 0.13092360 0.08286750 0.78620890
[148,] 0.12539432 0.07906505 0.79554063
[149,] 0.12308446 0.07763445 0.79928109
[150,] 0.12060186 0.07772703 0.80167111
[151,] 0.12655846 0.08369958 0.78974196
[152,] 0.12562248 0.08308802 0.79128950
[153,] 0.13803499 0.09350853 0.76845648
[154,] 0.12457690 0.83638630 0.03903680
[155,] 0.13857533 0.81944019 0.04198448
[156,] 0.15922659 0.79407403 0.04669938
[157,] 0.18212460 0.76631372 0.05156168
[158,] 0.20974021 0.73294897 0.05731082
[159,] 0.20708132 0.73652176 0.05639692
[160,] 0.25414907 0.68111754 0.06473339
[161,] 0.31442047 0.61197345 0.07360608
[162,] 0.35877186 0.56274151 0.07848662
[163,] 0.40229917 0.51552156 0.08217927
[164,] 0.43887497 0.47612275 0.08500228
[165,] 0.48363667 0.42889612 0.08746721
[166,] 0.45522215 0.45891588 0.08586198
[167,] 0.50585359 0.40696334 0.08718307
[168,] 0.57557601 0.33822583 0.08619816
[169,] 0.60223289 0.31206333 0.08570378
[170,] 0.63960291 0.27699175 0.08340534
[171,] 0.65029768 0.26732362 0.08237870
[172,] 0.67532433 0.24587842 0.07879725
[173,] 0.73507655 0.19098357 0.07393988
[174,] 0.76880457 0.15667407 0.07452136
[175,] 0.76003222 0.16947431 0.07049347
[176,] 0.62560022 0.29396297 0.08043681
[177,] 0.64658111 0.27519109 0.07822780
[178,] 0.64626344 0.27544304 0.07829352
[179,] 0.66394946 0.25980279 0.07624775
[180,] 0.67996769 0.24602229 0.07401002
[181,] 0.73147118 0.20025500 0.06827382
[182,] 0.75857013 0.17625701 0.06517286
[183,] 0.76717200 0.16826076 0.06456723
[184,] 0.77426432 0.16169221 0.06404347
[185,] 0.80373698 0.13565977 0.06060325
[186,] 0.80518916 0.13447402 0.06033682
[187,] 0.79411302 0.14371294 0.06217405
[188,] 0.77089375 0.16377598 0.06533027
[189,] 0.78880563 0.14689928 0.06429509
[190,] 0.79155725 0.13941857 0.06902419
[191,] 0.79070715 0.13473693 0.07455593
[192,] 0.76922280 0.14675389 0.08402331
[193,] 0.77637425 0.14220477 0.08142098
[194,] 0.76436928 0.14527665 0.09035407
[195,] 0.74664828 0.15484509 0.09850663
[196,] 0.71123462 0.16916439 0.11960099
[197,] 0.69873599 0.17903303 0.12223098
[198,] 0.12817758 0.82773107 0.04409135
[199,] 0.12250023 0.83538714 0.04211263
[200,] 0.12001228 0.83839791 0.04158980
[201,] 0.11551494 0.84521163 0.03927342
[202,] 0.11346319 0.84822938 0.03830744
[203,] 0.10992590 0.85346748 0.03660662
[204,] 0.10145701 0.86471023 0.03383276
[205,] 0.09691101 0.87041884 0.03267016
[206,] 0.09314032 0.87595152 0.03090816
[207,] 0.08933258 0.88148935 0.02917807
[208,] 0.08960222 0.88138141 0.02901637
[209,] 0.08487447 0.88709391 0.02803162
[210,] 0.08790979 0.88229766 0.02979255
[211,] 0.08219806 0.89097270 0.02682924
[212,] 0.09349764 0.87751393 0.02898844
[213,] 0.11505652 0.85063492 0.03430857
[214,] 0.13098290 0.83092361 0.03809349
[215,] 0.12179686 0.84100450 0.03719865
[216,] 0.14592286 0.81019968 0.04387747
[217,] 0.18238860 0.76550624 0.05210516
[218,] 0.20867436 0.73297186 0.05835378
[219,] 0.18292691 0.76348763 0.05358546
[220,] 0.47793633 0.43956367 0.08250000
[221,] 0.48838602 0.42939311 0.08222087
[222,] 0.49939626 0.41854159 0.08206215
[223,] 0.51152121 0.40648592 0.08199287
[224,] 0.38140750 0.54077595 0.07781655
[225,] 0.48841809 0.42846120 0.08312071
[226,] 0.53256893 0.38273790 0.08469317
[227,] 0.61700503 0.29621139 0.08678358
[228,] 0.64340563 0.26871432 0.08788006
[229,] 0.67546623 0.23336461 0.09116916
[230,] 0.66102720 0.24514291 0.09382988
[231,] 0.61595888 0.28876132 0.09527980
[232,] 0.56826769 0.33202378 0.09970853
[233,] 0.57182423 0.32699107 0.10118470
[234,] 0.57022069 0.32398504 0.10579427
[235,] 0.58414216 0.30637271 0.10948513
[236,] 0.57693201 0.31006817 0.11299982
[237,] 0.55841837 0.32152166 0.12005996
[238,] 0.55428214 0.31029983 0.13541804
[239,] 0.54189660 0.31091570 0.14718771
[240,] 0.54063954 0.30591957 0.15344089
[241,] 0.53226979 0.30335155 0.16437867
[242,] 0.13825362 0.81072212 0.05102426
[243,] 0.13082769 0.82147739 0.04769492
[244,] 0.12567786 0.82899773 0.04532441
[245,] 0.11858326 0.83920074 0.04221600
[246,] 0.11537116 0.84361878 0.04101006
[247,] 0.10523988 0.85843654 0.03632357
[248,] 0.10238668 0.86263557 0.03497775
[249,] 0.09777133 0.86951819 0.03271048
[250,] 0.09516655 0.87347713 0.03135632
[251,] 0.09822792 0.87022897 0.03154310
[252,] 0.09277850 0.87676727 0.03045423
[253,] 0.09637454 0.87292209 0.03070337
[254,] 0.08893323 0.88210912 0.02895764
[255,] 0.09730386 0.87219650 0.03049964
[256,] 0.12363848 0.83980979 0.03655174
[257,] 0.11986239 0.84460158 0.03553603
[258,] 0.12883672 0.83349363 0.03766966
[259,] 0.13715618 0.82363547 0.03920835
[260,] 0.14901744 0.80911178 0.04187078
[261,] 0.16030202 0.79540713 0.04429085
[262,] 0.16330648 0.79205729 0.04463622
[263,] 0.18605354 0.76456116 0.04938530

dunn_coeff normalized
0.5928260 0.3892389

I also tried performing a switching regression test upon the data, but have so far been unable to do so. This week, however, I will attempt to cluster using 3 cycles, and determining the clusters for each cycle.

Partial Draft:
I am also working upon my partial draft, and expect to have the introduction, literature review, and data interpretation drafted by the due date.

Thesis Update

Thesis Update: Fuzzy Clustering Results:

This week I performed a fuzzy clustering analysis using the “fanny” function from the ‘cluster’ package discussed previously in this blog upon the Compensation of Employee (Chain Volume Measure) and Gross Value Added data from Dataset 5204.0 from the Australian Bureau of Statistic in order to get a feeling on how to perform the final analysis upon the final data set (which was provided by Dr Poon). The results are as follows:

Fuzzy Clustering object of class 'fanny' :
m.ship.expon. 2
objective 86706.06
tolerance 1e-15
iterations 35
converged 1
maxit 500
n 17
Membership coefficients (in %, rounded):
[,1] [,2] [,3] [,4]
[1,] 87 2 6 4
[2,] 88 2 6 4
[3,] 15 35 22 28
[4,] 80 3 10 7
[5,] 10 4 70 16
[6,] 6 3 76 15
[7,] 4 4 14 78
[8,] 72 4 15 9
[9,] 15 5 63 17
[10,] 91 1 4 3
[11,] 7 5 51 37
[12,] 1 95 2 2
[13,] 6 4 67 23
[14,] 4 4 14 79
[15,] 8 15 20 57
[16,] 86 2 7 5
[17,] 62 5 21 12
Fuzzyness coefficients:
dunn_coeff normalized
0.5945847 0.4594462
Closest hard clustering:
[1] 1 1 2 1 3 3 4 1 3 1 3 2 3 4 4 1 1

Silhouette plot information:
cluster neighbor sil_width
10 1 3 0.82938820
1 1 3 0.81032816
2 1 3 0.80477324
16 1 3 0.80118019
4 1 3 0.76509472
8 1 3 0.69297873
17 1 3 0.57146445
3 2 4 0.06730237
12 2 4 -0.17685372
6 3 4 0.61400342
5 3 4 0.60246272
9 3 1 0.54122724
13 3 4 0.49619115
11 3 4 0.26723303
15 4 3 0.52706369
7 4 3 0.41649447
14 4 3 0.38930883
Average silhouette width per cluster:
[1] 0.75360110 -0.05477567 0.50422351 0.44428900
Average silhouette width of total data set:
[1] 0.5305671

136 dissimilarities, summarized :
Min. 1st Qu. Median Mean 3rd Qu. Max.
5182.8 21552.0 55122.0 62729.0 85846.0 177810.0
Metric : euclidean
Number of objects : 17

However, this data, upon discussion with Dr Poon, was deemed incorrect, as I had set the number of clusters for the fuzzy clustering. Therefore, this week, I will attempt to perform this test again without setting the cluster parameter. The fanny function, however, will not work without having the cluster being set, and thusly, I will attempt to use the e1071 package in R, as discussed last week, to perform this test. Additionally, if time permits, this week I will also be using the micEcon package’s switching regression function on the dataset provided by Dr Poon.

I have also started writing up my partial draft, and expect this to be completed by Friday of this week.

Thesis Update:

This week my meeting with Dr Poon highlighted my work for the coming week. I will be attempting to cluster, using the e1071 package (or possibly the clValid package) in R, based upon the data from Dataset 5204.0 from the Australian Bureau of Statistics available at:

http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/5204.02005-06?OpenDocument

In particular, this clustering analysis will be performed upon the Compensation of Employees by current prices according to industry (Table 59), according to Dr Poons instructions, as this would provide preliminary results for part of this project.

. The results will be posted up next week.

Additionally, I have been requested to create folders for each component of work which I am currently undertaking in order to archive and manage sources better in case further work needs to be conducted on this topic. As such, I have created 7 folders, as defined by Dr Poon (one folder for introduction, one for literature review, one for methodology, one for data modeling, one for analysis, one for discussion, one for conclusions).

Additionally, review of papers by Muller, Bernard, and Frantzen (see below for references), have given me insight into why this study needs to be conducted. Drawing upon these papers, it can be seen that a more extensive analysis (especially from Mullers Paper) needs to be conducted in order to reach a further level of understanding in the convergence and divergence of industries, as Muller states that past studies have had contrary results, with some papers finding convergence amongst both manufacturing and service sectors of industries, and others finding no convergence at all (Muller, pp 6-7).

Furthermore, in relation to this, Frantzen argues that while overall in OECD
countries there is a clear trend of convergence, and confirmation of neo-classical models, the estimation of panel data upon a national level, when accounting for country-specific fixed effects, challenges the model, thus creating a level of uncertainty. It has also clearly be stated by Frantzen that “the empirical work on convergence through technological diffusion is still limited”, thus providing further incentive to carry out this study upon Australian data.

Likewise, Bernard’s article states that “the debate over convergence has lost its way…becoming mired in a debate about 2% per year convergence rates and their robustness or lack thereof” (Bernard, pp2). My study currently will aim to provide empirical data calculated based on statistics gathered by the Australian Bureau of Statistics to further give grounds for the argument of “robustness, or lack thereof” (Bernard, pp2).


References:
Muller, G, 2000, A Glimpse on Sectoral Convergence of Productivity Levels, Halle Institute for Economic Research , Discussion Paper Nr 133

Bernard, A.B, Jones, C.I, 1996, Technology and Convergence, The Economic Journal, Vol 106. No 437.

Frantzen, D, 2004, Technological Diffusion and Productivity Convergence: A Study for Manufacturing in the OECD, Southern Economic Journal, 71(2), 352-376

Thesis

The purpose of the previous post is to illustrate the project plan which was created in the first week of this semester. So far, progress has been going according to plan. This plan, however, also provides for contingencies should they arise during the course of the project.

Thesis Update

This week I have outlined a few approaches to use in analysing the data once a data set has been decided upon. Firstly, I will use Fuzzy Clustering, and then Switching Regression, as outlined by Dr Poon, in order to see how I.T. correlates to Industry Structures. A second method I will be attempting, if time permits, would be to perform Switching Regression first, and then perform Fuzzy Clustering, in the method outlined by Hathaway and Bezdek in their 1993 paper titled Switching Regression and Fuzzy Clustering published in IEEE Transactions on Fuzzy Systems, Vol 1, No 3, August 1993,

I will attempt these two methods using firstly the "cor" and "fanny" functions for R which I have in literature on the internet (I have not tested these two functions as of yet, but will do so this week). However, this may only provide limited support for regression and clustering, and I may have to write the scripts to perform these functions in R, as I wish to do both Fuzzy C-Means, and Fuzzy C-Regression for the Fuzzy Clustering part of this project.

Additionally, we (Dr Poon's thesis group), have attempted to compile a data set in the past week, which we will be refining and adding to in the coming week.

Thesis Project - Update

This week, I travelled to the University of New South Wales to review their offline material. Located 3 relevant sources, which will be listed later. Furthermore, I have read the newest paper by Parham (2001), in order to execute the data collection part of this treatise.

Furthermore, I am currently implementing regression in "R" to get an idea of how to perform switching regression in the modelling part of this treatise.

This week, I will be attempting data collection via ABS, as well as coming up with an approach on how to tackle this problem. An idea I have right now would be to concentrate on Bezdek's paper on Switching Regression and Fuzzy Clustering dated 1993, which would allow me to develop an understanding of switching regression in relation to Fuzzy C-Regression. An alternative, which may be easier to perform (but which is considered an alternative, as Fuzzy C-Regression may yield more detailed results), would be that of utlilising Fuzzy C-Means in relation to Switching Regression, as Dr Poon has performed this before, and thus I would be able to draw on his experience if I were to implement fuzzy clustering using the Fuzzy C-Means method.

As an aside, I have created the project plan, which comprises of a table listing the times and requirements, as well as a list of sources (which is not complete, as I may locate sources later on which I have not been able to locate thus far).

Thesis Update

This week I spent more time on researching the impact of IT upon industry, as viewed from the production function f(x) = y(ICT, non-ICT), Note: f(x) is a productivity function which is determined by y, with variables both ICT and non-ICT related.

Have read most papers by Bryjolfsson, and have researched into his sources to gain a deeper understanding of the subject area, by viewing his bibliography and extending upon my research from that point. I am now also making arrangements to visit UNSW's library. I have scheduled my trip to Maquarie University for today, and will be looking at the following books:

1. Beyond the IT productivity paradox / edited by Leslie P. Willcocks and Stephanie Lester

2.Information technology and the productivity paradox : assessing the value of the investment in IT / Henry C. Lucas, Jr

3. The trouble with computers : usefulness, usability, and productivity / Thomas K. Landauer

Furthermore, have researched into clustering, in particular, fuzzy clustering, as well as started doing tutorials upon R, available from http://www.cyclismo.org/tutorial/R/, to gain knowledge of R in order to gain the required skills to perform both fuzzy clustering and switching regression using R upon industry data sets.

In my research with ABS, I have also located Australian National Accounts: Information and Communication Technology Satellite Account (from http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/5259.02002-03?OpenDocument), which includes reports on investment in ICT products, and investment in hardware and software, sorted by industry.

Meetings/Research

In the past two weeks, I have met with Dr Simon Poon twice to discuss the progress of my research into the topic area. We have set a schedule for the work, which is as follows:

By August 1st, w should have a clear idea of what we're doing, as well as be up to date on relevant literature

By September 1st, we should have finished all modelling (which in my case involves the application of fuzzy clustering, as well as switching regression, to a data set in order to generate results, and infer findings from these results).

By October 1st, we should have finished analysis of the modelling results.

We would also be using October to finish writing up the final copy of the thesis.

Additionally, I have conducted research via the library's online databases, based on preliminary articles provided by Dr Poon. Furthermore, I have gone to the Fisher, and Badham libraries, to conduct research into this area. I am planning on going to more libraries to conduct further offline research into the areas of clustering, and regression, in the coming week.

Blog Update 28/06/07

Due to exams, there was only minimal work done throughout this period, and as such, the blog was not updated.

Currently, I am learning Fuzzy Clustering, and Switching Regression techniques in order to perform the analysis which is required for my thesis, the Impact of IT upon Industry Structures. To find out the impact of IT upon industry, one must perform fuzzy clustering and switching regression upon a multi-attribute set of data in order to draw conclusions from the centoids derived from fuzzy clustering.

It has also been decided in the last meeting with Dr Poon that we use R (also known as GNU S), a free software environement statistical modelling, to achieve this aim. Details of this can be found at: http://www.r-project.org/

There is also a meeting with Dr Poon tomorrow to discuss the progress which has been made on this project.

Held Meeting with Dr Simon Poon on last Monday, discussed the thesis topic and what was required. Was given two additional papers to read through before meeting again. Have scheduled to read papers during this week, and also to make notes on paper in order to gain a better working knowledge of the papers.

So far the thesis topic, to my understanding, is about the change which has occurred to industry, as seen from a productivity point of view. My past meetings with Dr Poon have indicated that the change can be mapped as a function, and using fuzzy clustering to find centoids for groups (groups might be defined as an industry). To do this in detail, I require an understanding of econometrics, and also a knowledge of a statistical tool such as shazam to map these functions.

Hi, Gary's Thesis Blog has just been created to track the progress of the thesis which I am Attempting. This thesis will be on the Impact of IT on Industry Structure, supervised by Dr Simon Poon.

As of now, I am researching material for my next meeting with Simon Poon, in order to be more knowledgeable in this area.

Gary.