The homogeneity of scales of intellectual abilities: psychometric analysis

Relevance.

Nowadays the researchers commonly use a limited set of standard procedures and statistical coefficients when they develop psychometric instruments and investigate their structure. The routine using of such procedures without taking into account the specific features of psychometric scales can lead to incomplete or even inadequate results. In this context detailed consideration of the structure of psychometric instruments seems to be important and it may demand various non-standard ways of statistical analysis. 

Objectives. 

To conduct detailed analysis of the results of two intelligent subtests at the item level and to assess the sufficiency and adequacy of using standard methods for estimation of reliability and structural validity for these subtests.

Methods.

We analyze the data collected in intelligence testing of a large sample of respondents (11335 young adults). The respondents passed the KR-3 battery. In this study we examine in detail the structure of the subtests "Syllogisms" and "Analogies". Specifically, we estimated the reliability of the scales by the Cronbach's alpha coefficient, and the structure at the item level using the confirmatory factor analysis.

Results and conclusions.

Estimation of the reliability of the scales by Cronbach's alpha coefficient showed the importance of taking into account the time limitation, which is commonly used in intelligence tests. On the other hand, a detailed analysis of each subtest items made it possible to find out an additional factor which was not originally proposed in the factor structure. It is the factor of higher-order abilities of abstract analysis, whilst the subtest originally aimed at esteeming the special abilities. Confirmatory factor analysis showed improvement of fit when this factor was added. The results allow to conclude that the researcher may miss important properties of scales if one does not perform a detailed analysis of testing procedures and the structure of subtest at the item level, and so one may draw incomplete or inadequate conclusions about their psychometric properties.

References

1. Kornilova T.V., Kornilov A.S., Zirenko M.S., Chumakova M.A. (2019). Psychometric properties of the modified International Cognitive Ability Resource (ICAR) test battery. Natsional'nyi psikhologicheskii zhurnal (National psychological journal), 3 (35), 32–45. (in Russ.). 

2. Guidelines for organizing and conducting professional psychological selection in military educational institutions of the Ministry of Defense of the Russian Federation (2002). Moscow. (in Russ.).

3. Mitina O.V. (2013). Development and adaptation of psychological questionnaires. Moscow: Smysl. (in Russ.).

4. Gurevich K.M., Akimova M.K., Borisova E.M., Kozlova V.T., Loginova G.P. (1987).School test of mental development. Methodological recommendations for working with the test (for school psychologists). Moscow. (in Russ.).

5. SugonyaevK.V., RadchenkoYu.I. (2018). Spearman’s law of diminishing returns: investigation on large-scales russian samples. Vestnik Yuzhno-Ural'skogo gosudarstvennogo universiteta. Seriya: Psikhologiya (Bulletin of the South Ural State University. Series “Psychology”), 11 (1), 5–21. (in Russ.).

6. Eliseev O.P. (2003).R. Amthauer’s intelligence structure test. In Workshop on Personality Psychology (pp. 342–370). Saint-Petersburg: Piter. (in Russ.).

7. Ushakov D.V.Systematic approach in the psychology of intelligence: theory, approach, methodology. Barabanschikov V.A. (ed.) The idea of the systems in modern psychology (pp. 236–261). Moscow: Institut psikhologii RAS. (in Russ.).

8. Fer M.R., Bakarak V.R.(2010). Psychometrics: Intoduction. Chelyabinsk: Челябинск: Publishing Centre of YurGU. (in Russ.).

9. Barrett, P., Eysenck, H.J., & Lucking, S. (1986). Reaction time and intelligence: A replicated study. Intelligence, 10 (1), 9–40. 

10. Bentler, P.M. (1992). On the fit of models to covariances and methodology to the Bulletin. Psychological bulletin, 112 (3), 400–404. 

11. Bortolotti S.L.V., Tezza R., de Andrade D.F., Bornia A.C., de Sousa Júnior A.F. (2013). Relevance and advantages of using the item response theory. Quality & Quantity, 47 (4), 2341–2360.

12. Divgi, D.R. (1979). Calculation of the tetrachoric correlation coefficient. Psychometrika, 44 (2), 169–172. 

13. Dunn, T.J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British journal of psychology, 105 (3), 399–412. 

14. Ercan, I., Yazici, B., Sigirli, D., Ediz, B., & Kan, I. (2007). Examining Cronbach alpha, theta, omega reliability coefficients according to sample size. Journal of modern applied statistical methods, 6 (1), 291–303.

15. Estrada, E., Román, F.J., Abad, F.J., & Colom, R. (2017). Separating power and speed components of standardized intelligence measures. Intelligence, 61, 159–168.

16. Jensen, A.R., & Munro, E. (1979). Reaction time, movement time, and intelligence. Intelligence, 3 (2), 121–126.

17. Johnson-Laird P.N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press. 

18. Lazarević, L.B., Knežević, G., Mitić, M., & Đurić-Jočić, D. (2018). Psychometric properties of the Serbian version of the Wechsler adult intelligence scale: (WAIS-IV). Psihologija, 51 (3), 333–349.

19. McDonald R.R. (1999). Test Theory: A Unified Treatment. Mahwah, NJ: Lawrence Erlbaum Associates.

20. Molenaar D., Kő N., Rózsa S., Mészáros A. (2017). Differentiation of cognitive abilities in the WAIS-IV at the item level. Intelligence, 65, 48–59.

21. Muthén L.K., Muthén B. (2017). Mplus. The comprehensive modelling program for applied researchers: user’s guide. Los Angeles, CA: Muthén & Muthén.

22. Netemeyer R.G., Bearden W.O., Sharma S. (2003). Scaling procedures: Issues and applications. Sage Publications. 

23. Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21 (2), 173–184. 

24. Sheppard, L.D., & Vernon, P.A. (2008). Intelligence and speed of information-processing: A review of 50 years of research. Personality and individual differences, 44 (3), 535–551. 

25. Shwabe I. Nature, Nurture and Item Response Theory – A Psychometric Approach to Behaviour Genetics. Thesis for: PhD, 2016.