Exploring analytical and holistic processing in facial expression recognition

Background. The study explores two main processes of perception of facial expression: analytical (perception based on individual facial features) and holistic (holistic and non-additive perception of all features). The relative contribution of each process to facial expression recognition is still an open question.

Objective. To identify the role of holistic and analytical mechanisms in the process of facial expression recognition.

Methods. A method was developed and tested for studying analytical and holistic processes in the task of evaluating subjective differences of expressions, using composite and inverted facial images. A distinctive feature of the work is the use of a multi-dimensional scaling method, by which a judgment of the contribution of holistic and analytical processes to the perception of facial expressions is based on the analysis of the subjective space of the similarity of expressions obtained when presenting upright and inverted faces.

Results. It was shown, first, that when perceiving upright faces, a characteristic clustering of expressions is observed in the subjective space of similarities of expression, which we interpret as a predominance of holistic processes; second, by inversion of the face, there is a change in the spatial configuration of expressions that may reflect a strengthening of analytical processes; in general, the method of multidimensional scaling has proven its effectiveness in solving the problem of the relation between holistic and analytical processes in recognition of facial expressions.

Conclusion. The analysis of subjective spaces of the similarity of emotional faces is productive for the study of the ratio of analytical and holistic processes in the recognition of facial expressions.

References

Adolphs, R. (2002). Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behavioral and cognitive neuroscience reviews, 1(1), 21-62.

Barabanschikov V. A. (2012) Facial expressions and perception of it. Moscow. Institut psikhologii RAN. (in Russ.).

Barabanshchikov V.A., Zhegallo A.V., Ivanova L.A. (2010) Characteristic of inverted face image expression recognition. Experimental psychology in Russia: traditions and perspectives (pp. 218-223) Moscow: Institut psikhologii RAN. (in Russ.).

Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1-68.

Bimler, D. L., & Paramei, G. V. (2006). Facial-expression affective attributes and their configural correlates: components and categories. The Spanish journal of psychology, 9(1), 19-31.

Bondarenko Ya. A., Men'shikova G. Ya. (2016) Specificity of perception of expressions of composite faces. International scientific conference in memory of E.N. Sokolov and Ch.A. Izmailov Human-Neuron-Model (pp. 46-51). Моscоw: MGU. (in Russ.).

Borg, I., & Groenen, P. J. (2005). Modern multidimensional scaling: Theory and applications. Springer Science & Business Media.

Boucsein, W., Schaefer, F., Sokolov, E. N., Schröder, C., & Furedy J. J. (2001). The color-vision approach to emotional space: Cortical evoked potential data. Integrative Physiological and Behavioral Science, 36(2), 137-153. https://doi.org/10.1007/BF02734047

Calder, A. J., & Jansen, J. (2005). Configural coding of facial expressions: The impact of inversion and photographic negative. Visual cognition, 12(3), 495-518.

Calder, A., Rhodes, G., Johnson, M., & Haxby, J. (Eds.). (2011). Oxford handbook of face perception. Oxford University Press, 177-194.

Calvo, M. G., & Lundqvist, D. (2008). Facial expressions of emotion (KDEF): Identification under different display-duration conditions. Behavior research methods, 40(1), 109-115.

Carey, S., & Diamond, R. (1977). From piecemeal to configurational representation of faces. Science, 195(4275), 312-314.

Carey, S., & Diamond, R. (1994). Are faces perceived as configurations more by adults than by children? Visual cognition, 1(2-3), 253-274.

Chellappa, R., Sinha, P., & Phillips, P. J. (2010). Face recognition by computers and humans. Computer, 43(2), 46-55.

Chen, M. Y., & Chen, C. C. (2010). The contribution of the upper and lower face in happy and sad facial expression classification. Vision Research, 50(18), 1814-1823.

Darwin, C., & Prodger, P. (1998). The expression of the emotions in man and animals. Oxford University Press, USA.

Ekman, P., Friesen, W. V., Hager, J. C. (2002). The Facial Action Coding Systems, Salt Lake City: Research Nexus eBook. London: Weidenfeld & Nicolson.

Freire, A., Lee, K., Symons, L. A. (2000). The face-inversion effect as a deficit in the encoding of configural information: Direct evidence. Perception, 29(2), 159-170.

Hoffmann, H., Kessler, H., Eppel, T., Rukavina, S., & Traue, H. C. (2010). Expression intensity, gender and facial emotion recognition: Women recognize only subtle facial emotions better than men. Acta psychologica, 135(3), 278-283.

Izmailov Ch. A., Sokolov E. N., Chernorizov A. M. Psychophysiology of color vision. Mоscоw: MGU. 1989, 39-45. (in Russ.).

Izmailov Ch.A., Korshunova S.G., Shekhter M.S., Potapova A.Ya. (2009) Visual differentiation of complex stimuli: emotional expressions of a human face. Teoreticheskaya i eksperimental'naya psikhologiya (Theoretical and Experimental Psychology), 2 (1), 5-22. (in Russ.).

Izmajlov Ch.A., Korshunova S.G., & Sokolov E.N (1999) Spherical model of distinguishing emotional expressions in the human face model. Zhurnal vysshej nervnoj dejatel’nosti (Journal of Higher Nervous Activity), 49 (2), 186-199. (in Russ.).

Kiselnikov A.A., Sergeev A.A., Vinitskiy D.A. (2019). A Four-Dimensional Spherical Model of Interaction Between Color and Emotional Semantics. Psychology in Russia: State of the Art, 12(1), 48-66.

Korolkova O.A.(2013) The effect of categoricality of perception: the main approaches and psychophysical models. Eksperimental'naya psikhologiya (Experimental psychology), 1, 61–75. (in Russ.).

Kruskal, J. B. (1964). Nonmetric multidimensional scaling: a numerical method. Psychometrika, 29(2), 115-129.

Leder, H., & Bruce, V. (2000). When inverted faces are recognized: The role of configural information in face recognition. The quarterly journal of experimental psychology Section A, 53(2), 513-536.

Leppänen, J. M., Kauppinen, P., Peltola, M. J., & Hietanen, J. K. (2007). Differential electrocortical responses to increasing intensities of fearful and happy emotional expressions. Brain research, 1166, 103-109.

Maurer, D., Le Grand, R., & Mondloch, C. J. (2002). The many faces of configural processing. Trends in cognitive sciences, 6(6), 255-260.

McKone, E., & Yovel, G. (2009). Why does picture-plane inversion sometimes dissociate perception of features and spacing in faces, and sometimes not? Toward a new theory of holistic processing. Psychonomic bulletin & review, 16(5), 778-797.

Menshikova, G. Ya. (2010). Facial expression recognition with the use of chimeric face technique. Psychology in Russia: State of the art, 3(1),278-286.

Menshikova, G. Ya., Bimler, D., Bondarenko, Ya., & Paramei, G. (2016). Composite facial expressions: half-face diagnostic features dominate emotion discrimination. Perception, 45 (2), 235-236.

Palermo, R., & Coltheart, M. (2004). Photographs of facial expression: Accuracy, response times, and ratings of intensity. Behavior Research Methods, Instruments, & Computers, 36(4), 634-638.

Panksepp, J. (2011). The basic emotional circuits of mammalian brains: do animals have affective lives? Neuroscience & Biobehavioral Reviews, 35(9), 1791-1804.

Paramei, G. V., Bimler, D. L., & Skwarek, S. J. (2011). No effect of inversion on perceived similarity of facial expression of emotion. Proceedings of Fechner Day, 27, 59-64.

Picard, R. W. (1995). Affective computing-MIT media laboratory perceptual computing section technical report No. 321. Cambridge, MA, 2139.

Psalta, L., Young, A. W., Thompson, P., & Andrews, T. J. (2014). Orientation-sensitivity to facial features explains the Thatcher illusion. Journal of vision, 14(12), 9.

Rezlescu, C., Susilo, T., Wilmer, J. B., & Caramazza, A. (2017). The inversion, part-whole, and composite effects reflect distinct perceptual mechanisms with varied relationships to face recognition. Journal of Experimental Psychology: Human Perception and Performance, 43(12), 1961.

Rosenthal, G., Levakov, G., & Avidan, G. (2018). Holistic face representation is highly orientation-specific. Psychonomic bulletin & review, 25(4), 1351-1357.

Rothermund, K., & Koole, S. L. (2018). Three decades of Cognition & Emotion: A brief review of past highlights and future prospects. Cognition and Emotion, 32(1), 1-12.

Russell, J. A. (2009). Emotion, core affect, and psychological construction. Cognition and emotion, 23(7), 1259-1283.

Russell, J. A., & Bullock, M. (1986). On the dimensions preschoolers use to interpret facial expressions of emotion. Developmental Psychology, 22(1), 97.

Russell, J. A.,Barrett L. (1999). Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of personality and social psychology. 76(5).805-819.

Scholsberg, H. (1941). A scale for the judgment of facial expressions. Journal of experimental psychology, 29(6), 497-510.

Shepard, R. N. (1962). The analysis of proximities: multidimensional scaling with an unknown distance function. Psychometrika, 27(2), 125-140.

Sinha, P., Balas, B., Ostrovsky, Y., & Russell, R. (2006). Face recognition by humans: Nineteen results all computer vision researchers should know about. Proceedings of the IEEE, 94(11), 1948-1962. doi: 10.1109/JPROC.2006.884093

Sokolov E.N. Essays on Psychophysiology of Consciousness. Moscow: MGU, 2010. (in Russ.).

Sokolov, E. N., & Boucsein, W. (2000). A psychophysiological model of emotion space. Integrative Physiological and Behavioral Science, 35(2), 81-119.

Tanaka, J. W., & Simonyi, D. (2016). The “parts and wholes” of face recognition: A review of the literature. The Quarterly Journal of Experimental Psychology, 69(10), 1876-1889.

Tanaka, J. W., Kaiser, M. D., Butler, S., & Le Grand, R. (2012).Mixed emotions: Holistic and analytic perception of facial expressions. Cognition & emotion, 26(6), 961-977.doi: 10.1080/02699931.2011.630933

Torgerson, W. S. (1958). Theory and methods of scaling. 1958. RE Krieger Pub. Co.

Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., & Nelson, C. (2009). The NimStim set of facial expressions: judgments from untrained research participants. Psychiatry research, 168(3), 242-249.

Tsao, D. Y., & Livingstone, M. S. (2008). Mechanisms of face perception. Annu. Rev. Neurosci., 31, 411-437.

Vartanov, A., Ivanov, V., & Vartanova, I. (2020). Facial expressions and subjective assessments of emotions. Cognitive Systems Research, 59, 319-328.

Wells, L. J., Gillespie, S. M., & Rotshtein, P. (2016). Identification of emotional facial expressions: Effects of expression, intensity, and sex on eye gaze. PloS one, 11(12).

Woodworth, R. S. (1937). Experimental Psychology. New York: Holt, 1938. Department of Psychology Dartmouth College Hanover, New Hampshire.

Yovel, G., & Kanwisher, N. (2005). The neural basis of the behavioral face-inversion effect. Currentbiology, 15(24), 2256-2262.