Prosody, gesture, and self-adaptors: A case study of Autism Spectrum Disorder for large corpora collection
Palabras clave:
multimodality, Autism Spectrum Disorder, gestures, prosody, self-adaptorsDerechos de autor 2023 CHIMERA: Revista de Corpus de Lenguas Romances y Estudios Lingüísticos
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial 4.0.
Resumen
Individuals with Autism Spectrum Disorder (ASD) display distinctive speech patterns and bodily movements. This pilot study examines spontaneous interactions between an individual with ASD and a typically developing peer (age 19), incorporating monological and dialogical contexts. The analysis, grounded in the Language into Act Theory framework, explores the information structure of the speech and linguistic parameters influenced by prosody, such as utterance boundaries, information structure, speech disfluency, mean length of prosodic units, and speech rate. The study also employs Kita's model to analyze bodily movements, including gestures and self-adaptors, and their temporal relation with speech. Notable findings reveal that ASD speech is characterized by a monotonous information structure and prosodic contour, featuring slower and longer units with a limited rate variation and information type. On the gestural side, the ASD subject exhibits fewer gestures and more self-adaptors, with some instances of asynchrony between gestures and speech. This pilot study serves as a foundational step for a broader corpus-based project dedicated to exploring the development of pragmatic skills in individuals with ASD.
Descargas
Citas
Amir, N., Vered, S.V., Izre'el, S. 2004. Characteristics of Intonation Unit Boundaries in Spontaneous Spoken Hebrew: Perception and Acoustic Correlates. In: B. Bel, I. Marlien (eds.) Proceedings of Speech Prosody 2004, ISCA, 677–680
Andrén Mats (2010) Children’s Gestures from 18 to 30 months, PhD thesis, University of Lund.
Asperger, H. (1944). Die" Autistichen Psychopathen" im Kindesalter. Archiv Psychiat Nervenkrankheiten, 117, 76-136.
Augustyn Paul R, A, Klin A, Volkmar FR. Perception and production of prosody by speakers with autism spectrum disorders. J Autism Dev Disord. 2005 Apr;35(2):205-20. doi: 10.1007/s10803-004-1999-1. PMID: 15909407
Austin JL. (1962), How to do things with words, Oxford, Oxford University Press.
Beccaria, F., Gagliardi, G., Kokkinakis, D. 2022. Extraction and Classification of Acoustic Features from Italian Speaking Children with Autism Spectrum Disorders. In Proceedings of the RaPID Workshop - 13th LREC, pages 22–30, Marseille, France. European Language Resources Association.
Biancalani, S., Gagliardi, G., Innocenti, M. (2023). Aspetti soprasegmentali e pragmatici dell’eloquio di bambini di età scolare con disturbo dello spettro autistico. Uno studio pilota. In M. Castagneto, M. Ravetto (a cura di) «La comunicazione parlata vercelli 2021», Roma Aracne.
Breckinridge Church Ruth, Martha W. Alibali & Spencer D. Kelly (eds.), Why gesture? How the hands function in speaking, thinking and communicating, 353-377. Philadelphia/Amsterdam: John Benjamins.
Bressem J, Ladewig SH., Müller C. (2013). Linguistic Annotation System for Gestures (LASG). In Müller C., Cienki A., Fricke E. Ladewig SH., McNeill D, Teßendorf S. (eds.), «Body - Language - Communication: An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science 38)» Vol. 1, Berlin: De Gruyter Mouton, 1098–1125.
Boersma, P., Weenink, D. (2021). Praat: doing phonetics by computer [Computer program]. Version 6.2.06, retrieved 23 January 2022 from https://www.praat.org.
Cantalini, G. (2022). Corpus multimodale annotato per lo studio della gestualità co-verbale nel “parlato-parlato” e nel “parlato-recitato”. In E. Cresti, M. Moneglia (eds.) «Corpora e Studi Linguistici, Atti del LIV Congresso Internazionale di Studi della Società di Linguistica Italiana (Online, 8-10 settembre 2021)». 135-149.
Cantalini, G., Moneglia, M. 2020. The annotation of Gesture and Gesture / Prosody synchronization in Multimodal Speech Corpora. Journal of Speech Science, V. 9, 1-24
Cantalini G., Moneglia, M., Gagliardi, G., Proietti, M. 2020. La relazione gesto / prosodia e la sua variabilità. Il parlato spontaneo di contro alla performance attorale. In: A. De Meo, F. Dovetto (eds) La Comunicazione Parlata. Roma: Aracne, 63-89.
Corley, M., & Hartsuiker, R. J. (2003). Hesitation in speech can… um… help a listener understand. Proceedings of the Annual Meeting of the Cognitive Science Society, 25(25), 1–10.
Cavalcante, F. A. (2016). The topic unit in spontaneous American English: A corpus- based study. Belo Horizonte: Federal University of Minas Gerais.
Chafe, W. (1994). Discourse, consciousness and time. The flow and displacement of conscious experience in speaking and writing. Chicago, IL: Chicago University Press.
Chan, S., Khader, M., Ang, J., Chin, J., & Chai, W. (2016). To behave like a liar: Nonverbal cues to deception in an asian sample. Journal of Police and Criminal Psychology, 31, 165-172.
Chu, N. & Kita, S. (2008) Spontaneous Gestures During Mental Rotation Tasks: Insights Into the Microdevelopment of the Motor Strategy, Journal of Experimental Psychology General 137(4):706-23, DOI: 10.1037/a0013157
Chui K. (2005). Temporal patterning of speech and iconic gestures in conversational discourse. J. Prag 871–887. 10.1016/j.pragma.2004.10.016
Chui, K., Lee, C. Y., Yeh, K., & Chao, P. C. (2018). Semantic processing of self-adaptors, emblems, and iconic gestures: An ERP study. Journal of Neurolinguistics, 47, 105-122.
Colgan, E. S., Lanter, E., McComish, C., Watson, L. R., Crais, E. R., Baranek, G. T. (2006). Analysis of social interaction gestures in infants with autism. Child Neuropsychology, 12, 307-319.
Cresti, E. (2000). Corpus di italiano parlato, Volume I, in «Studi di grammatica italiana pubblicati dall'Accademia della Crusca». Firenze: Accademia della Crusca.
Cresti, E. (2020), The pragmatic analysis of speech and its illocutionary classification according to the Language into Act Theory, in Izre ?el S., Mello H., Panunzi A., Raso T. (eds.) «In search of basic units of spoken language: A corpus-driven approach», Amsterdam, John Benjamins, 181-219.
Cresti, E. (2021). The appendix of comment according to language into act theory: corpus-based research. CHIMERA 8, 46–69.
Cresti, E., Moneglia, M. (2018). The illocutionary basis of information structure. Language into Act Theory (L-AcT). In E. Adamou, K. Haude, M. Vanhove (eds.) «Information structure in lesser-described languages: Studies in prosody and syntax». 359-401. Amsterdam: John Benjamins.
Danieli, M., Garrido, J.M., Moneglia, M., Panizza, A., Quazza, S., Swerts, M. (2004) Evaluation of Consensus on the Annotation of Prosodic Breaks in the Romance Corpus of Spontaneous Speech C-ORAL-ROM. In: MT. Lino, MF. Xavier, F. Ferreira, R. Costa, R. Silva (eds) Proceedings of the 4th LREC Conference. Paris: ELRA. 1513-1516.
de Marchena, A., Eigsti, I. (2010). Conversational Gestures in Autism Spectrum Disorders: Asynchrony but not Decreased Frequency. Autism Research, 3: 311-322. https://onlinelibrary.wiley.com/doi/full/10.1002/aur.159
Duffy, C., Healy, O. (2011). Spontaneous Communication in Autism Spectrum Disorder: A Review of Topographies and Interventions. Research in Autism Spectrum Disorders, 5, 977-983.
Eigsti, I., Schuh, J., Mencl, E., Schultz, R., Paul, R. (2011). The neural underpinnings of prosody in autism. In Child Neuropsychology: A Journal on Normal and Abnormal Development in Childhood and Adolescence.
ELAN (Version 6.7) [Computer software]. (2023). Nijmegen: Max Planck Institute for Psycholinguistics, The Language Archive. Retrieved from https://archive.mpi.nl/tla/elan
Esteve-Gibert, N., & Prieto, P. 2013. Prosodic structure shapes the temporal realization of intonation and manual gesture movements. Journal of Speech, Language, and Hearing Research, 563, 850–864. https://doi.org/10.1044/1092-43882012/12-0049.
Filipe MG, Frota S, Castro SL, Vicente SG. Atypical prosody in Asperger syndrome: perceptual and acoustic measurements. J Autism Dev Disord. 2014 Aug;44(8):1972-81. doi: 10.1007/s10803-014-2073-2. PMID: 24590408.
Fox Tree, J. E. (2001). Listeners’ uses of um and uh in speech comprehension. Memory & Cognition, 29(2), 320–326.
Froiland, JM., Davison, ML. (2016) Home literacy, television viewing, fidgeting and ADHD in young children. Educational Psychology, 36:8, 1337-1353, DOI: 10.1080/01443410.2014.963031
Gorman, K., Olson, L., Hill, A. P., Lunsford, R., Heeman, P. A., & van Santen, J. P. (2016). Uh and um in children with autism spectrum disorders or language impairment. Autism Research, 9(8), 854–865.
Graziano M., Nicoladis E., Marentette P. (2020). How Referential Gestures Align With Speech: Evidence From Monolingual and Bilingual Speakers. Lang. Learn. 70 266–304. 10.1111/lang.12376
Fusaroli, R., Lambrechts, A., Bang, D., Bowler, DM., Gaigg, SB. (2017). Is voice a marker for Autism spectrum disorder? A systematic?review and meta-analysis. Autism Research, 10, 384–407. DOI:10.1002/aur.1678
't Hart, Johan (1981). "Differential sensitivity to pitch distance, particularly in speech". The Journal of the Acoustical Society of America. 69 (3): 811–821.
t’ Hart, J., Collier, R., Cohen, A. (1990). A perceptual study on intonation. An experimental approach to speech melody, Cambridge, Cambridge University Press.
Irvine, C. A., Eigsti, I. M., & Fein, D. A. (2016). Uh, um, and autism: Filler disfluencies as pragmatic markers in adolescents with optimal outcomes from autism spectrum disorder. Journal of Autism and Developmental Disorders, 46(3), 1061–1070.
Izre ?el S., Mello H., Panunzi A., Raso T. (eds.) 2020 «In search of basic units of spoken language: A corpus-driven approach», Amsterdam, John Benjamins.
Izre'el, S, Mettouchi, A. 2015. Representation of Speech in CorpAfroAs. Transcriptional Strategies and Prosodic Units. In: A. Mettouchi, M. Vanhove, D. Caubet (eds.) Corpus-based Studies of Lesser-described Languages: The CorpAfroAs corpus of spoken AfroAsiatic languages. Amsterdam: Benjamins. 13–41.
Janke, V., & Perovic, A. (2017). Advanced syntax and primary pragmatics in children with ASD. In L. R. Naigles (Ed.), Innovative investigations of language in autism spectrum disorder (pp. 141–161). Walter de Gruyter GmbH; American Psychological Association. https://doi.org/10.1037/15964-008
Kanner, L. (1943). Autistic disturbances of affective contact. Nervous Child, 2(3), 217-250.
Kendon, A. (2004). Gesture. Cambridge.
Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and gesture (pp. 162-185). Cambridge Cambridge University Press
Kita, S., van Gijn I., van der Hulst H. (1998). Movement phases in signs and co-speech gestures, and their transcription by human coders. In Wachsmuth I., Fröhlich M. (eds), «Gesture and Sign Language in Human-Computer Interaction» Berlin: Springer, 23–35.
Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16–32.
Ladewig SH, Bressem J. A linguistic perspective on the notation of gesture phases. In Müller C, Cienki A, Fricke E, Ladewig SH, McNeill D, Teßendorf S. (eds), «Body - Language - Communication: An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science 38)» Vol. 1, Berlin: De Gruyter Mouton, 2013, 1060–1079.
Lausberg, H. 2013.Understanding Body Movement. A Guide to Empirical Research on Nonverbal Behaviour, With an Introduction to the NEUROGES Coding System. Frankfurt am Main: Peter Lang,
Lin, W., Orton, I., Li, Q., Pavarini, G., & Mahmoud, M. (2021). Looking at the body: Automatic analysis of body gestures and self-adaptors in psychological distress. IEEE Transactions on Affective Computing.
Loehr, D. (2007). Aspects of rhythm in gesture and speech. Gesture, 7, 179–214
Loehr, D. P. 2012. Temporal, structural, and pragmatic synchrony between intonation and gesture. Laboratory Phonology, 31., 71–89. https://doi.org/10.1515/lp-2012-0006
Loehr, D. (2014). Gesture and prosody. In C. Mu?ller, A. Cienki, E. Fricke, S. Ladewig, D. McNeill, J. Bressem (eds.), «Handbu?cher zur Sprach- und Kommunikationswissenschaft / Handbooks of Linguistics and Communication Science (HSK) 38/2» 1381-1391. de Gruyter. doi.org/10.1515/9783110302028.1381
Lord, C., Paul, R. (1997). Language and communication in autism. In DJ. Cohen, FR. Volkmar (eds.) «Handbook of autism and pervasive developmental disorders (2nd ed.)» 195–225. New York, NY: Wiley.
Lord, C., Rutter, M., & LeCouteur, A. (1994). Autism Diagnostic Interview-Revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive develop- mental disorders. Journal of Autism and Developmental Disorders, 24, 659–685.
Lord, C., Rutter, M., DiLavore, P.C., Risi, S. (2002). Autism diagnostic observation schedule (ADOS). Los Angeles: Western Psychological Services.
Mahmoud, M., Morency, L. P., & Robinson, P. (2013, December). Automatic multimodal descriptors of rhythmic body movement. In Proceedings of the 15th ACM on International conference on multimodal interaction (pp. 429-436).
Maffia, M., De Micco, R., Pettorino, M., Siciliano, M., Tessitore, A., De Meo, A. (2021). Speech Rhythm Variation in Early-Stage Parkinson's Disease: A Study on Different Speaking Tasks. Front. Psychol., Psychology of Language, Volume 12. https://doi.org/10.3389/fpsyg.2021.668291
Martin, P. (2004). WinPitch Corpus: a text to speech alignment tool for multimodal corpora. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisboa, Portugal, European Language Resources Association (ELRA), 537–540.
Martin, P. 2009. Intonation du Franc?ais. Paris: Armand Colin.
Mehrabian, A., & Friedman, S. L. (1986). An analysis of fidgeting and associated individual differences. Journal of Personality, 54(2), 406-429.
McCann et al., 2007;
McGregor, K. K., & Hadden, R. R. (2020). Brief Report:“Um” fillers distinguish children with and without ASD. Journal of Autism and Developmental Disorders, 50(5), 1816–1821.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Mehrabian, A., & Friedman, S. L. (1986). An analysis of fidgeting and associated individual differences. Journal of Personality, 54(2), 406-429.
McNeill, D. (2005). Gesture and thought. University of Chicago Press.
Miller, J., Weinert, R. 1998. Spontaneous Spoken Language. Syntax and Discourse. Oxford: Clarendon Press.
Moneglia, M. (2005). The C-ORAL-ROM Resource. In: Cresti, E., Moneglia, M. (eds).. C-ORAL-ROM. Integrated Reference Corpora for Spoken Romance Languages, pp. 1-70, AMSTERDAM: Benjamins. DOI: https://doi.org/10.1075/scl.15.03mon
Moneglia, M., Raso, T., Malvessi-Mittmann, M., Mello, H. 2010. Challenging the perceptual relevance of prosodic breaks in multilingual spontaneous speech corpora: C-ORAL-BRASIL / C-ORAL-ROM. In: Speech Prosody 2010, W1.09, Satellite workshop on Prosodic Prominence: Perceptual, Automatic Identification. Chicago. https://www.isca-speech.org/archive/sp2010/sp10_2010.html, 2010
Moneglia, M., Raso, T. (2014). Notes on Language into Act Theory (L-AcT). In T. Raso, H. Mello (eds.), «Spoken Corpora and Linguistic Studies» 468–495. Amsterdam: John Benjamins. https://doi.org/10.1075/scl.61.15mon
Mundy Peter, Marian Sigman, Judy Ungerer, Tracy Sherman (1986) DEFINING THE SOCIAL DEFICITS OF AUTISM: THE CONTRIBUTION OF NON-VERBAL COMMUNICATION MEASURES 27, 5, 657-669
Neff, M., Toothman, N., Bowmani, R., Fox Tree, J. E., & Walker, M. A. (211). Don’t scratch! Self-adaptors reflect emotional stability. In Intelligent Virtual Agents: 10th International Conference, IVA 2011, Reykjavik, Iceland, September 15-17, 2011. Proceedings 11 (pp. 398-411). Springer Berlin Heidelberg.
Panunzi A., Gregori L. (2012), DB-IPIC. An XML database for the representation of information structure in spoken language, in Panunzi A., Raso T., Mello H. (a cura di) «Pragmatics and prosody. Illocution, modality, attitude, information patterning and speech annotation», Firenze, Firenze University Press.
Panunzi, A, Gregori, L, Rocha, B. 2020. Comparing annotations for the prosodic segmentation of spontaneous speech: Focus on reference units. In: S. Izre'el, H. Mello, A. Panunzi, T. Raso (eds.) In Search of Basic Units of Spoken Language. A corpus-driven approach. Amsterdam: Benjamins. 403-431
Rescorla, Leslie A., and Paige Safyer. "Lexical Composition in Children with Autism Spectrum Disorder (ASD)." Journal of Child Language 40, no. 1 (2013): 47-68, doi: 110.1017/S0305000912000232.
Robins, D.L., Fein, D., Barton, M.L., & Green, J.A. (2001). The Modified Checklist for autism in toddlers: An initial study investigating the early detection of autism and pervasive developmental disorders. Journal of Autism and Develop- mental Disorders, 31, 131–144.
Rohrer, P. L., Prieto, P. & Delais-Roussarie, E. 2019. Beat gestures and prosodic domain marking in French. In: S. Calhoun, P. Escudero, M. Tabain & P. Warren (eds) Proceedings of the 19th International Congress of Phonetic Sciences. Australasian Speech Science and Technology Association Inc. 1500-1504.
Rohrer, P. L. 2022. A temporal and pragmatic analysis of gesture-speech association: A corpus-based approach using the novel MultiModal MultiDimensional M3D labeling system. Ph.D. thesis Bercelona: University Pompeu Fabra
Saccone, V. (2022). Le unità del parlato e dello scritto mediato dal computer a confronto. La dimensione testuale della comunicazione spontanea. Edizioni dell’Orso, Alessandria.
Saccone, V., Panunzi, A., (2020), Le unità di comment multiplo. Analisi secondo la Teoria della Lingua in Atto, in De Meo/Dovetto (a cura di) «La comunicazione parlata. Atti del Congresso SLI–GSCP – Università degli Studi di Napoli “L’Orientale” (Napoli, 12-14 dicembre 2018)», Napoli, Aracne; 263-286.
Saccone, V., Panunzi, A. (2023), Strutture parentetiche nel parlato italiano: classificazione funzionale e identificazione prosodica, in M. Castagneto, M. Ravetto (eds.) «La comunicazione parlata. Atti del Congresso SLI–GSCP – Università del Piemonte Orientale (Vercelli, 5-7 maggio 2021)», Napoli, Aracne.
Saccone, V., Trillocco, S. (2022) Segmentation of the Speech Flow for the Evaluation of Spontaneous Productions in Pathologies Affecting the Language Capacity. A Case Study of Schizophrenia, in Proceedings of the RaPID-4 @LREC 2022, Marseille, 94–99, © European Language Resources Association (ELRA), licensed under CC-BY-NC 4.0.
Saccone, V., Trillocco, S., Moneglia, M. (2023). Markers of Schizophrenia at the Prosody/Pragmatics interface. Evidence from corpora of spontaneous speech interactions, in Front. Psychol., Sec. Psychology of Language, Volume 14. doi.org/10.3389/fpsyg.2023.1233176
Saccone, V. Trombetta, C. 2021. Parenthetical Units and Structures in Italian and German spoken language Prosodic and textual analysis. CHIMERA. Romance Corpora and Linguistic Studies. V.8, 1-23. https://doi.org/10.15366/chimera2021.8.001
Saccone, V., Vieira, M., Panunzi, A. (2018), Complex Illocutive Units in Language into Act Theory: an analysis of non-terminal prosodic breaks of Bound Comments and Lists in JoSS Special Issue: Spoken Corpora advances: prosody as the crux of speech segmentation, annotation and multilevel linguist, State University of Campinas; 7(2); 51-64.
Shattuck-Hufnagel, S., & Ren, A. (2018). The prosodic characteristics of non-referential co-speech gestures in a sample of academic-lecture-style speech. Frontiers in Psychology, 9, Article 1514. https://doi.org/10.3389/fpsyg.2018.01514
Shattuck-Hufnagel, S., Ren, P. L., & Tauscher, E. 2010. Are torso movements during speech timed with intonational phrases? In: Proceedings of the International Conference on Speech Prosody. ISCA Archive. 1–4.
So, W., Wong, M.K., Lui, M. & Yip, V. (2015). The development of co-speech gesture and its semantic integration with speech in 6- to 12-year-old children with autism spectrum disorders. Autism, 19(8); 956-968. DOI: 10.1177/1362361314556783
Sparaci, L. (2008). Embodying gestures: The Social Orienting Model and the study of early gestures in autism, Phenom Cogn Sci 7, 203–223 DOI 10.1007/s11097-007-9084-9
Sparaci, L., Lasorsa, FR., Capirci, O. (2019). More Than Words: Gestures in Typically Developing Children and in Children with Autism. In Grove, N., Launonen, K. (eds.) «Manual Sign Acquisition in Children with Developmental Disabilities», , NOVA Science Publishers.
Swerts, M., & Krahmer, E. 2010. Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions. Journal of Phonetics, V. 38, 197–206. https://doi.org/10.1016/j.wocn.2009.10.002
Wetherby, AM., Prutting, CA. (1984). Profiles of communicative and cognitive-social abilities in autistic children. Journal of Speech and Hearing Research, 27, 364–377.
Wing, L. (1981) Asperger's syndrome: A clinical account. Psychological Medicine 11:115–29.