In this project, negation is investigated as a multimodal phenomenon. The multimodality of language – that is, extending the analysis of spoken language to speech-accompanying bodily expressions – has been of continuous interest in psychology (e.g., Barsalou 2019; Hostetter & Alibali 2008; Kendon 2004; McNeill 1992) and has seen increased attention in linguistics (e.g., Ebert & Ebert 2014; Fricke 2012; Schlenker 2018). Taking an experimental psycholinguistic perspective, we focus on the relationship between speech and gesture during the comprehension and production process in both German-speaking adults and children. Negation has been suggested as an ideal candidate for a thorough investigation of the multimodality of language (Morgenstern et al. 2018). Negation is a fundamental property of human language, which all languages express through specific lexical items (e.g., not; Moro 2015). In addition, negation and related concepts can be communicated beyond verbal expressions, including recurrent manual gestures (e.g., Bressem & Müller 2014), head shaking (e.g., Kendon 2002), and facial expressions (Benitez-Quiroz et al. 2016). However, there remains a need for a systematic experimental investigation of nonverbal negation and its interactions with verbal negation. In the first two work packages, we address comprehension aspects of nonverbal negation from the listener’s perspective. We will investigate the types (RQ1) and functions (RQ2) of nonverbal negation and their integration (RQ3) with verbal negation in both adults and children. In the third work package, we aim to open our investigations to the speaker’s perspective and investigate aspects of adults’ usage (RQ4) of nonverbal and verbal negation to identify the contexts in which verbal and nonverbal negation is used. Within the CRC, our project belongs to Domain C and aims to contribute to the question of whether there are universal cognitive processes in the processing of negation (QC.1). This line of research can shed light on the question of whether negation is grounded in nonverbal cognitive domains or whether negation has an abstract linguistic representation separate from other cognitive domains. Such line of evidence can indirectly inform the Neg-Plus versus the Neg-Only hypothesis by specifying which cognitive processes are involved in negation processing and inform us regarding the question of whether negation must be understood as a multimodal phenomenon including categorical (speech) and imagistic (gesture) components or whether negation can be reduced to categorical components. More broadly, our case study on negation can contribute to overarching issues within the study of multimodal language, which still await a conclusive answer.
In summary, our three work packages investigate negation as a multimodal phenomenon from a psycholinguistic and cognitive-oriented processing perspective. At the end of the first potential funding phase, we aim to have established for a defined set of nonverbal negation markers how they fit into existing linguistic and processing models for multimodal language. In the second funding period of the CRC, we plan to extend this line of research to pictorial negation to investigate whether gestures have a privileged status in interaction with verbal negation or whether other non-verbal forms can be integrated as well. In the longer perspective, we also plan to study clinical populations to investigate whether certain cognitive systems (e.g., inhibitory system) - well-known to be impaired in certain non-neurotypical populations - play a specific role for verbal and nonverbal negation comprehension (e.g., Dudschig et al. 2021).
