The coordinated interplay of scene and utterance in real-time comprehension

Matthew W. Crocker　

Abstract:
This paper outlines a range of recent empirical and computational modeling results concerning the role of the immediate scene during spoken language comprehension. Evidence from a series of eye-tracking studies will be presented which demonstrate that our attention in the immediate visual scene rapidly reflects real-time language processing. In particular, anticipatory eye-movements in the scene reveal the rapid on-line influence of prosody, morpho- syntax, lexico-semantics, and world-knowledge on utterance interpretation and disambiguation.
Further, evidence from both eye-tracking and event-related potential studies will be presented which demonstrate that information extracted from the scene can in turn immediately influence linguistic disambiguation. Further, it will be argued that scene information may have priority over stereotypical knowledge, when the two sources of information are in conflict.
On the basis of our findings, I will argue for the existence of a highly synchronized "coordinated interplay" mechanism underlying situated scene-utterance comprehension. I will then conclude with a presentation of recent connectionist modeling results which aim to account for this human behavior, and will speculate on the possible origins of the mechanism based on theories of language acquisition.