Abstract Umata

Talking through Graphics

Ichiro Umata

ATR Media Integration & Communications Research Laboratories,
Kyoto, Japan

Abstract

In everyday situations, people communicate with each other by combining information from various sources, and growing number of researchers are now interested in communication with visual means. However, most linguistic researches have focused on a combination of language and gestures, and little is known about the interaction between language and graphical resources. Conversational exchanges that involve external graphical representations are fairly common in our daily lives. People give and ask directions by referring to maps, and they draw diagrams and pictures in discussing where to place living room furniture. When explaining directions to someone, one often says such phrases as "The train leaves every 10 minutes from here," pointing to, say, a Baker station icon on a map. In this case, the word "here" literally denotes the Baker station icon on the map. However, the train leaves from Baker station utterance metonymically describes the real world situation in which the train leaves from the station every 10 minutes, and the station corresponds to the icon on the map literally denoted by the word "here". Although we have no means to know what the word "here" really refers to without the map, the word is actually referring not to the icon on the map but to the Baker station in the real world. The map serves to refer to the real world object, and people see a real world situation through the map contingent on the systematic correspondency between the map and the world. This type of mediated language use in such contexts is so natural and common that people may not even be aware of the phenomenon. In this study, we conducted an empirical investigation into the kind of mediated references used in conversational interactions that involve maps and other graphical resourses. Based on an examination of our conversational data, we found that mediated references can be performed in both directions between the representation and the world. We also note that mediated references apply both to the level of individual entities and to the level of relations held among individuals. We then argue that the mediated use of language enables us to communicate effectively by providing rich reference possibilities and integrating the information of a graphic representation and the information conveyed by a linguistic expression.

Publications

Umata, I., Shimojima, A., and Katagiri, Y. (2000). An Informational Analysis of the Mediated Use of Language in Graphics Communication. In Holmqvist, K., Kühnlein, P., and Rieser, H. (Eds.), Proceedings of the Workshop on Integrating Information from Different Channels in Multi-Media-Contexts (pp. 48-55).
Umata, I., Shimojima, A., and Katagiri, Y. (2000). Talking Through Graphics: An Empirical Study of the Sequential Integration of Modalities. In Gleitman, L. R., and Joshi, A. K. (Eds.), Proceedings of the Twenty-second Annual Conference of the Cognitive Science Society (pp. 529-534). Mahwah, NJ: Erlbaum.

to SFB-Homepage

created by: Anke Weinberger (2000-09-08).
maintained by: Anke Weinberger (2000-09-08).