Pointing and grasping in concert

Hannes Rieser

Fakultät für Linguistik und Literaturwissenschaft
Universität Bielefeld

Abstract
In my talk the focus is laid on the denotional function of demonstrations affiliated to referring expressions and to some lesser extent on graspings. The referring expressions are mostly definite descriptions like "the yellow bolt" or simple demonstratives accompanied by a pointing gesture. However, both pointings and graspings matter because the empirical data on which the talk is based deal with object identification games, i.e. task-oriented dialogues of a special sort, where a description giver singles out an object with some description plus a demonstration/pointing and the object-identifier tries to identify the object. He frequently does so using a clarification question or a check-back such as "This one?" grasping at the same time a particular object, bolt, nut, disc or whatever. This shows that grasping and reference are linked just as pointings and reference are. In addition, object identification games are also prototypical examples of situated communication, the investigation of which has been one of the main objectives of the SFB 360.

So much for providing some general context. At the beginning I explain the title of the talk, its allusion to agent?s coordination in a sort of alternation and I also point out that the whole research reported is linked up with the concept of Embodied Communication, especially the VR-agent Max. Ensuing, a short overview of Cognitive Science approaches to gesture is provided. There is a very brief encounter with Peirce, Quine, Wittgenstein and Heidegger. Empirical data from the object identification games as introduced above are presented showing that demonstration is part of natural language. Then different functions of demonstration such as pointing to objects, regions and directions are distinguished, based on annotated and rated empirical data. A multi-modal interface integrating the information coming from the gesture channel and the verbal channel is laid out, where special emphasis is put on the underspecification properties of multi-modal content. Residing to recent SDRT, it is shown how the information from the multi-modal interface can in principle be integrated into dialogue theory. Finally, an appraisal of the interdisciplinary work consisting of informatics and linguistics is given.

Bibliography
Kranstedt, A., Lücking, A., Pfeiffer, Th., Rieser, H. Wachsmuth, I.: 2005, Deixis: How to Determine Demonstrated Objects. In: Proceedings of the Gesture Workshop'05, to appear with Springer Verlag.
Kranstedt, A., Lücking, A., Pfeiffer, Th., Rieser, H. Wachsmuth, I.: 2006, Deictic Object Reference in Task-oriented Dialogue. To appear in Rickheit, G. & Wachsmuth, I. (Eds.) (2006). Situated Communication. Mouton de Gruyter: Berlin, New York.
Lücking, A., Rieser, H., & Stegmann, J. (2004). Statistical support for the study of structures in multimodal dialogue. In Catalog '04. Proceedings of the 8th Workshop on the Semantics and Pragmatics of Dialogue. Univ. Pompeu Fabra, pp. 56-64.
Poesio, M. and Rieser, H. (2005). Prolegomena to a theory of Completions, Continuations, and Coordination in Dialogue. MS, submitted.
Poncin, Chr. and Rieser, H. (2006). Multi-speaker Utterances and Co-ordination in Task-oriented Dialogue. To appear in Journal of Pragmatics 751, pp.
Rieser, H. (2004). Pointing in Dialogue. In Catalog '04, op. cit. pp. 93-101.

sfb-logo Zur Startseite Erstellt von: Anke Weinberger (2005-11-07).
Wartung durch: Anke Weinberger (2005-11-07).