Emotion

Latest research suggests that not only does the Companion need to understand its user's emotions but also that it needs its own emotions to enable it to understand the world and motivate its behaviour.

During the course of a conversation people constantly monitor how they are doing and adapt their strategies to fulfil their current goals. To capture this feature on the Companions project we have to work out how our agents can judge a users emotional state and adjust their own. Humans have a wide range of underlying motivations but for a Companion its goals are always focused on the user: it needs to keep them happy while reconstructing their life narrative. To do this it needs to detect when a user isn't happy and adjust its own behaviour in response, experiencing positive 'emotions' when it succeeds.

Given the challenges in deriving emotion from speech we are taking a multimodal approach, unifying emotional information from both the tone and the content of the users utterances. By maintaining a model of both the users and its own personality our Companion can learn to produce utterances, rich with emotional content, that are specifically designed to reassure and encourage the user. By adapting to the user's personality in this way the Companion develops its own promoting an increasingly natural range of conversations.

Embodied Conversational Agents (ECAs) such as the Companion have been the dominant framework for research in affective interfaces because of the anthropomorphic communication situations they recreate. Three main determinants of affective dialogue with ECAs can be identified:

1. Social use of language
2. Task performance and appraisal
3. ECA personality and relation to the user

Humour is an important feature if ECAs are to become more believable and are to establish a persistent relation to the user. It can be seen as a communicational ability deeply rooted in the linguistic abilities of the dialogue system. Previous research in computational models of humour has evidenced the role of semantic categorisation and semantic relations between terms involved in the generation of the humorous effects.

This can be further refined, as in dialogue there is a specific expression of witticisms and punchlines, which has been specifically investigated in the context of ECAs for interactive story telling by Cavazza and Charles (2005) who have shown that witticisms in dialogue could be generated by contrasting the semantic contents of an utterance and its reply.

Politeness is another important topic for affective ECAs, and like previous topics it is strongly related to Multimodal Dialogue through various aspects such as use of language, non-verbal behaviour, dialogue strategies, etc. Andre et al. (2004) have presented a new dialogue model that integrates a social theory of politeness with a cognitive theory of emotions.

The rationale they put forward is that the persona effect would generate expectations on the social behaviour of ECAs, politeness being part of such behaviour. The system's strategy is to decrease the perceived threat posed by a speech act by using politeness strategies whose extent is determined by an assessment of the user's emotional state.

ECAs are expected to carry out tasks for which they are instructed and sometimes appraised (through direct or implicit feedback) by the user. Emotional theories such as the OCC model have been successfully linked to task performance, in particular, anticipation of plan failure (Gratch and Marsella, 2001). Whenever ECAs must carry out tasks for which they have to generate an appropriate sequence of actions, such an emotional representation would be relevant.

The study of Marsella and Gratch (2003) on 'coping' in ECAs has defined coping as a response to the appraised significance of events, and has distinguished between problem-focused coping and emotion-focused coping. The latter could open the way to a variety of communication strategies that should be reflected in the dialogue content for the ECA, arguably at several levels of dialogue modelling, from individual utterances to dialogue strategies.

The concept of empathy constitutes a high-level characterisation of the interaction experience as well as a description of a form of relation between the user and an ECA. It has been explored from both perspectives, ie ECA inducing empathy (eg the FearNot! system, Paiva et al., 2005) and agents demonstrating empathy towards the user.

Cavalluzzi et al. (2004) have shown how dialogue strategies of an advice-giving ECA could generate empathy. In their system (giving advice about addiction situations), they use a categorisation of the users' 'state of change' (eg, contemplation of change, preparation of change, action, maintenance, relapse...) to determine the best response to be given by the ECA. This approach has been integrated into the 'Information State model' of the TRINDI Dialogue environment.

In addition, part of the believability of an affective ECA derives from its being associated with a consistent personality. Personality models have been related to various abilities of ECA, from use of language (see above) to display of emotions. For instance, Egges et al. (2003) have related the OCEAN personality model to OCC parameters in order to animate ECAs with specific personalities.

Overall, there appears to be consensus in the literature that the same mechanism we have identified for affective ECA (politeness, humour, etc.) should play a central role for the establishment of trust as well, in addition to some specific dialogue control mechanisms aimed at maintaining the relation between the user and the ECA.

Bickmore and Cassell (2001) have carried out specific research on the establishment of trust in ECAs. They have identified three determinants of trust between a user and an ECA: familiarity, solidarity and 'knowledge of trust behaviour'. Each of these can be supported by specific conversational strategies (building common ground supports familiarity; avoiding face threats supports solidarity...), and the set of conversational strategies can be incorporated into a generic dialogue feature of 'small talk'.

Rees and Nass (1996) have shown in various experiments that users liked ECAs that matched their own personality and that users responded better to agents that would demonstrate humour, or agents that would flatter them. In addition, there seems to be a relation between ECA politeness and the perception of their trustworthiness by the user, albeit not always as it could be anticipated: in some specific settings, users' trust in an ECA can be adversely affected by a (probably excessive) display of politeness by the agent (Nass and Lee, 2000; Rickenberg and Reeves, 2000).

Traditional computer models of language have seen language as a conduit for meaning. It seems computers are often treated as social actors (Rees and Nass, 1996) and so, presumably, can use humour, gossip, story telling and other forms of casual conversation to build trust between the human and a conversational machine.

Companions will extend existing computational models of dialogue management to include casual conversation, small talk, entertainment, games and diversion.