Toward Believable Acting for Autonomous Animated Characters

An acting performance operates on many levels simultaneously, from its physical qualities to its psychological ones. Thus we have found it helpful to break down the monolithic concept of believability into a set of more specific design principles, each of which is more tractable to analysis and simulation. This list is tailored to the domain of animated acting, combining insights from the fields of psychology, animation, and human-computer interaction, and in particular draws inspiration from Gomes et al. [2013], who also decomposed believability into a set of individual attributes. (For a detailed comparison, see Appendix A.)

For a character's acting performance to be believable, it should demonstrate the following principles:

1. Coherence of identity: maintains consistent characteristics over time so as to appear to have a single identity. A common violation of this principle is to allow an entirely different algorithm to take over behavior in certain situations, such as when an NPC abruptly delivers pre-scripted dialog, or a toy robot's persona is replaced by a virtual assistant.
   
2. Physical movement: obeys physical laws consistent with the rules of its world. Note that the rules of the character's world may differ from the rules of our world. This is acceptable so long as those rules are internally consistent.
   
3. Biological movement: moves as if supported by a biological structure such as rigid bones with flexible joints and muscles. The human visual perception system is primed to recognize biological movement [Johansson 1973] which makes this quality especially important for animated characters. Animation principles such as squash and stretch, arcs, slow in and out, follow through and overlapping action [Thomas and Johnston 1981] all exist primarily to support the perception of biological movement.
   
4. Self-propulsion: appears to move on its own initiative, rather than only being pushed or pulled by external forces. Self-propelled entities are perceived by infants as agents, and their actions are perceived as goal-directed [Luo and Baillargeon 2005]. A common violation of this principle is a marionette whose movements are obviously controlled by a puppeteer pulling strings. Skillful puppetry involves creating an illusion of self-propulsion so compelling that the viewer forgets the puppeteer and strings exist.
   
5. Contingent interaction: reacts in a timely way to events in the world. Humans, even infants, attribute mental states, particularly perception and goal-directedness, to objects if they react contingently to their environment [Johnson and Ma 2005]. A common violation of this principle is when an autonomous character plays back a fixed animation clip for some time period, during which it cannot respond to interruptions of any kind.
   
6. Self-motivation: appears to act on its own internal motivations, and pursue goals to satisfy its needs. Goal-oriented behavior is recognized in psychology as a precursor for perception of agency and animacy, even in infants [Gergely and Csibra 2003]. More complex characters should also be capable of concurrent pursuit of multiple goals [Loyall 1997].
   
7. Attention: appears to pay attention to relevant entities in the world around it. This includes attention appropriateness, i.e. that the object of the character's attention is something relevant to it, and attention legibility, that this fact is displayed in a way that the viewer can perceive. Attention provides a powerful tool to communicate a character's inner state to the viewer—what it can perceive, is thinking about, or considers important. Timely changes in attention also provide an opportunity to demonstrate contingent interaction.
   
8. Emotion: clearly conveys emotions appropriate to the viewer's understanding of its circumstances, motivations, and goals. This too requires both appropriateness and legibility. Note that multiple aspects of a character's behavior can convey information about its emotional state: its choice of actions (or inaction), modulations of those actions (e.g. speed of movement), or explicit display of emotional affect (e.g. posture or facial expression).
   
9. Explainability: behaves in a way that allows the viewer to form an explanation consistent with the observed evidence. This principle applies to a performance taken as a whole: it is not necessary that every action be completely explainable the moment it occurs—in fact, the mystery can deepen the viewer's engagement in the experience—as long as the viewer can retroactively explain it later [Gaver et al. 2003]. For this principle the timing and sequencing of events is important: for example, an action will be more explainable if preceded by a relevant shift in attention. A common violation of believability occurs when a character reacts to an event that it didn't perceive.
   
10. Thought: displays evidence of thinking in a biologically plausible way. By thinking we mean activity that is purely mental, such as remembering, considering, understanding, planning or deciding. Overtly portraying such thought processes helps convey a sense of inner life, and is especially important for characters from which the viewer expects any degree of intelligence. Thinking can also affect the character's attention and emotions: a character that thinks may attend to its thoughts, and show changes in emotion contingent on those thoughts. The timing of such changes should reflect the viewer's expectation of how long they would naturally take. A software-based character can violate this principle in either direction, e.g. by reacting instantly to stimuli that it should need some time to understand, or by pausing for an implausibly long duration to plan movement.
    
11. Sociality: senses and acts on inferences about the beliefs, desires, and intentions of other beings. Humans are profoundly social creatures and reason about other beings with an intentional model [Lieberman 2013]. We therefore expect a relatable character to reason and act similarly towards us or other agents in its world.
    
12. Personality: has unique behavioral and expressive traits that distinguish it from its peers as an individual. The dramatic arts, particularly writing and acting, stress the importance of unique behavior to prevent the character from being seen as ‘stock’ or unconvincing—and designers of believable characters would be well served to think about opportunities to express uniqueness even in the details of movement [Loyall 1997].
    
13. Change with experience: behaves in a way that shows the effect of its past experiences. By change with experience we do not mean merely acquiring information about the world, but rather change in behavior patterns due to experiences relevant to the character's motivations. Such behavior change suggests that the experience was meaningful to the character and creates room to read emotions into the experience. Even if the viewer did not witness the event that caused the change, the behavior itself is often sufficient. Artificial agents that change from experience are hypothesized as more life-like and shown to engender greater empathy in humans [Darling et al. 2015].
    

This list of principles is not a recipe, but rather a set of tools for guiding one's choices when designing and animating an autonomous character. The principles do not all apply equally to every character: each one is only necessary to the extent that the viewer's expectations require it. For example, a mechanical robot might set low expectations for biological movement, and a photorealistic squirrel may be perfectly believable without evidence of thought.

Our system is comprised of two main parts: a brain trained with RL to choose a desirable course of action (Section 5) and a procedurally animated body which produces a continuous stream of expressive, biophysical movement (Section 6). The brain consists of a drive-based intrinsic reward system, an action policy and set of value functions produced by RL, and emotion and attention modules that rely on the policy and value functions. The body provides the brain with observations about its environment, and the brain sends command signals to the body (see Figure 1). This cycle of observation and command occurs at a frequency of 5Hz, analogous to the traditional animation timing scheme of a handful of “beats” per second [Thomas and Johnston 1981].

Our character lives in a simple 3D landscape (Figure 2) with gentle hills, treacherous cliffs, and rocks and other obstacles that can obstruct its view and movement. Fruit grows on trees and bushes, falls from them, and rolls down the hillsides.

Our character's world also contains critters, which are simple pest-like creatures programmed to compete for food and attack or flee when threatened, and the user, who can help or harm the character by petting, feeding, or poking it.

The 3D character named Axo is a cute quadruped made of disconnected primitive shapes, about the size of a house cat, with big round eyes, an expressive headdress, and a broad tail. Its terrarium-like habitat is stylized but pleasant, featuring soft hills, steep cliffs, a waterfall, and fruit-bearing trees and bushes. Its enemies, the Critters, are small tetrahedra.

A character's visual design sets the viewer's expectations for its capabilities. If the character resembles a familiar species, it activates prior beliefs about similar animals. Also, more detailed designs set higher expectations of realistic movement. Humans excel at filling in missing details: viewers can perceive a complete walking figure from just a handful of moving dots [Johansson 1973]. However, viewers are far less tolerant of conflicting information [Chaminade et al. 2007; Saygin et al. 2012]. Thus, the more complex the character design, the greater the risk of breaking the viewer's suspension of disbelief.

For these reasons, we opted for a simplified, stylized visual design. Our character, Axo, is a quadruped made of primitive shapes, with rudimentary features designed to convey only the most basic information through posture, movement and expression (see Figures 2 and 3). This ensures that all information shown to the viewer is entirely under our system's control, and sets the viewer's expectations appropriately, increasing the likelihood that the character's behavior will meet or exceed them.

An important step in the design of a believable character is choosing what should matter to it, and therefore what should drive its behavior. This corresponds to the principle of self-motivation. Inspired by Maslow's Hierarchy of Needs [Maslow 1943], we defined a fundamental set of motivational drives, closely related to the Hullian drives described by Konidaris and Barto [2006]. These drives organize the character's appraisal of events and states of its world. Specifically, the reward we designed for RL to maximize is solely dependent on the values of these motivational drives, making them the foundation of the character's behavior.

Our design is inspired by a refinement to the classic agent/environment RL paradigm that was introduced by Barto [2013, Figure 2], in which an “organism” contains an RL agent and an internal environment, and this internal environment interacts with an external environment. In this paradigm of intrinsically motivated RL, reward is produced as a combination of internal and external signals and events. In our implementation, the internal environment includes both the body layer and the brain layer (excluding for the RL agent itself) in Figure 1, and the external environment is the world.

Our current system includes three basic drives: nourishment, safety, and rest. We chose these drives because of how relatable and legible they are to the user in a simple environment, as well as for how the interplay and conflict between them can result in complex, interesting emergent behavior and emotional expression.

We define D to be the set of all drives. The mechanics of each drive d ∈ D, described as a tuple (p_d, α_d, U_d), determine how to calculate a corresponding drive value d_t ∈ [0, 1] at time t. In the absence of relevant stimuli, a drive value slowly regresses to its set-point p_d by increasing or decreasing by α_d at each time step. Drive values are also affected by events in the environment (e.g. nourishment increases upon eating) according to each functional rule u in the rule set U_d for that drive (see detailed instantiations of rules in Appendix C.1). The update of a given drive value d_t at time t, as a function of the state s and action taken a, is defined as follows:

In summary, a character can increase or decrease its reward by acting to influence the probability of various drive-relevant events occurring, which in turn increases or decreases its drive values. These changes in drive values change the character's reward.

To demonstrate the principle of emotion appropriateness, the character's displayed emotions must be grounded in its current subjective experience. And to demonstrate attention appropriateness, it must observably attend to whatever is most relevant to its current emotional state and behavior. When juxtaposed against the character's choice of actions, these two observable details combine to support the principle of explainability (see Figure 3). Thus, in addition to sending action commands to the body, the brain must also produce emotion and attention signals.

A comparison of three similar scenarios in which a character is walking toward a blue sphere and away from a green cube. In the first scenario, the character displays no emotion and is not looking directly at either object. In the second, the character looks happy and is looking at the blue sphere. In the third, the character looks afraid and is looking at the green cube.

The emotion system generates three types of emotions. Immediate emotions reflect the current state of each motivation drive and its reward component. Anticipatory emotions predict future reward for each motivational drive (e.g. safety anticipation will be low when danger is near). Lastly, surprise emotions are generated from the derivatives of the anticipatory emotions, enabling the character to react to sudden changes in its prospects. For a detailed description of how these signals are generated, see Appendix E.

Prediction of future reward, or value, has previously been used to provide anticipatory emotion signals [Jacobs et al. 2014], but structuring our reward as a sum of motivational reward components enables improvements upon such past work. Using the same data that is generated to train a behavior policy, the brain concurrently trains a drive-decomposed value function to predict cumulative future reward for each individual motivational component. This drive-wise prediction provides more specific signals for anticipatory emotional expression than would a single value function.

The attention system identifies the most emotionally salient object for each drive by first iterating through all visible objects and evaluating the drive-decomposed value function with a counterfactually generated observation where the corresponding object is removed. The object that, when removed, results in the largest change becomes the referent for the corresponding anticipatory emotion. The referent with the largest change is then chosen as the character's attention target.

Following the principles of sociality and change with experience, we designed some opportunities for basic interaction between our character and the user.

We detect the user's presence and head position using the device's built-in camera and mirror them with an avatar at the edge of the environment. This places the user in the virtual space, as if the screen was their first-person view.

The user is thus observable by and can interact with the character, either positively by petting and feeding it, or negatively by poking it, via touchscreen gestures. Besides their immediate impact on the character's drives (feeding increases nourishment, poking decreases safety, etc.), the system tracks (and includes in the observation) a scalar statistic of these interactions, called the user score. This is a heuristically estimated quantity, based on occurrences of negative and positive interactions. This allows training the RL policy with simulated users, each assigned and behaving according to a fixed score from a wide range of potential scores. This prepares the character to adapt its behavior to the changing score of the real user at runtime.

Despite its simplicity, this approach was sufficient to engender basic reasoning about the user and for primitive social behaviors to emerge (see Section 7).

The RL observation vector consists of both information about the character's internal state such as the values of its motivational drives, and sensed elements of the environment such as relative positions of visible objects as well as navigability of the terrain. For more detail see Appendix B.

The RL policy outputs an action vector of six real-valued numbers. The first two elements are the chosen forward and turn accelerations. The remaining four values represent discrete action commands (grab, drop, eat, and roar). The discrete action corresponding to the highest value is selected, unless none exceed a given threshold, in which case no discrete action is selected. The procedural animation system receives the action command and attempts to execute it in the world; whether it is successful or not depends on the state of the world and preconditions of the action (e.g. objects that are out of reach can not be picked up).

The policy is trained using the Proximal Policy Optimization RL algorithm [Schulman et al. 2017] using parameters and minor modifications specified in Appendix D. Our training infrastructure [Espeholt et al. 2019] coordinates several thousand parallel agents, each running episodes that last four minutes in simulation time but significantly less wall-clock time. A full training run takes one to four wall-clock hours and represents approximately 55,000 simulation hours of experience. Given our choice of algorithm and architecture, this training duration is the empirical threshold at which the performance plateaus and we consider the policy ready for evaluation (see Section 5.6).

Training episodes are instantiated from a curriculum of five to ten episode templates. Each template describes the initial configuration for a stochastically instantiated scene—including landscape features, food sources, and motivational drive values—designed to expose the RL agent to salient scenarios that accelerate its acquisition of useful behaviors and increase its robustness to environment variations.

Thus our workflow for developing an autonomous character replaces direct behavior scripting with the more high-level task of authoring training scenarios. Rather than narrowing the situational space and explicitly describing every expected behavior within that space, we elicit the emergence of complex behavior autonomously, by describing a set of training scenarios and allowing the RL agent to learn behavior within this curriculum.

Once trained, we validate that the brain's action choices and emotional reactions function as expected by running the RL policy through a series of tests. Some tests pass or fail based on whether certain conditions are met, such as eats when hungry, escapes canyon, gets surprised by critters, and avoids falling from cliff. Other tests, which we call emotional calisthenics,³ are more subjective in nature: we place Axo in a situation designed to trigger an interesting sequence of behaviors, and we visually evaluate the believability of the resulting behavioral performance as a whole. (See examples in the supplemental video.)

The brain's continuous movement signals can specify a range of movement from fast and intense to slow and subtle. The locomotion system must capture the nuance these signals provide, translating them into plausible locomotion that demonstrates self-propulsion as well as physical and biological movement.

Our locomotion system works similarly to that found in many game engines: it is rooted in a base node that traverses the virtual world's navigation mesh based on the brain's continuous movement signals, which smoothly control its forward acceleration and Y rotation velocity. A gait system procedurally animates Axo's feet, rib cage, and pelvis relative to this base node, with timing driven by a phase value based on distance traveled. As the base node's velocity changes, the system blends between gaits, resulting in smooth transitions, e.g. from walk to trot to gallop. Each gait is hand-crafted by an animator to caricature the weight shifts and physical relationships between body parts, creating the illusion that the character's own muscles are driving it, supporting self-propulsion and biological movement. The feet automatically conform to the underlying terrain to ensure solid ground contact, supporting physical movement.

The attention system supports the principle of attention legibility by translating the brain's attention signals into continuous movement designed to clearly convey to the viewer the character's focus of interest at all times: when the brain's attention shifts to a new referent object, it triggers a blink animation on the character's eyes, and the head turns crisply to follow the new referent.⁴ The timing of this action draws the viewer's eye to the moment of change.

The brain's discrete action commands grab, drop, and eat trigger procedural head animations that perform the respective actions, if conditions allow. (For example, a grab command will only result in an action if an object is within reach, and an eat command will only take effect if Axo is holding food.) The roar command triggers animation on the character's entire body and face, and emits a signal to scare away critters.

The emotion expression system supports the principles of emotion legibility and explainability by translating the brain's emotion signals into human-readable acting via salient changes in posture, gait, and facial expression.

The brain produces nine real-valued emotion signals, consisting of an immediate, anticipated, and surprise value for each of its three motivational drives (safety, nourishment and rest). The body's emotion expression system filters these raw signals and produces triggering events when they cross certain thresholds. These filtered signals and triggers are used to control a finite state machine consisting of different emotion adverb states (described in more detail in Appendix F).

Some emotions are only legible when they have the stage entirely to themselves. For these emotions, we create emotion verbs, which are discrete, time-bounded actions that override other actions and expressions throughout their duration. For example, the surprise emotion verb, triggered by the brain's surprise signals, causes the character to stop whatever it's doing, blink and stand up alertly for one second before continuing about its business.

The locomotion, gait, attention, and emotion systems run concurrently, resulting in a single, unified performance that continuously expresses the brain's motivational state. Driven by this performance are simple physics-based spring joint chains for the character's headdress and tail, which provide automatic overlapping action and follow-through, reinforcing the principle of physical movement. These flexible appendages also support self-propulsion by making it clear that the character's body is what causes them to move.

There is as yet no established measure of believability for synthetic character animation. Nor is there broad consensus on benchmarks for evaluating interactive synthetic characters of any kind [Fitrianie et al. 2020]. Thus we lean on existing tools to serve as a proxy. Specifically, in our user studies we applied two instruments from the fields of human-robot interaction and psychology: the Godspeed questionnaires [Bartneck et al. 2009], which benchmark experience with and perception of synthetic characters (particularly used with social robots in HRI), and Dimensions of Mind Perception [Gray et al. 2007], which measures the degree to which subjects perceive an entity (human or otherwise) as having a mind.

We conducted a user study with 54 users, 80% female, with diverse backgrounds, recruited through a screening process designed to be inclusive and equitable. Data from 50 completed surveys was collected under informed consent. Respondents ran an interactive application featuring Axo in its native habitat (Figure 2). Users were instructed to run the application at full screen on laptop computers in their homes for at least 60 minutes. We did not specify how actively users should engage with the app, allowing them to control where their experience lay on the spectrum from passive to interactive. Users then responded to a survey with 32 items, including seven-point differential scale prompts on multiple constructs (anthropomorphism, animacy, likeability, and perceived intelligence) as part of the Godspeed questionnaires, as well as prompts for seven-point ratings (“strongly agree” to “strongly disagree”) on mind perception dimensions, including mental and emotional capacities. (See Appendix G for the full survey.)

Results from the Godspeed questionnaires are plotted in Figures 4 and 10. The character generally received its highest scores on the animacy and likeability Godspeed questionnaires. On every semantic differential of these two questionnaires, at least $75\%$ of users gave ratings at or above the midpoint. The other two Godspeed questionnaires—on anthropomorphism and perceived intelligence—nonetheless also tended to have more favorable than unfavorable responses: on every semantic differential, at least $50\%$ of users gave ratings at or above the midpoint.

A chart showing questionnaire responses for five semantic differentials on a 1 to 7 scale: Dead/Alive (mean value 5.83), Mechanical/Organic (mean 4.62), Inert/Interactive (mean 4.73), Apathetic/Responsive (mean 4.81), and Stagnant/Lively (mean 5.46). This data appears in tabular form in Table 2.

A chart showing responses to questions about Axo's ability to experience various emotions, on a 7-point scale from "strongly disagree" to "strongly agree". For fear, hunger, joy and pain, about 80 percent of respondents chose "somewhat agree" or higher. For embarrassment, only 20 percent did.

Figure 5 shows some results from users’ evaluations of the dimensions of mind perception. A majority of users agreed or strongly agreed that the character experiences fear, hunger, joy, and pain. We also asked about embarrassment, an emotion we did not explicitly design the character to exhibit and would not expect to emerge without a drive or other mechanism related to social inclusion; users correspondingly tended to disagree or express neutrality regarding the character feeling embarrassment.

Participants were also asked to describe Axo in their own words. Of the responses that included pronouns, 41% used the pronoun it, whereas 59% used personifying pronouns such as he, she or the singular they, a behavior consistent with the perception of an animal as someone, not something [Wales and Siemund 2009].