Multimodal Integration

Abstract— Learning by demonstration and,learning by instruction offers a potentially more powerful paradigm than programming,robots directly for specific tasks. Learning in humans,or primates substantially benefits from demonstration of actions or instruction by language in the appropriate context and there is initial neurocognitive cortical evidence for such processes. Cortical assemblies have been identified in the cortex that activate in response to

Multimodal signalling can improve or maximize information exchange. A challenge is to show that two independent signals, such as vocalizations and visual displays, are deliberately coordinated. Male zebra finches, Taeniopygia guttata, signal visually and acoustically during courtship, performing a stereotyped dance while singing. The male approaches the female hopping in a zigzag pattern, turning his body axis, and wiping his beak repeatedly on or above the perch. The only previous quantitative study of song and dance choreography in zebra finches revealed that the distribution of all movements during song was not strongly patterned across birds but very similar in fathers and sons. This raises the possibility that particular movements may follow a choreography. Here we report that three operationally defined dance movements, 'beak wipe' (BA), 'turn-around' (TA) and 'hop', occurred with different frequencies and speed during singing than during silence. BW, TA and hops clustered significantly at the start and end of song bouts and were arranged in a nonrandom fashion. In addition, BW, but not TA, were performed faster during song than nonsong. Finally, hops coincided significantly more often than expected by chance with particular notes. Together, these results suggest that male zebra finches integrate their song and dance during courtship. This may help females to identify courting males in a noisy environment and evaluate the intensity and quality of the courtship performance. Our results underscore that the choreography of movement gestures with learned vocalizations, such as hand gestures accompanying speech, is a further parallel between human and avian signalling. They invite future investigations into the underlying neural mechanisms and consequences for mate choice.