8 Pedagogical Agents in Multimedia Digital Science Storybooks

Faculty Mentor: S. Claire Son (Educational Psychology, University of Utah)

With the rising use of electronic devices, multimedia digital storybooks are one way that children can learn vocabulary that is not a part of everyday or content language (e.g., science vocabulary). These digital storybooks utilize digital enhancements (e.g., animation, contingency, interactivity, etc.), and their multimedia characteristics offer opportunities for enhancing early learning through promoting dual coding and reducing the effort to match verbal and nonverbal/visual information. However, overall evidence on the effectiveness of multimedia features of digital storybooks is mixed. Some studies highlight positive effects of multimedia enhancements on learning (Son et al., 2020; Son & Butcher, 2024) while others suggest learning disruption by additional digital features through depleting information processing resources (Bus et al., 2015). Further, their effects on child learning and comprehension are not uniform, often depending on child characteristics (e.g., age, vocabulary skills, working memory) (Choi et al., 2021; Kirkorian et al., 2022; Son et al., 2020; Son & Butcher, 2024).

Among several digital features, pedagogical agents — interactive guides or tutors designed to support children’s learning, comprehension, and engagement in digital storybooks — have been found to support learning within multimedia environments (Atkinson, 2002; Dunsworth & Atkinson, 2007). These agents can appear differently with different roles (e.g., instructor role, social role, etc.) and behaviors (e.g., pointing, teaching, encouraging), which is hypothesized to affect young children’s vocabulary learning. To reflect the variety of roles pedagogical agents can take, the current study explored pedagogical agents taking the roles of a teacher and a character. Agents taking a teacher role would help attentional focus by pointing and gesturing to relevant information in the story while agents taking a character role would be interactive and social by sharing experiences and engaging with the reader.

The current study investigated the impact of pedagogical agents on early science word learning. We expected that children in either agent condition (e.g., teacher agent or character agent) would demonstrate greater science word learning than those in the no agent condition because of attentional support and teaching accompanied by the visual presence of the agent. We also expected that children in the character agent condition would demonstrate greater science word learning than those in the teacher agent condition because of cognitive engagement and motivation from the agent’s interactive prompting and feedback.

A subset of preschoolers (N=99) for a larger study on science learning participated in the current study. These preschoolers completed reading a bird beak digital storybook at the time of the study. Participating children were from eight Head Start sites in the Mountain West region. Preschoolers listened to a multimedia digital storybook on the different types of bird beaks (e.g., cracking beak, hunting beak, and drilling beak), their structural characteristics (e.g., cracking beaks are short and wide), behaviors (e.g., a cracking beak cracks and crunches nuts and seeds), and functions (e.g., using beaks to capture and eat food) individually on a touch-screen device. Before listening to a digital storybook, children were tested on their initial skills in picture vocabulary (general vocabulary) and science vocabulary, using the Woodcock Johnson IV Test of Academic Achievement battery. Based on their scores, children were split into low versus high skill groups (i.e., mean split) for each test, and children in each skills group were randomly assigned into reading conditions (i.e., stratified random assignment) to make sure each reading condition had a mix of children with differing science and vocabulary skills. Children were randomly assigned to one of three conditions of the digital book reading, each differing by the presence/type of pedagogical agent: no agent (see Figure 1), teacher agent (see Figure 2), or character agent (see Figure 3). For the teacher agent condition, the teacher agent was located out of the animation frame and worked as an out-of-story instructor, teaching and pointing to direct attention. For the character agent condition, the character agent was located within the animation frame and worked as inside-the-story social character asking questions and providing feedback. After listening, preschoolers’ science vocabulary skills in bird beaks were tested using an in- house measure of breadth of vocabulary (receptive and expressive vocabulary) and depth of vocabulary (concept understanding of vocabulary words).

Comparison of post-test science vocabulary skills across conditions was analyzed using R. Results showed trends of relatively high vocabulary scores of children in the character agent condition, but statistical tests demonstrated no significant differences in post-test scores of total science vocabulary (see Figure 4a), depth of vocabulary (see Figure 4b), or breadth of vocabulary (see Figure 4c) across the three agent conditions. Interaction analysis was conducted to examine potential interaction effects between child initial science vocabulary and agent condition in predicting science vocabulary in bird beaks. Results revealed no significant interactions in predicting total science vocabulary, depth of vocabulary, or breadth of vocabulary scores.

These results indicated that overall, the presence of a pedagogical agent does not make a significant difference in preschoolers’ science vocabulary learning. The presence of an agent by itself may not be enough to promote science vocabulary learning. These agents might not be engaging enough on their own, which is why no significant differences were seen across the three agent conditions. Future studies should test additional child characteristics like attention or engagement when interacting with pedagogical agents in digital storybooks. Children in the current study would interact differently with agents with some children responding to prompts/questions or pointing to relevant parts of the screen. However, the current study did not measure their engagement. For learning vocabulary, interacting with pedagogical agents could be more important than just the presence of pedagogical agents. Ultimately, understanding what features in digital storybooks foster children’s early word learning can guide the creation of new educational materials and planning of vocabulary instruction.

Bibliography