2/5/2025 | 11:00 AM - 12:20 PM | Grand Peninsula D

bestie: An immersive, interactive, intelligent storytelling companion

Author(s)

Tyler Bell | University of Iowa

Abstract

We present bestie, an innovative AI-driven interactive and immersive storytelling application primarily designed to foster creativity and exploration in children. Leveraging multimodal large language models (LLMs) and image generators, bestie uses both text and speech to facilitate natural interactions with the user. The LLM serves dual purposes: (1) enabling engaging conversations with the user and (2) collaboratively generating interactive stories. Users communicate verbally with bestie using their voices to make decisions that affect story progression, alter details, and ask questions in real-time. As bestie drives and narrates the story, it also generates spatial images corresponding to scenes and characters from the story. These images are presented as 3D meshes or point clouds within a virtual reality (VR) environment. This immersive, interactive, and generative approach to storytelling adapts to individual users, encouraging imagination, collaboration, and emotional connection through a collaborative narrative creation. Our prototype demonstrates the potential of combining AI and VR to create dynamic and participatory educational and entertaining content for young learners. This opens new avenues at the intersection of AI, VR, and human-computer interaction, paving the way for novel approaches to storytelling, education, interpersonal connection, content creation, and artistic expression.