The Benefits of Multimodal "Reading" of Our Audiobooks and Magazines

This article explores the multimodal storytelling we use from a scientific perspective... just so you know... our fun and useful content is also educational.

Less digital fatigue, more curiosity.

In the digital age, new ways of consuming literature and media have emerged, allowing the integration of multisensory experiences into reading and learning.

Artificial intelligence is fundamentally changing the situation.

Audiobooks and multimodal reading—where a person can choose how to enjoy a cocktail of content—allow them to read, listen, and view visual elements either simultaneously or separately. Applications offer an inventive way to produce content and an interactive approach for engaging with it.

The combination of different media, such as illustrated texts, which can be both read and listened to on various devices, brings several benefits in terms of comprehension, retention, and engagement.

What do studies say about this multimodal approach?

How does reading, listening to, and viewing visuals of the same work affect learning and understanding when used together or separately?

The combination of reading, listening, and viewing in a multimodal form offers numerous benefits for comprehension, retention, and engagement. Scientific studies support the idea that activating multiple senses simultaneously improves learning outcomes, as it enables the processing of information through different channels.

Illustrated works that can be both read and listened to offer a richer and more immersive experience, allowing for deeper understanding and flexibility in content consumption.

Moreover, the interactive features of digital platforms, along with increased social engagement, further enrich the learning process.

Whether for education or entertainment, multimodal platforms for reading and listening represent a new way of consuming content, providing a highly adaptable and immersive way to absorb information.

This approach not only improves learning outcomes but also creates an interactive, engaging experience that can cater to the needs and preferences of different learners.

The Science Behind Multimodal Learning

Multimodal learning involves using more than one communication channel, such as visual, auditory, or kinesthetic, to enhance the learning process.

Research shows that activating multiple senses can significantly improve comprehension and retention. A study by Mayer and Moreno (2003) on multimedia learning suggests that when learners receive information through more than one channel (e.g., sound and images), they are more likely to process and retain information effectively.

This is supported by Paivio's dual-coding theory, which states that both verbal and non-verbal information is processed in separate channels in the brain, and when both are activated, they create stronger memory traces.

The Benefits of Reading and Listening Simultaneously

Listening to audiobooks while reading text has been shown to improve comprehension and fluency, particularly for readers who struggle with acquisition.

For example, Woodall's (2010) research shows that listening while reading helps learners process language faster because they are utilizing both auditory and visual memory.

This dual input strengthens learning by providing multiple pathways for absorbing information.

The Sensory Enjoyment Benefits

Better comprehension and retention: When readers—or rather, users of multimodal content—both hear and see the text, they can absorb information through two different sensory channels, reinforcing their understanding.

Reed and Vaughn (2012) found in their research that learners who read while listening to an audiobook achieved higher levels of comprehension than those who only read or listened.

Increased engagement: Multimodal formats, especially those including images, add a dynamic dimension to the reading experience and enhance engagement.

Visual elements help illustrate complex ideas, making them easier to understand. When combined with narration, these images create a more immersive experience.

For example, illustrated audiobooks or e-books with images allow readers to make connections between visual and auditory inputs, promoting deeper understanding and retention.

Reducing Cognitive Load

Multimodal learning can help reduce cognitive load. According to Sweller's cognitive load theory (Sweller, 1988), the brain has a limited capacity for processing information. By distributing information across different channels—reading text, hearing it, and viewing related images—the brain can process more information without becoming overloaded.

The Impact of Visual Elements on Comprehension

Illustrations and visual content play a key role in multimodal learning. When combined with text and sound, they enhance the understanding of complex concepts.

A study by Park and Hopkins in 2017 showed that students exposed to both visual and auditory elements when learning new material demonstrated better retention and problem-solving skills compared to those who relied solely on text.

Images help reinforce material by providing context and additional information. For example, in educational audiobooks or multimodal texts, diagrams, charts, or pictures can clarify abstract ideas, making them more concrete.

This visual support is particularly beneficial for visual learners, for whom reading or listening alone may be challenging.

It is also a great aid to nonfiction authors.

Interactive Features and Social Engagement

Another important aspect of multimodal content consumption is its interactive and social dimension.

With advancements in technology, readers can now engage with content across various platforms, share their thoughts, and participate in discussions in real-time.

Digital books and audiobooks with social features, such as discussion forums or collaborative platforms, can enhance learning by allowing readers to reflect on what they have read or listened to. Such active participation is essential for deeper cognitive processing and critical thinking.

Moreover, mobile devices and interactive screens allow users to comment on specific chapters or parts of the work they have read, listened to, or watched. This fosters a collaborative learning environment where individuals can exchange ideas and interpretations, leading to a richer understanding of the material.

The Flexibility of Multimodal Learning

One of the greatest advantages of multimodal reading is the flexibility it offers. Different users have different learning preferences—some may prefer to listen to an audiobook during their commute, while others may find it easier to read and view images on a tablet at home.

Many people may still prefer printed books... but print editions are shrinking...

Multimodal content formats allow each user to choose their preferred way of consuming content, whether it's reading, listening, or watching. The ability to switch between these methods or use them simultaneously ensures that content remains accessible to a wide audience.

In my book Kohtaamisyhteiskunta (Edita, 1994), I predicted that soon content would not be created for a specific medium—whether a book, magazine, or radio program—but rather the idea behind the content would determine its optimal format and distribution method. This has happened and continues to happen at such a fast pace that we hardly notice it.

In educational settings, this multichannel flexibility is invaluable. Visually impaired students can listen to content, while those with auditory processing difficulties can rely more on visual elements, such as text and illustrations.

The ability to cater to diverse learning needs ensures that no learner is left behind.

We are experimenting with this.