Emotion Alignment Between Text and Speech is Limited: A Cross-Modal Study

Underline Science Inc.

Abstract

Although expressive TTS systems aim to capture human-like emotion, little is known about how well emotional signals in text correspond to those in speech. In this short paper, we investigate the emotion alignment (Valence, Arousal, Dominance) between text and speech models. We use 8 large language models for identifying text emotions and two audio models for speech analysis, across three genres: Podcasts, Audiobooks and TED talks. Findings show that while language models perform well on emotion recognition from situational text, and the audio models performs well on speech, they show alignment on the Valence dimension only. Further, the genre of the content significantly impacts the alignment performance: audiobooks exhibit better text-audio correlation than TED talks. Our results highlight the limitations of text-based emotion expression in TTS systems: performance depends on genre, is mostly limited to Valence, and more context fails to improve emotional alignment between text and speech.

Authors 4

1. Association for Computational Linguistics 2026
2. Joost Broekens Aachen

   Leiden University

   Affiliation as printed

   Leiden University

3. David Lindevelt
4. Suzan Verberne Aachen

   Leiden University

   Affiliation as printed

   Leiden University