Automatic Generation of Multimodal 4D Effects for Immersive Video Watching Experiences

Abstract

The recent trend of watching content using over-the-top (OTT) services pushes the 4D movie industry to seek a way of transformation. To address the issue, this paper suggests an AI-driven automatic 4D effects generation algorithm applied to a low-cost comfort chair. The system extracts multiple features using psychoacoustic analysis, saliency detection, optical flow, and an LLM-based thermal effect synthesis, and maps them into various sensory displays such as vibration, heat, wind, and poking automatically. To evaluate the system, a user study with 21 participants across seven film genres was conducted. The results showed that 1) there was a general improvement with 4D effects in terms of immersion, concentration, and expressiveness, and 2) multisensory effects were particularly useful in action and fantasy movie scenes. The suggested system could be directly used in current general video-on-demand services.