A visual-physiology multimodal system for detecting outlier behavior of participants in a reality TV show

Abstract

This study proposes an outlier detection system based on the visual-physiology multimodal data system for a Korean reality TV show, "Perfect on Paper." The goal of this system is to provide the program production team and the master of ceremonies with real-time facial expressions and bio-signal analysis results of program participants. Using this information, the program production team and master of ceremonies can easily identify the participants with unusual behavior. We propose an insole-type hardware that measures the wearer's skin conductivity, temperature, and motion. We also suggest a dynamic time warp-based clustering algorithm for the outlier and micro-expression detection technique. The system was developed starting from the program planning phase through a 6-month period in collaboration with the production team. The developed system analyzed the biometric information and facial expressions of 88 participants in 11 episodes and provided rapid feedback to the production team at the shooting spot in real time. Finally, we present a case where a visual-physiology multimodal system can be useful for a real TV broadcasting production.