HAPtics: Human Action Prediction in Real-time via Pose Kinematics

Abstract

Recognizing human actions in real-time presents a fundamental challenge, particularly when humans interact with other humans or objects in a shared space. Such systems must be able to recognize and assess real-world human actions from different angles and viewpoints. Consequently, a substantial volume of multi-dimensional human action training data is essential to enable data-driven algorithms to operate effectively in real-world scenarios. This paper introduces the Action Clip dataset, which provides a comprehensive 360-degree view of human actions, capturing rich features from multiple angles. Additionally, we describe the design and implementation of Human Action Prediction via Pose Kinematics (HAPtics), a comprehensive pipeline for real-time human pose estimation and action recognition, all achievable with standard monocular camera sensors. HAPtics utilizes a skeleton modality by transforming initially noisy human pose kinematic structures into skeletal features, such as body velocity, joint velocity, joint angles, and limb lengths derived from joint positions, followed by a classification layer. We have implemented and evaluated HAPtics using four different datasets, demonstrating competitive state-of-the-art performance in pose-based action recognition and real-time performance at 30 frames per second on a live camera. The code and dataset are available at: https://github.com/RaiseLab/HAPtics. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.