A modality-wise multi-modal transfer learning and dual stream CNN-Transformer hybrid network for ACC-sEMG-based continuous estimation of finger joint angles

Abstract

Continuous estimation of finger joint angles (CEFJA) by decoding surface electromyography (sEMG) from the upper-limb has garnered significant attention due to its ability to provide synchronized and proportional control of robotic hands. However, two challenges still exist: (1) CEFJA using only sEMG cannot achieve satisfactory accuracy, and (2) the accuracy of CEFJA drops sharply in the cross-subjects scenarios as sEMG varies significantly among different subjects, which implies that despite utilizing more subjects for training, the accuracy of each subject is still low. As three-axis accelerometers (ACC) can provide information directly related to upper limb movement, here we first propose a novel CNN-Transformer hybrid network called Fusionformer for ACC-sEMGbased CEFJA to address the challenge (1), and then propose a multi-modal transfer learning strategy called Modality-Wise Adversarial Discriminative Domain Adaptation (MW-ADDA) to address challenge (2). The Fusionformer consists of a representation learning module with separate ACC and sEMG branches, each integrating dual-stream CNN-Transformer networks to capture both short-term and long-term dependencies, and a modal fusion module employing Multi-Modal Cross-Attention (MMCA) to combine modality-specific dependencies. The MW-ADDA strategy applies separate domain discriminators for each modality to enable effective cross-subjects knowledge transfer. We conducted four groups of experiments on the widely-used datasets Ninapro DB2 and DB7. Results show that the proposed Fusionformer outperforms the state-of-the-art methods and that the utilization of MW-ADDA can significantly improve the accuracy of Fusionformer on the target subject. Additionally, interpretability analysis provides insights into model decision-making mechanisms and reveals the roles of different architectural components.