FusionGait-transformer: multimodal transformer-based framework for robust gait recognition and gait-condition classification

Abstract

Gait analysis benefits from both full-body motion context and accurate temporal dynamics. Vision-only approaches degrade under occlusion and viewpoint change, while inertial-only approaches miss full body context. FusionGait transformer is a multimodal framework that fuses pose and silhouette representations with inertial time series using transformer self-attention and bidirectional cross-attention. A per-modality reliability gate weights each stream by its instantaneous signal quality, so degradation in one modality shifts the fused representation toward the other. Masked multimodal pretraining and domain adversarial training improve stability under domain shift. Evaluation uses the CMU motion capture dataset under a subject-independent protocol, with inertial signals generated analytically from the same trajectories. The framework reaches Rank-1 identification accuracy of 98.9% with equal error rate (EER) of 1.8%, and gait-condition classification F1-score of 95.7% with area under the ROC curve (AUC-ROC) of 97.6%. Reliability gating is the largest single contributor to these figures. Deployment is characterized on workstation, edge, and gateway hardware, and transmitting embeddings rather than video reduces the uplink requirement by more than four orders of magnitude, establishing feasibility for wearable healthcare settings. Reported results describe performance under controlled benchmark conditions. Clinical diagnostic claims remain conditional on validation with real wearable sensors and clinically labeled cohorts.