Spatio-temporal interactive reasoning model for multi-group activity recognition

Huang, Jianglan, Li, Lindong, Qing, Linbo ORCID: https://orcid.org/0000-0003-3555-0005, Tang, Wang, Wang, Pingyu, Guo, Li ORCID: https://orcid.org/0000-0003-1272-8480 and Peng, Yonghong (2024) Spatio-temporal interactive reasoning model for multi-group activity recognition. Pattern Recognition, 159. 111104. ISSN 0031-3203

Preview

Accepted Version Available under License Creative Commons Attribution. Download (941kB) | Preview

Abstract

Multi-group activity recognition aims to recognize sub-group activities in multi-person scenes. Existing works explore group-level features by simply using graph neural networks for reasoning about the individual interactions and directly aggregating individual features, which cannot fully mine the interactions between people and between sub-groups, resulting in the loss of useful information for group activity recognition. To address this problem, this paper proposes a Spatio-Temporal Interactive Reasoning Model (STIRM) to better exploit potential spatio-temporal interactions for multi-group activity recognition. In particular, we present an interactive feature extraction strategy to explore correlation features between individuals by analyzing the features of their nearest neighbor. We design a new clustering module that combines the action similarity feature and spatio-temporal trajectory feature to divide people into small groups. In addition, to obtain rich and accurate group-level features, a group interaction reasoning module is constructed to explore the interactions between different small groups and among people in the same group and exclude people who have less impact on group activities according to their importance. Extensive experiments on the Social-CAD, PLPS and JRDB-PAR datasets indicate the superiority of the proposed method over state-of-the-art methods.