Abusive activity detection with multi-modality based on convolutional neural network

Abusive activity frequently occurs in various places, including nursing homes for the elderly. However, it is difficult to detect because it has various forms and is not easy to define. Therefore, in this study, we try to detect using the Convolutional Neural Network (CNN). Data based on multi-modality (image and sound) was used to improve detection performance. In case of the image, a Motion History Image (MHI) was used to reflect motion information. The developed multi-modality-based detection system showed a performance of 0.807 (Accuracy) and 0.858 (F1 score). By comparing with the system using only image or sound, there is an improvement of 0.132 ~ 0.17 (Accuracy) and 0.094 ~ 0.138 (F1 score)