How To Use Time Information Effectively? Combining With Time Shift Module For Lipreading

In this paper, we improve this point. Firstly, Time Shift Module (TSM) is applied to two different front-ends (full 2D CNN based and mixture of 2D and 3D convolution) to enhance the ability of time information extraction. Secondly, we verify the influence of different shift proportion of TSM and different frame rate input on extracting time information. Thirdly, we compare the number of moving frames in TSM. We find that shifting in the time dimension by ±1 can better extract the short-term time features, and excessive movement will reduce the performance of 2D CNN. We have verified our method on two challenging word-level lipreading datasets LRW and LRW-1000, and achieved new state-of-the-art performance.