EVALUATING PARAMETER-EFFICIENT TRANSFER LEARNING APPROACHES ON SURE BENCHMARK FOR SPEECH UNDERSTANDING
Li Yingting (Beijing University of Posts and Telecommunications); Ambuj Mehrish (SUTD); RISHABH BHARDWAJ (Singapore University of Technology and Design); Navonil Majumder (SUTD); Bo Cheng (Beijing University of Posts and Telecommunications); Shuai Zhao (Beijing University of Posts and Telecommunications); Amri Zadeh (Amazon Science); Rada Mihalcea (University of Michigan); Soujanya Poria (Singapore University of Technology and Design)
-
SPS
IEEE Members: $11.00
Non-members: $15.00
Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and LoRA with only 0.94% of trainable parameters.