Boosting Prompt-based Few-shot Learners through Out-of-domain Knowledge Distillation

Prompt-based learning improves the performance of Pre-trained Language Models (PLMs) over few-shot learning and is suitable for low-resourced scenarios. However, it is challenging to deploy large PLMs online. Knowledge Distillation (KD) can compress large PLMs into small ones; yet, few-shot KD for prompt-tuned PLMs is challenging due to the lack of training data and the capacity gap between teacher and student models. I We propose Boost-Distiller, the first few-shot KD algorithm for prompt-tuned PLMs with the help of the out-of-domain data. Apart from distilling the model logits, Boost-Distiller specifically considers heuristically-generated fake logits that improve the generalization abilities of student models. We further leverage the cross-domain model logits, weighted with domain expertise scores that measure the transferablity of out-of-domain instances. Experiments over various datasets show Boost-Distiller consistently outperforms baselines by a large margin.