Selective Token Generation for Few-shot Natural Language Generation

Abstract

Natural language modeling with limited training data is a challenging problem, and manyalgorithms make use of large-scale pretrainedlanguage models (PLMs) for this due to itsgreat generalization ability. Among them, additive learning that incorporates a task-specificadapter on top of the fixed large-scale PLMhas been popularly used in the few-shot setting. However, this added adapter is still easyto disregard the knowledge of the PLM especially for few-shot natural language generation (NLG) since an entire sequence is usuallygenerated by only the newly trained adapter.Therefore, in this work, we develop a noveladditive learning algorithm based on reinforcement learning (RL) that selectively outputs language tokens between the task-general PLMand the task-specific adapter during both training and inference. This output token selection over the two generators allows the adapterto take into account solely the task-relevantparts in sequence generation, and thereforemakes it more robust to overfitting as well asmore stable in RL training. In addition, toobtain the complementary adapter from thePLM for each few-shot task, we exploit aseparate selecting module that is also simultaneously trained using RL. Experimental results on various few-shot NLG tasks including question answering, data-to-text generation and text summarization demonstrate thatthe proposed selective token generation significantly outperforms the previous additive learning algorithms based on the PLMs.