P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks

🤖 Plain-English Summary

Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. It matches the performance of finetuning while having only 0.1%-3% tuned parameters.

🔑 Key Findings

• However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models.
• We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality.
• We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks.

🔗 Related Resources