PaLM: Scaling Language Modeling with Pathways

🤖 Plain-English Summary

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale.

🔑 Key Findings

• To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
• We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods.
• We demonstrate continued benefits of scaling by achieving advanced few-shot learning results on hundreds of language understanding and generation benchmarks.

🔗 Related Resources