Sigmoid Loss for Language Image Pre-Training

🤖 Plain-English Summary

We propose a simple pairwise sigmoid loss for imagetext pre-training. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient.

🔑 Key Findings

• Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization.
• The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes.
• With only four TPUv4 chips, we can train a Base CLIP model at 4k batch size and a Large LiT model at 20k batch size, the latter achieves 84.5% ImageNet zero-shot accuracy in two days.

🔗 Related Resources