Learning Transferable Visual Models From Natural…

CLIP asks how an image model can recognize categories described in words without training a new classifier for every task. Read it to see how two encoders turn a batch into an all-pairs similarity matrix, how the true image-text pairs teach a shared embedding space, and why prompts make zero-shot recognition flexible but not automatically reliable.

Reading focus: How a batch of N image-text pairs becomes N² candidate matches, with the observed pairs forming the target diagonal. Why normalized image and text embeddings plus a symmetric contrastive loss make written class descriptions usable at inference time. Where CLIP's promise stops: prompt wording, class taxonomies, English-centric web data, heavy compute, weak tasks, and deployment risk still matter.

ICML 2021. Radford et al.. 35 min read, easy difficulty.