Contrastive Language-Image Pretraining (CLIP) learns joint embeddings by contrasting positive and negative image-text pairs using a dual-encoder architecture.
Key Uses:
Zero-shot image classification, content moderation, and visual search.
Hacks:
Use CoOp (Context Optimization) to learn continuous prompt vectors. Fine-tune with LoRA on custom datasets for domain adaptation. Combine with visual question answering (VQA) pipelines. Leverage CLIP embeddings for unsupervised clustering of visual data.
References:
Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.