SOTAVerified

Image-to-Text Retrieval

Image-text retrieval is the process of retrieving relevant images based on textual descriptions or finding corresponding textual descriptions for a given image. This task is interdisciplinary, combining techniques from computer vision, and natural language processing. The primary challenge lies in bridging the semantic gap — the difference between how visual data is represented in images and how humans describe that information using language. To address this, many methods focus on learning a shared embedding space where both images and text can be represented in a comparable way, allowing their similarities to be measured and facilitating more accurate retrieval.

Source: Extending CLIP for Category-to-Image Retrieval in E-commerce

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–50 of 59 papers

TitleStatusHype
AltCLIP: Altering the Language Encoder in CLIP for Extended Language CapabilitiesCode4
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsCode4
ONE-PEACE: Exploring One General Representation Model Toward Unlimited ModalitiesCode3
Sigmoid Loss for Language Image Pre-TrainingCode3
Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal AlignmentCode2
Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEsCode2
RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote SensingCode2
Learning Transferable Visual Models From Natural Language SupervisionCode2
Oscar: Object-Semantics Aligned Pre-training for Vision-Language TasksCode2
Efficient Remote Sensing with Harmonized Transfer Learning and Modality AlignmentCode2
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language TransformersCode1
A Deep Local and Global Scene-Graph Matching for Image-Text RetrievalCode1
A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal RetrievalCode1
Align before Fuse: Vision and Language Representation Learning with Momentum DistillationCode1
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision ModelsCode1
FETA: Towards Specializing Foundation Models for Expert Task ApplicationsCode1
FLAVA: A Foundational Language And Vision Alignment ModelCode1
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and LanguagesCode1
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic TasksCode1
Learning Relation Alignment for Calibrated Cross-modal RetrievalCode1
Vision-Language Dataset DistillationCode1
Negative Pre-aware for Noisy Cross-modal MatchingCode1
PRIOR: Prototype Representation Joint Learning from Medical Images and ReportsCode1
Prototype-based Aleatoric Uncertainty Quantification for Cross-modal RetrievalCode1
Rethinking Benchmarks for Cross-modal Image-text RetrievalCode1
UPop: Unified and Progressive Pruning for Compressing Vision-Language TransformersCode1
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-TrainingCode1
Accept the Modality Gap: An Exploration in the Hyperbolic Space—0
Is Cross-modal Information Retrieval Possible without Training?—0
DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding—0
Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization—0
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval—0
Hierarchical Gumbel Attention Network for Text-based Person Search—0
CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?—0
Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization—0
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images—0
Towards a Visual-Language Foundation Model for Computational Pathology—0
Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning—0
SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs—0
Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration—0
A survey on knowledge-enhanced multimodal learning—0
Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval—0
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training—0
ABC: Achieving Better Control of Multimodal Embeddings using VLMs—0
Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation—0
When are Lemons Purple? The Concept Association Bias of Vision-Language Models—0
Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution—0
Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset—0
DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation—0
GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language ModelsCode0
Show:102550
← PrevPage 1 of 2Next →
All datasetsCOCO (Common Objects in Context)Flickr30kWHOOPS!AIC-ICCFETA Car-ManualsRUC-CAS-WenLanCOCORSICD

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1OscarRecall@1099.8—Unverified
2OscarRecall@1098.3—Unverified
3Unicoder-VLRecall@1097.2—Unverified
4BLIP-2 (ViT-G, fine-tuned)Recall@185.4—Unverified
5ONE-PEACE (ViT-G, w/o ranking)Recall@184.1—Unverified
6BLIP-2 (ViT-L, fine-tuned)Recall@183.5—Unverified
7DVSARecall@1074.8—Unverified
8IAISRecall@167.78—Unverified
9CLIP (zero-shot)Recall@158.4—Unverified
10FLAVA (ViT-B, zero-shot)Recall@142.74—Unverified
#ModelMetricClaimedVerifiedStatus
1InternVL-G-FT (finetuned, w/o ranking)Recall@197.9—Unverified
2ONE-PEACE (finetuned, w/o ranking)Recall@197.6—Unverified
3BLIP-2 ViT-G (zero-shot, 1K test set)Recall@197.6—Unverified
4InternVL-C-FT (finetuned, w/o ranking)Recall@197.2—Unverified
5BLIP-2 ViT-L (zero-shot, 1K test set)Recall@196.9—Unverified
6ERNIE-ViL 2.0Recall@196.1—Unverified
7ALBEFRecall@195.9—Unverified
8UNITERRecall@187.3—Unverified
9GSMNRecall@176.4—Unverified
10LGSGMRecall@171—Unverified
#ModelMetricClaimedVerifiedStatus
1BLIP2 FlanT5-XXL (Text-only FT)Specificity94—Unverified
2BLIP2 FlanT5-XXL (Fine-tuned)Specificity84—Unverified
3BLIP2 FlanT5-XL (Fine-tuned)Specificity81—Unverified
4BLIP LargeSpecificity77—Unverified
5CoCa ViT-L-14 MSCOCOSpecificity72—Unverified
6BLIP2 FlanT5-XXL (Zero-shot)Specificity71—Unverified
7CLIP ViT-L/14Specificity70—Unverified
#ModelMetricClaimedVerifiedStatus
1ERNIE-ViL2.0Recall@133.7—Unverified
2CMCLRecall@120.3—Unverified
3ERNIE-ViL2.0Recall@119—Unverified
#ModelMetricClaimedVerifiedStatus
1FETA's CLIP-MIL (Many-Shot Image-to-text)R@135.5—Unverified
2FETA's CLIP-MIL (Many-Shot Image-to-text)R@129—Unverified
#ModelMetricClaimedVerifiedStatus
1CMCLRecall@136.1—Unverified
2CMCLRecall@136—Unverified
#ModelMetricClaimedVerifiedStatus
1SigLIP (ViT-L, zero-shot)Recall@170.6—Unverified
#ModelMetricClaimedVerifiedStatus
1GeoRSCLIP-FTImage to Text Recall@122.14—Unverified