SOTAVerified

Cross-Modal Retrieval

Cross-Modal Retrieval (CMR) is a task of retrieving items across different modalities, such as image, text, video, and audio. The core challenge of CMR is the heterogeneity gap, which arises because data from different modalities have distinct representations, making direct comparison difficult. To address this, most CMR methods focus on learning a shared latent embedding space. In this space, concepts from different modalities are projected, allowing their similarity to be measured using a distance metric.

Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A Reproducibility Study

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 351–400 of 522 papers

TitleStatusHype
Learning Soft-Attention Models for Tempo-invariant Audio-Sheet Music Retrieval—0
Learning Sparse Disentangled Representations for Multimodal Exclusion Retrieval—0
Learning Structural Representations for Recipe Generation and Food Retrieval—0
Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays—0
New Ideas and Trends in Deep Multimodal Content Understanding: A Review—0
Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation—0
Objects that Sound—0
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval—0
OmniVL:One Foundation Model for Image-Language and Video-Language Tasks—0
Online Asymmetric Similarity Learning for Cross-Modal Retrieval—0
On the Importance of Text Preprocessing for Multimodal Representation Learning and Pathology Report Generation—0
Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval—0
Pairwise Relationship Guided Deep Hashing for Cross-Modal Retrieval—0
PATFinger: Prompt-Adapted Transferable Fingerprinting against Unauthorized Multimodal Dataset Usage—0
Pathology Report Generation and Multimodal Representation Learning for Cutaneous Melanocytic Lesions—0
Perfect match: Improved cross-modal embeddings for audio-visual synchronisation—0
PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting—0
Pix2Map: Cross-modal Retrieval for Inferring Street Maps from Images—0
Preserving Semantic Neighborhoods for Robust Cross-modal Retrieval—0
Progressive Domain-Independent Feature Decomposition Network for Zero-Shot Sketch-Based Image Retrieval—0
Ranking-based Deep Cross-modal Hashing—0
Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation—0
Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images—0
Retrieval-based Disentangled Representation Learning with Natural Language Supervision—0
Retrieving and Highlighting Action with Spatiotemporal Reference—0
Revisiting Cross Modal Retrieval—0
Revolutionizing Text-to-Image Retrieval as Autoregressive Token-to-Voken Generation—0
RREH: Reconstruction Relations Embedded Hashing for Semi-Paired Cross-Modal Retrieval—0
Sample-Specific Debiasing for Better Image-Text Models—0
SA-Person: Text-Based Person Retrieval with Scene-aware Re-ranking—0
Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping—0
Scale-Semantic Joint Decoupling Network for Image-text Retrieval in Remote Sensing—0
Second Place Solution of WSDM2023 Toloka Visual Question Answering Challenge—0
Seeing Speech and Sound: Distinguishing and Locating Audios in Visual Scenes—0
Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes—0
See What You See: Self-supervised Cross-modal Retrieval of Visual Stimuli from Brain Activity—0
Self-supervised Modal and View Invariant Feature Learning—0
Self-Supervised Modality-Invariant and Modality-Specific Feature Learning for 3D Objects—0
Self-Supervised Visual Representations for Cross-Modal Retrieval—0
Semantic Adversarial Network for Zero-Shot Sketch-Based Image Retrieval—0
Semantic Compositions Enhance Vision-Language Contrastive Learning—0
SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs—0
Simple to Complex Cross-modal Learning to Rank—0
Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild—0
Sound Source Localization is All about Cross-Modal Alignment—0
Start from Video-Music Retrieval: An Inter-Intra Modal Loss for Cross Modal Retrieval—0
SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval—0
T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency—0
Task-adaptive Asymmetric Deep Cross-modal Hashing—0
Learning Joint Embedding for Cross-Modal Retrieval—0
Show:102550
← PrevPage 8 of 11Next →
All datasetsCOCO 2014Flickr30kRecipe1M+RSICDRSITMDChEBI-20MSCOCO-1kSoundingEarthCUHK-PEDESFlickr-8kMSCOCOMS-COCO-2014

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1MaMMUT (ours)Image-to-text R@170.7—Unverified
2VASTText-to-image R@168—Unverified
3X2-VLM (large)Text-to-image R@167.7—Unverified
4BEiT-3Text-to-image R@167.2—Unverified
5XFM (base)Text-to-image R@167—Unverified
6X2-VLM (base)Text-to-image R@166.2—Unverified
7PTP-BLIP (14M)Text-to-image R@164.9—Unverified
8OmniVL (14M)Text-to-image R@164.8—Unverified
9VSE-GradientText-to-image R@163.6—Unverified
10X-VLM (base)Text-to-image R@163.4—Unverified
#ModelMetricClaimedVerifiedStatus
1X2-VLM (large)Image-to-text R@198.8—Unverified
2X2-VLM (base)Image-to-text R@198.5—Unverified
3BEiT-3Image-to-text R@198—Unverified
4OmniVL (14M)Image-to-text R@197.3—Unverified
5ERNIE-ViL 2.0Image-to-text R@197.2—Unverified
6Aurora (ours, r=128)Image-to-text R@197.2—Unverified
7X-VLM (base)Image-to-text R@197.1—Unverified
8VSE-GradientImage-to-text R@197—Unverified
9ALIGNImage-to-text R@195.3—Unverified
10VASTText-to-image R@191—Unverified
#ModelMetricClaimedVerifiedStatus
1VLPCook (R1M+)Image-to-text R@174.9—Unverified
2VLPCookImage-to-text R@173.6—Unverified
3T-Food (CLIP)Image-to-text R@172.3—Unverified
4T-FoodImage-to-text R@168.2—Unverified
5X-MRSImage-to-text R@164—Unverified
6H-TImage-to-text R@160—Unverified
7SCANImage-to-text R@154—Unverified
8ACMEImage-to-text R@151.8—Unverified
9VLPCookImage-to-text R@145.2—Unverified
10AdaMineImage-to-text R@139.8—Unverified
#ModelMetricClaimedVerifiedStatus
1HarMA (w/ GeoRSCLIP)Mean Recall38.95—Unverified
2GeoRSCLIP-FTMean Recall38.87—Unverified
3GLISAMean Recall37.69—Unverified
4RemoteCLIPMean Recall36.35—Unverified
5PE-RSITR (MRS-Adapter)Mean Recall31.12—Unverified
6PIRMean Recall24.46—Unverified
7DOVEMean Recall22.72—Unverified
8SWANMean Recall20.61—Unverified
9GaLRMean Recall18.96—Unverified
10AMFMNMean Recall15.53—Unverified
#ModelMetricClaimedVerifiedStatus
1HarMA (w/ GeoRSCLIP)Image-to-text R@132.74—Unverified
2GeoRSCLIP-FTImage-to-text R@132.3—Unverified
3GLISAImage-to-text R@132.08—Unverified
4RemoteCLIPImage-to-text R@128.76—Unverified
5PE-RSITR (MRS-Adapter)Image-to-text R@123.67—Unverified
6PIRImage-to-text R@118.14—Unverified
7DOVEImage-to-text R@116.81—Unverified
8GaLRImage-to-text R@114.82—Unverified
9SWANImage-to-text R@113.35—Unverified
10AMFMNImage-to-text R@110.63—Unverified
#ModelMetricClaimedVerifiedStatus
1CLASS (ORMA)Hits@167.4—Unverified
2ORMAHits@166.5—Unverified
3Song et al.Hits@156.5—Unverified
4CLASS (AMAN)Hits@151.1—Unverified
5DSOKRHits@151—Unverified
6AMANHits@149.4—Unverified
7All-EnsembleHits@134.4—Unverified
8MLP1Hits@122.4—Unverified
9GCN2Hits@122.3—Unverified
#ModelMetricClaimedVerifiedStatus
1NAPRegImage-to-text R@181.9—Unverified
2Dual-path CNNImage-to-text R@141.2—Unverified
#ModelMetricClaimedVerifiedStatus
1ResNet-18Median Rank565—Unverified
2GeoCLAPMedian Rank159—Unverified
#ModelMetricClaimedVerifiedStatus
1Dual PathText-to-image Medr2—Unverified
#ModelMetricClaimedVerifiedStatus
1NAPRegImage-to-text R@156.2—Unverified
#ModelMetricClaimedVerifiedStatus
13SHNetImage-to-text R@185.8—Unverified
#ModelMetricClaimedVerifiedStatus
1NAPRegText-to-image R@143—Unverified