SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–50 of 213 papers

TitleStatusHype
DeepSeek-VL: Towards Real-World Vision-Language UnderstandingCode7
ImageBind: One Embedding Space To Bind Them AllCode5
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init AttentionCode5
InstructIR: High-Quality Image Restoration Following Human InstructionsCode4
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic AlignmentCode4
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringCode2
PHemoNet: A Multimodal Network for Physiological SignalsCode2
Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language ModelsCode2
Linguistic-Aware Patch Slimming Framework for Fine-grained Cross-Modal AlignmentCode2
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep LearningCode2
Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic SegmentationCode2
Audio-Conditioned U-Net for Position Estimation in Full Sheet ImagesCode1
Multimodal Foundation Models For Echocardiogram InterpretationCode1
Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-TrainingCode1
Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature FusionCode1
More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery ClassificationCode1
Multimodal-Enhanced Objectness Learner for Corner Case Detection in Autonomous DrivingCode1
On the Adversarial Robustness of Multi-Modal Foundation ModelsCode1
Are These Birds Similar: Learning Branched Networks for Fine-grained RepresentationsCode1
Learning Multimodal Data Augmentation in Feature SpaceCode1
Jointly Fine-Tuning “BERT-like” Self Supervised Models to Improve Multimodal Speech Emotion RecognitionCode1
Learning Semantic Relationship Among Instances for Image-Text MatchingCode1
MinkLoc++: Lidar and Monocular Image Fusion for Place RecognitionCode1
MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal FusionCode1
Multimodal Attention-based Deep Learning for Alzheimer's Disease DiagnosisCode1
Multimodal Deep LearningCode1
Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes ChallengeCode1
Multi-Modal Experience Inspired AI CreationCode1
Multimodal Neural DatabasesCode1
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment AnalysisCode1
Image and Text fusion for UPMC Food-101 \ BERT and CNNsCode1
LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language ModelsCode1
aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range PerceptionCode1
Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing ApplicationsCode1
CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information RetrievalCode1
Analysis of Social Media Data using Multimodal Deep Learning for Disaster ResponseCode1
HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding ModelsCode1
Contrastive Language-Image Pre-training for the Italian LanguageCode1
CardioLab: Laboratory Values Estimation and Monitoring from Electrocardiogram Signals -- A Multimodal Deep Learning ApproachCode1
Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI DevelopmentCode1
Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense KnowledgeCode1
Image Search With Text Feedback by Visiolinguistic Attention LearningCode1
Formalizing Multimedia Recommendation through Multimodal Deep LearningCode1
Detecting Video Game Player Burnout with the Use of Sensor Data and Machine LearningCode1
Distilling Audio-Visual Knowledge by Compositional Contrastive LearningCode1
HEALNet: Multimodal Fusion for Heterogeneous Biomedical DataCode1
Deep Learning for Android Malware Defenses: a Systematic Literature ReviewCode1
Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate DiagnosisCode1
MMEA: Entity Alignment for Multi-Modal Knowledge GraphsCode1
HYDRA: A multimodal deep learning framework for malware classificationCode1
Show:102550
← PrevPage 1 of 5Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified