SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 76–100 of 213 papers

TitleStatusHype
HEALNet: Multimodal Fusion for Heterogeneous Biomedical DataCode1
Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal LearningCode0
Dynamic Task and Weight Prioritization Curriculum Learning for Multimodal ImageryCode0
MalFake: A Multimodal Fake News Identification for Malayalam using Recurrent Neural Networks and VGG-16—0
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic AlignmentCode4
HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk FactorsCode0
Multimodal Deep Learning for Scientific Imaging Interpretation—0
A multimodal deep learning architecture for smoking detection with a small data approach—0
Formalizing Multimedia Recommendation through Multimodal Deep LearningCode1
Multimodal Guidance Network for Missing-Modality Inference in Content ModerationCode0
Multimodal Foundation Models For Echocardiogram InterpretationCode1
On the Adversarial Robustness of Multi-Modal Foundation ModelsCode1
PromptStyler: Prompt-driven Style Generation for Source-free Domain GeneralizationCode1
ARC-NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features—0
A scoping review on multimodal deep learning in biomedical images and texts—0
Multimodal Deep Learning for Personalized Renal Cell Carcinoma Prognosis: Integrating CT Imaging and Clinical DataCode0
A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Media—0
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep LearningCode2
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language LearningCode0
Towards Balanced Active Learning for Multimodal ClassificationCode1
ImageBind: One Embedding Space To Bind Them AllCode5
Multimodal Neural DatabasesCode1
Performance Optimization using Multimodal Modeling and Heterogeneous GNN—0
Building Multimodal AI ChatbotsCode0
Towards Unified AI Drug Discovery with Multiple Knowledge Modalities—0
Show:102550
← PrevPage 4 of 9Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified