SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 101–150 of 213 papers

TitleStatusHype
Research on Image Recognition Technology Based on Multimodal Deep Learning—0
Feature importance to explain multimodal prediction models. A clinical use caseCode0
ECC Analyzer: Extract Trading Signal from Earnings Conference Calls using Large Language Model for Stock Performance Prediction—0
ViTextVQA: A Large-Scale Visual Question Answering Dataset for Evaluating Vietnamese Text Comprehension in ImagesCode0
In the Search for Optimal Multi-view Learning Models for Crop Classification with Global Remote Sensing DataCode0
Describe-and-Dissect: Interpreting Neurons in Vision Networks with Language Models—0
Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecasting—0
A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection—0
Restoring Ancient Ideograph: A Multimodal Multitask Neural Network ApproachCode0
Multimodal deep learning approach to predicting neurological recovery from coma after cardiac arrest—0
Cultural-Aware AI Model for Emotion RecognitionCode0
Multimodal Learning To Improve Cardiac Late Mechanical Activation Detection From Cine MR Images—0
Multimodal Deep Learning of Word-of-Mouth Text and Demographics to Predict Customer Rating: Handling Consumer Heterogeneity in Marketing—0
Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep LearningCode0
Multimodal Urban Areas of Interest Generation via Remote Sensing Imagery and Geographical Prior—0
Predicting the Skies: A Novel Model for Flight-Level Passenger Traffic Forecasting—0
Multimodal self-supervised learning for lesion localization—0
Integrating Chemical Language and Molecular Graph in Multimodal Fused Deep Learning for Drug Property Prediction—0
A graph-based multimodal framework to predict gentrification—0
SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration—0
TextAug: Test time Text Augmentation for Multimodal Person Re-identification—0
Multimodal deep learning for mapping forest dominant height by fusing GEDI with earth observation data—0
Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal LearningCode0
Dynamic Task and Weight Prioritization Curriculum Learning for Multimodal ImageryCode0
MalFake: A Multimodal Fake News Identification for Malayalam using Recurrent Neural Networks and VGG-16—0
HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk FactorsCode0
Multimodal Deep Learning for Scientific Imaging Interpretation—0
A multimodal deep learning architecture for smoking detection with a small data approach—0
Multimodal Guidance Network for Missing-Modality Inference in Content ModerationCode0
ARC-NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features—0
A scoping review on multimodal deep learning in biomedical images and texts—0
Multimodal Deep Learning for Personalized Renal Cell Carcinoma Prognosis: Integrating CT Imaging and Clinical DataCode0
A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Media—0
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language LearningCode0
Performance Optimization using Multimodal Modeling and Heterogeneous GNN—0
Building Multimodal AI ChatbotsCode0
Towards Unified AI Drug Discovery with Multiple Knowledge Modalities—0
P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data—0
A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials—0
Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma—0
Show me your NFT and I tell you how it will perform: Multimodal representation learning for NFT selling price prediction—0
Zorro: the masked multimodal transformerCode0
A survey on knowledge-enhanced multimodal learning—0
Language-Assisted Deep Learning for Autistic Behaviors Recognition—0
MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data—0
LAVIS: A Library for Language-Vision IntelligenceCode0
Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning—0
R2D2 at SemEval-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes—0
Vision-Aided Frame-Capture-Based CSI Recomposition for WiFi Sensing: A Multimodal Approach—0
Detection of Propaganda Techniques in Visuo-Lingual Metaphor in Memes—0
Show:102550
← PrevPage 3 of 5Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified