SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 101–150 of 213 papers

TitleStatusHype
DeepStroke: An Efficient Stroke Screening Framework for Emergency Rooms with Multimodal Adversarial Deep Learning—0
TextAug: Test time Text Augmentation for Multimodal Person Re-identification—0
DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis—0
Leveraging Audio Gestalt to Predict Media Memorability—0
Deep Learning for Technical Document Classification—0
Listen to Your Favorite Melodies with img2Mxml, Producing MusicXML from Sheet Music Image by Measure-based Multimodal Deep Learning-driven Assembly—0
Deep learning evaluation using deep linguistic processing—0
Deep Coupled-Representation Learning for Sparse Linear Inverse Problems with Side Information—0
M2D: A Multi-modal Framework for Automatic Medical Diagnosis—0
MalFake: A Multimodal Fake News Identification for Malayalam using Recurrent Neural Networks and VGG-16—0
MDL-CW: A Multimodal Deep Learning Framework With Cross Weights—0
Data-driven geophysics: from dictionary learning to deep learning—0
P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data—0
Correlation Net: Spatiotemporal multimodal deep learning for action recognition—0
Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma—0
The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System—0
CarbonSense: A Multimodal Dataset and Baseline for Carbon Flux Modelling—0
BMMDetect: A Multimodal Deep Learning Framework for Comprehensive Biomedical Misconduct Detection—0
MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data—0
The Role of Emotions in Informational Support Question-Response Pairs in Online Health Communities: A Multimodal Deep Learning Approach—0
Multimodal Approach for Metadata Extraction from German Scientific Publications—0
Audio-Visual Approach For Multimodal Concurrent Speaker Detection—0
Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions—0
A survey on knowledge-enhanced multimodal learning—0
Multimodal deep learning approach for joint EEG-EMG data compression and classification—0
Multimodal deep learning approach to predicting neurological recovery from coma after cardiac arrest—0
Multimodal Deep Learning-Empowered Beam Prediction in Future THz ISAC Systems—0
Multimodal Deep Learning for Finance: Integrating and Forecasting International Stock Markets—0
Multimodal Deep Learning for Flaw Detection in Software Programs—0
Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications—0
Multimodal deep learning for mapping forest dominant height by fusing GEDI with earth observation data—0
Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech Samples—0
Timing Is Everything: Finding the Optimal Fusion Points in Multimodal Medical Imaging—0
A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials—0
Multimodal Deep Learning for Scientific Imaging Interpretation—0
Multimodal deep learning for short-term stock volatility prediction—0
Multimodal Deep Learning for Stroke Prediction and Detection using Retinal Imaging and Clinical Data—0
A C++ library for Multimodal Deep Learning—0
Multimodal Deep Learning Framework for Image Popularity Prediction on Social Media—0
Multimodal Deep Learning of Word-of-Mouth Text and Demographics to Predict Customer Rating: Handling Consumer Heterogeneity in Marketing—0
Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma—0
Vision-Aided Frame-Capture-Based CSI Recomposition for WiFi Sensing: A Multimodal Approach—0
Multimodal Deep Unfolding for Guided Image Super-Resolution—0
Multi-Modal Detection of Alzheimer's Disease from Speech and Text—0
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer—0
Multimodal Emotion Recognition Using Multimodal Deep Learning—0
A Self-supervised Multimodal Deep Learning Approach to Differentiate Post-radiotherapy Progression from Pseudoprogression in Glioblastoma—0
A scoping review on multimodal deep learning in biomedical images and texts—0
A Review on Methods and Applications in Multimodal Deep Learning—0
ARC-NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features—0
Show:102550
← PrevPage 3 of 5Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified