SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 101–125 of 213 papers

TitleStatusHype
Exploring Multimodal Features and Fusion Strategies for Analyzing Disaster Tweets—0
Fine-grained Video Attractiveness Prediction Using Multimodal Deep Learning on a Large Real-world Dataset—0
From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation—0
Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network—0
How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning—0
Hybrid Attention based Multimodal Network for Spoken Language Classification—0
Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning—0
Improved Multimodal Deep Learning with Variation of Information—0
Improving Neonatal Care: An Active Dry-Contact Electrode-based Continuous EEG Monitoring System with Seizure Detection—0
Innovative Framework for Early Estimation of Mental Disorder Scores to Enable Timely Interventions—0
Integrating Chemical Language and Molecular Graph in Multimodal Fused Deep Learning for Drug Property Prediction—0
Integrating Vision and Location with Transformers: A Multimodal Deep Learning Framework for Medical Wound Analysis—0
Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecasting—0
Leveraging Audio Gestalt to Predict Media Memorability—0
Listen to Your Favorite Melodies with img2Mxml, Producing MusicXML from Sheet Music Image by Measure-based Multimodal Deep Learning-driven Assembly—0
M2D: A Multi-modal Framework for Automatic Medical Diagnosis—0
MalFake: A Multimodal Fake News Identification for Malayalam using Recurrent Neural Networks and VGG-16—0
MDL-CW: A Multimodal Deep Learning Framework With Cross Weights—0
P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data—0
Multimodal Deep Unfolding for Guided Image Super-Resolution—0
Multi-Modal Detection of Alzheimer's Disease from Speech and Text—0
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer—0
Multimodal Emotion Recognition Using Multimodal Deep Learning—0
Multimodal Fusion of Glucose Monitoring and Food Imagery for Caloric Content Prediction—0
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications—0
Show:102550
← PrevPage 5 of 9Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified