SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 151–175 of 213 papers

TitleStatusHype
DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing DataCode0
Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes—0
A Review on Methods and Applications in Multimodal Deep Learning—0
Emotion Based Hate Speech Detection using Multimodal Learning—0
Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network—0
Multimodal Approach for Metadata Extraction from German Scientific Publications—0
From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation—0
DeepStroke: An Efficient Stroke Screening Framework for Emergency Rooms with Multimodal Adversarial Deep Learning—0
Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions—0
A Multimodal Deep Learning Model for Cardiac Resynchronisation Therapy Response Prediction—0
Deep Learning for Technical Document Classification—0
Listen to Your Favorite Melodies with img2Mxml, Producing MusicXML from Sheet Music Image by Measure-based Multimodal Deep Learning-driven Assembly—0
DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis—0
Digital Taxonomist: Identifying Plant Species in Community Scientists' Photographs—0
How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning—0
Recent Advances and Trends in Multimodal Deep Learning: A Review—0
Multimodal Deep Learning Framework for Image Popularity Prediction on Social Media—0
A multimodal deep learning framework for scalable content based visual media retrievalCode0
Where and When: Space-Time Attention for Audio-Visual Explanations—0
The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System—0
Robust Sensor Fusion Algorithms Against Voice Command Attacks in Autonomous VehiclesCode0
Leveraging Audio Gestalt to Predict Media Memorability—0
Predicting Online Video Advertising Effects with Multimodal Deep Learning—0
Multi-Modal Detection of Alzheimer's Disease from Speech and Text—0
Multimodal Learning for Hateful Memes DetectionCode0
Show:102550
← PrevPage 7 of 9Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified