SOTAVerified

Multimodal Deep Learning

Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.

One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.

Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 101–150 of 213 papers

TitleStatusHype
P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data—0
A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials—0
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init AttentionCode5
Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma—0
Show me your NFT and I tell you how it will perform: Multimodal representation learning for NFT selling price prediction—0
Zorro: the masked multimodal transformerCode0
Multimodal Deep LearningCode1
Learning Semantic Relationship Among Instances for Image-Text MatchingCode1
Learning Multimodal Data Augmentation in Feature SpaceCode1
Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing ApplicationsCode1
Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate DiagnosisCode1
A survey on knowledge-enhanced multimodal learning—0
aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range PerceptionCode1
Language-Assisted Deep Learning for Autistic Behaviors Recognition—0
MultiCrossViT: Multimodal Vision Transformer for Schizophrenia Prediction using Structural MRI and Functional Network Connectivity Data—0
Bayesian Prompt Learning for Image-Language Model GeneralizationCode1
LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language ModelsCode1
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringCode2
LAVIS: A Library for Language-Vision IntelligenceCode0
Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning—0
TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival PredictionCode1
Multi-Modal Experience Inspired AI CreationCode1
R2D2 at SemEval-2022 Task 5: Attention is only as good as its Values! A multimodal system for identifying misogynist memes—0
Multimodal Attention-based Deep Learning for Alzheimer's Disease DiagnosisCode1
Vision-Aided Frame-Capture-Based CSI Recomposition for WiFi Sensing: A Multimodal Approach—0
Detection of Propaganda Techniques in Visuo-Lingual Metaphor in Memes—0
DeepGraviLens: a Multi-Modal Architecture for Classifying Gravitational Lensing DataCode0
Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic SegmentationCode2
Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes—0
A Review on Methods and Applications in Multimodal Deep Learning—0
Emotion Based Hate Speech Detection using Multimodal Learning—0
Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network—0
Multimodal Approach for Metadata Extraction from German Scientific Publications—0
From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation—0
DeepStroke: An Efficient Stroke Screening Framework for Emergency Rooms with Multimodal Adversarial Deep Learning—0
Contrastive Language-Image Pre-training for the Italian LanguageCode1
Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep LearningCode1
Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions—0
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment AnalysisCode1
A Multimodal Deep Learning Model for Cardiac Resynchronisation Therapy Response Prediction—0
Deep Learning for Technical Document Classification—0
Listen to Your Favorite Melodies with img2Mxml, Producing MusicXML from Sheet Music Image by Measure-based Multimodal Deep Learning-driven Assembly—0
DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis—0
Digital Taxonomist: Identifying Plant Species in Community Scientists' Photographs—0
How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning—0
Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-TrainingCode1
Recent Advances and Trends in Multimodal Deep Learning: A Review—0
Multimodal Deep Learning Framework for Image Popularity Prediction on Social Media—0
A multimodal deep learning framework for scalable content based visual media retrievalCode0
Where and When: Space-Time Attention for Audio-Visual Explanations—0
Show:102550
← PrevPage 3 of 5Next →

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1Two Branch Network (Text - Bert + Image - Nts-Net)Accuracy96.81—Unverified