SOTAVerified

Multimodal Emotion Recognition

This is a leaderboard for multimodal emotion recognition on the IEMOCAP dataset. The modality abbreviations are A: Acoustic T: Text V: Visual

Please include the modality in the bracket after the model name.

All models must use standard five emotion categories and are evaluated in standard leave-one-session-out (LOSO). See the papers for references.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–50 of 180 papers

TitleStatusHype
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement LearningCode5
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction TuningCode4
SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion RecognitionCode4
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion RecognitionCode3
MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wildCode2
MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised LearningCode2
Recent Trends of Multimodal Affective Computing: A Survey from NLP PerspectiveCode2
Hierarchical Hypercomplex Network for Multimodal Emotion RecognitionCode2
PHemoNet: A Multimodal Network for Physiological SignalsCode2
Multimodal Emotion Recognition with High-level Speech and Text FeaturesCode1
Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in ConversationCode1
Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion RecognitionCode1
Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross AttentionCode1
Music Mood Detection Based On Audio And Lyrics With Deep Neural NetCode1
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion RecognitionCode1
Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responsesCode1
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion RecognitionCode1
Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological SignalsCode1
Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion RecognitionCode1
Jointly Fine-Tuning “BERT-like” Self Supervised Models to Improve Multimodal Speech Emotion RecognitionCode1
Decoupled Multimodal Distilling for Emotion RecognitionCode1
Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion RecognitionCode1
DialogueRNN: An Attentive RNN for Emotion Detection in ConversationsCode1
MMER: Multimodal Multi-task Learning for Speech Emotion RecognitionCode1
MSAF: Multimodal Split Attention FusionCode1
COGMEN: COntextualized GNN based Multimodal Emotion recognitioNCode1
Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion RecognitionCode1
Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature FusionCode1
Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive LossCode1
Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal RepresentationsCode1
A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion RecognitionCode1
Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality InteractionCode1
Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalitiesCode1
FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition InferenceCode1
Emotion Recognition in Audio and Video Using Deep Neural NetworksCode1
A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in ConversationsCode1
Cooperative Sentiment Agents for Multimodal Sentiment AnalysisCode1
Cross Attentional Audio-Visual Fusion for Dimensional Emotion RecognitionCode1
GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion RecognitionCode1
GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion RecognitionCode1
A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party ConversationsCode1
Curriculum Learning Meets Directed Acyclic Graph for Multimodal Emotion RecognitionCode1
Towards Emotion Analysis in Short-form Videos: A Large-Scale Dataset and BaselineCode1
Joint Multimodal Transformer for Emotion Recognition in the WildCode1
Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion RecognitionCode1
EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion UnderstandingCode1
A proposal for Multimodal Emotion Recognition using aural transformers and Action Units on RAVDESS datasetCode1
A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognitionCode1
CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion RecognitionCode1
Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion RecognitionCode1
Show:102550
← PrevPage 1 of 4Next →
All datasetsIEMOCAP-4MELDIEMOCAPCMU-MOSEI-SentimentCMU-MOSEI-Sentiment-3Expressive hands and faces dataset (EHF).MELD-Sentiment

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F186.52—Unverified
2JoyfulWeighted F185.7—Unverified
3COGMENWeighted F184.5—Unverified
4DANNAccuracy82.7—Unverified
5MMERAccuracy81.7—Unverified
6PATHOSnet v2Accuracy80.4—Unverified
7Self-attention weight correction (A+T)Accuracy76.8—Unverified
8CHFusionAccuracy76.5—Unverified
9bc-LSTMWeighted F174.1—Unverified
10Audio + Text (Stage III)F170.5—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F166.71—Unverified
2Audio + Text (Stage III)Weighted F165.8—Unverified
3JoyfulWeighted F161.77—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F172.81—Unverified
2JoyfulWeighted F170.5—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F144.93—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F166.73—Unverified
#ModelMetricClaimedVerifiedStatus
1SMPLify-Xv2v error52.9—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F174.31—Unverified