SOTAVerified

Multimodal Emotion Recognition

This is a leaderboard for multimodal emotion recognition on the IEMOCAP dataset. The modality abbreviations are A: Acoustic T: Text V: Visual

Please include the modality in the bracket after the model name.

All models must use standard five emotion categories and are evaluated in standard leave-one-session-out (LOSO). See the papers for references.

Papers

Recently AddedMost HypedMost ActiveNeeds VerificationMost Verified

Showing 1–50 of 180 papers

TitleStatusHype
R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement LearningCode5
SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion RecognitionCode4
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction TuningCode4
MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion RecognitionCode3
Hierarchical Hypercomplex Network for Multimodal Emotion RecognitionCode2
PHemoNet: A Multimodal Network for Physiological SignalsCode2
Recent Trends of Multimodal Affective Computing: A Survey from NLP PerspectiveCode2
MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wildCode2
MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised LearningCode2
Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution ShiftsCode1
Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion RecognitionCode1
Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion RecognitionCode1
EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion UnderstandingCode1
SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion RecognitionCode1
Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion RecognitionCode1
Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion RecognitionCode1
Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross AttentionCode1
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion RecognitionCode1
Cooperative Sentiment Agents for Multimodal Sentiment AnalysisCode1
MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language ModelsCode1
Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion RecognitionCode1
Joint Multimodal Transformer for Emotion Recognition in the WildCode1
Curriculum Learning Meets Directed Acyclic Graph for Multimodal Emotion RecognitionCode1
GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion RecognitionCode1
Towards Emotion Analysis in Short-form Videos: A Large-Scale Dataset and BaselineCode1
Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion RecognitionCode1
Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality InteractionCode1
A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in ConversationsCode1
Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral Physiological SignalsCode1
CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion RecognitionCode1
A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party ConversationsCode1
Decoupled Multimodal Distilling for Emotion RecognitionCode1
Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal RepresentationsCode1
Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalitiesCode1
FV2ES: A Fully End2End Multimodal System for Fast Yet Effective Video Emotion Recognition InferenceCode1
GA2MIF: Graph and Attention Based Two-Stage Multi-Source Information Fusion for Conversational Emotion DetectionCode1
Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive LossCode1
GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion RecognitionCode1
COGMEN: COntextualized GNN based Multimodal Emotion recognitioNCode1
MMER: Multimodal Multi-task Learning for Speech Emotion RecognitionCode1
A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion RecognitionCode1
Predicting emotion from music videos: exploring the relative contribution of visual and auditory information to affective responsesCode1
Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion RecognitionCode1
A proposal for Multimodal Emotion Recognition using aural transformers and Action Units on RAVDESS datasetCode1
Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion ShiftsCode1
Cross Attentional Audio-Visual Fusion for Dimensional Emotion RecognitionCode1
A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognitionCode1
Multimodal Emotion Recognition with High-level Speech and Text FeaturesCode1
MSAF: Multimodal Split Attention FusionCode1
Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature FusionCode1
Show:102550
← PrevPage 1 of 4Next →
All datasetsIEMOCAP-4MELDIEMOCAPCMU-MOSEI-SentimentCMU-MOSEI-Sentiment-3Expressive hands and faces dataset (EHF).MELD-Sentiment

Benchmark Results

#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F186.52—Unverified
2JoyfulWeighted F185.7—Unverified
3COGMENWeighted F184.5—Unverified
4DANNAccuracy82.7—Unverified
5MMERAccuracy81.7—Unverified
6PATHOSnet v2Accuracy80.4—Unverified
7Self-attention weight correction (A+T)Accuracy76.8—Unverified
8CHFusionAccuracy76.5—Unverified
9bc-LSTMWeighted F174.1—Unverified
10Audio + Text (Stage III)F170.5—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F166.71—Unverified
2Audio + Text (Stage III)Weighted F165.8—Unverified
3JoyfulWeighted F161.77—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F172.81—Unverified
2JoyfulWeighted F170.5—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F144.93—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F166.73—Unverified
#ModelMetricClaimedVerifiedStatus
1SMPLify-Xv2v error52.9—Unverified
#ModelMetricClaimedVerifiedStatus
1GraphSmileWeighted F174.31—Unverified