Video Captioning

Video Captioning is a task of automatic captioning a video by understanding the action and event in the video which can help in the retrieval of the video efficiently through text.

Source: NITS-VC System for VATEX Video Captioning Challenge 2020

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 126–150 of 473 papers

Title	Date	Tasks	Status	Hype
Multi-modal Dense Video Captioning	Mar 17, 2020	Automatic Speech RecognitionAutomatic Speech Recognition (ASR)	CodeCode Available	1
Shot2Story20K: A New Benchmark for Comprehensive Understanding of Multi-shot Videos	Dec 16, 2023	Video Captioningvideo narration captioning	CodeCode Available	1
From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping	Apr 26, 2023	DecoderImage Captioning	CodeCode Available	1
VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation	Feb 18, 2025	Text-to-Video GenerationVideo Captioning	CodeCode Available	1
Hierarchical Video-Moment Retrieval and Step-Captioning	Mar 29, 2023	Information RetrievalMoment Retrieval	CodeCode Available	1
Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation	Aug 17, 2016	Caption GenerationDecoder	CodeCode Available	1
IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning	Sep 26, 2024	Image CaptioningRetrieval	CodeCode Available	1
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization	Oct 9, 2024	Audio captioningLarge Language Model	—Unverified	0
Capturing Rich Behavior Representations: A Dynamic Action Semantic-Aware Graph Transformer for Video Captioning	Feb 19, 2025	Knowledge DistillationObject	—Unverified	0
End-to-end Generative Pretraining for Multimodal Video Captioning	Jan 20, 2022	Action ClassificationDecoder	—Unverified	0
A Restricted Visual Turing Test for Deep Scene and Event Understanding	Dec 6, 2015	Question AnsweringVideo Captioning	—Unverified	0
Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning	Jun 5, 2017	Caption GenerationDecoder	—Unverified	0
Prediction and Description of Near-Future Activities in Video	Aug 2, 2019	PredictionVideo Captioning	—Unverified	0
End-to-end Dense Video Captioning as Sequence Generation	Apr 18, 2022	Dense Video CaptioningDescriptive	—Unverified	0
End-to-end Dense Video Captioning as Sequence Generation	Jan 16, 2022	Dense Video CaptioningDescriptive	—Unverified	0
Adaptive Feature Abstraction for Translating Video to Text	Nov 23, 2016	Video Captioning	—Unverified	0
Hierarchical memory decoder for visual narrating	Sep 1, 2020	DecoderImage Captioning	—Unverified	0
End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering	Oct 10, 2016	Language ModelingLanguage Modelling	—Unverified	0
Encoder-Decoder Based Long Short-Term Memory (LSTM) Model for Video Captioning	Oct 2, 2023	DecoderSentence	—Unverified	0
Empirical Autopsy of Deep Video Captioning Frameworks	Nov 21, 2019	DecoderLanguage Modelling	—Unverified	0
FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning	Oct 20, 2024	DiagnosticVideo Captioning	—Unverified	0
E-MMAD: Multimodal Advertising Caption Generation Based on Structured Information	Nov 16, 2021	Caption Generationvalid	—Unverified	0
AdaCM^2: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction	Jan 1, 2025	GPUQuestion Answering	—Unverified	0
Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives	Dec 14, 2024	DescriptiveLanguage Modeling	—Unverified	0
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos	Dec 25, 2023	Image GenerationText to Image Generation	—Unverified	0

Show:10 25 50

← PrevPage 6 of 19Next →

All datasets MSR-VTT MSVD YouCook2 VATEX ActivityNet Captions MSRVTT-CTN MSVD-CTN Hindi MSR-VTT TVC ChinaOpen-1k MSVD-Indonesian Shot2Story20K

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	mPLUG-2	CIDEr	80	—	Unverified
2	VAST	CIDEr	78	—	Unverified
3	GIT2	CIDEr	75.9	—	Unverified
4	VLAB	CIDEr	74.9	—	Unverified
5	COSA	CIDEr	74.7	—	Unverified
6	VALOR	CIDEr	74	—	Unverified
7	MaMMUT (ours)	CIDEr	73.6	—	Unverified
8	VideoCoCa	CIDEr	73.2	—	Unverified
9	RTQ	CIDEr	69.3	—	Unverified
10	HowToCaption	CIDEr	65.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	MaMMUT	CIDEr	195.6	—	Unverified
2	VLAB	CIDEr	179.8	—	Unverified
3	COSA	CIDEr	178.5	—	Unverified
4	VALOR	CIDEr	178.5	—	Unverified
5	mPLUG-2	CIDEr	165.8	—	Unverified
6	HowToCaption	CIDEr	154.2	—	Unverified
7	HiTeA	CIDEr	146.9	—	Unverified
8	Vid2Seq	CIDEr	146.2	—	Unverified
9	VIOLETv2	CIDEr	139.2	—	Unverified
10	RTQ	CIDEr	123.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VAST	BLEU-4	18.2	—	Unverified
2	UniVL + MELTR	BLEU-4	17.92	—	Unverified
3	UniVL	BLEU-4	17.35	—	Unverified
4	VideoCoCa	BLEU-4	14.2	—	Unverified
5	VLM	BLEU-4	12.27	—	Unverified
6	E2vidD6-MASSvid-BiD	BLEU-4	12.04	—	Unverified
7	TextKG	BLEU-4	11.7	—	Unverified
8	COOT	BLEU-4	11.3	—	Unverified
9	COSA	BLEU-4	10.1	—	Unverified
10	HowToCaption	BLEU-4	8.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VALOR	BLEU-4	45.6	—	Unverified
2	VAST	BLEU-4	45	—	Unverified
3	COSA	BLEU-4	43.7	—	Unverified
4	VideoCoCa	BLEU-4	39.7	—	Unverified
5	IcoCap (ViT-B/16)	BLEU-4	37.4	—	Unverified
6	IcoCap (ViT-B/32)	BLEU-4	36.9	—	Unverified
7	VASTA (Kinetics-backbone)	BLEU-4	36.25	—	Unverified
8	CoCap (ViT/L14)	BLEU-4	35.8	—	Unverified
9	ORG-TRL	BLEU-4	32.1	—	Unverified
10	NITS-VC	BLEU-4	20	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VideoCoCa	BLEU4	14.7	—	Unverified
2	VLTinT (ae-test split) C3D/Ling	BLEU4	14.5	—	Unverified
3	VLCap (ae-test split) - Appearance + Language	BLEU4	13.38	—	Unverified
4	COOT (ae-test split) - Only Appearance features	BLEU4	10.85	—	Unverified
5	MART (ae-test split) - Appearance + Flow	BLEU4	10.33	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	CEN	CIDEr	49.87	—	Unverified
2	GIT	CIDEr	32.43	—	Unverified
3	SEM-POS	CIDEr	26.01	—	Unverified
4	AKGNN	CIDEr	25.9	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	CEN	CIDEr	63.51	—	Unverified
2	GIT	CIDEr	45.63	—	Unverified
3	SEM-POS	CIDEr	37.16	—	Unverified
4	AKGNN	CIDEr	35.08	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SBD_Keyframe	BLEU4	41.01	—	Unverified
2	V+S-Att-based	BLEU4	36.2	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VAST	BLEU-4	19.9	—	Unverified
2	COSA	BLEU-4	18.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	GVT	BLEU4	17.7	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VNS-GRU (Cross-Lingual)	BLEU-4	58.68	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Shot2Story	CIDEr	37.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Vid2Seq	CIDEr	120.5	—	Unverified