Video Captioning

Video Captioning is a task of automatic captioning a video by understanding the action and event in the video which can help in the retrieval of the video efficiently through text.

Source: NITS-VC System for VATEX Video Captioning Challenge 2020

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 176–200 of 473 papers

Title	Date	Tasks	Status	Hype
Models See Hallucinations: Evaluating the Factuality in Video Captioning	Mar 6, 2023	Text GenerationVideo Captioning	—Unverified	0
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning	Feb 27, 2023	Dense Video CaptioningLanguage Modeling	CodeCode Available	2
STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-training	Feb 20, 2023	Language ModellingObject	—Unverified	0
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video	Feb 1, 2023	Action ClassificationImage Classification	CodeCode Available	4
Temporal Perceiving Video-Language Pre-training	Jan 18, 2023	Action LocalizationContrastive Learning	—Unverified	0
Exploiting Auxiliary Caption for Video Grounding	Jan 15, 2023	Contrastive LearningDense Video Captioning	—Unverified	0
HiVLP: Hierarchical Interactive Video-Language Pre-Training	Jan 1, 2023	RetrievalSelf-Supervised Learning	—Unverified	0
ReGen: A good Generative Zero-Shot Video Classifier Should be Rewarded	Jan 1, 2023	Action ClassificationAction Recognition	—Unverified	0
Exploring Group Video Captioning with Efficient Relational Approximation	Jan 1, 2023	Video Captioning	—Unverified	0
Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?	Dec 31, 2022	Data AugmentationRetrieval	CodeCode Available	2
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training	Dec 30, 2022	cross-modal alignmentTGIF-Action	—Unverified	0
METEOR Guided Divergence for Video Captioning	Dec 20, 2022	Hierarchical Reinforcement LearningScene Understanding	CodeCode Available	0
Contextual Explainable Video Representation: Human Perception-based Understanding	Dec 12, 2022	Action DetectionAction Recognition	CodeCode Available	0
MAViC: Multimodal Active Learning for Video Captioning	Dec 11, 2022	Active LearningDecoder	—Unverified	0
VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners	Dec 9, 2022	Question AnsweringRetrieval	—Unverified	0
VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning	Nov 28, 2022	DiversitySentence	CodeCode Available	1
Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning	Nov 28, 2022	FADVideo Captioning	CodeCode Available	0
Aligning Source Visual and Target Language Domains for Unpaired Video Captioning	Nov 22, 2022	TranslationVideo Captioning	—Unverified	0
Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations	Nov 21, 2022	Contrastive LearningRepresentation Learning	CodeCode Available	1
Visual Commonsense-aware Representation Network for Video Captioning	Nov 17, 2022	Caption GenerationQuestion Answering	CodeCode Available	1
Event and Entity Extraction from Generated Video Captions	Nov 5, 2022	Caption GenerationDense Video Captioning	CodeCode Available	0
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality	Nov 1, 2022	Data AugmentationImage Retrieval	CodeCode Available	1
Vision-Language Pre-training: Basics, Recent Advances, and Future Trends	Oct 17, 2022	Few-Shot LearningImage Captioning	CodeCode Available	3
Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks	Oct 10, 2022	RetrievalText to Video Retrieval	—Unverified	0
Thinking Hallucination for Video Captioning	Sep 28, 2022	HallucinationVideo Captioning	CodeCode Available	1

Show:10 25 50

← PrevPage 8 of 19Next →

All datasets MSR-VTT MSVD YouCook2 VATEX ActivityNet Captions MSRVTT-CTN MSVD-CTN Hindi MSR-VTT TVC ChinaOpen-1k MSVD-Indonesian Shot2Story20K

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	mPLUG-2	CIDEr	80	—	Unverified
2	VAST	CIDEr	78	—	Unverified
3	GIT2	CIDEr	75.9	—	Unverified
4	VLAB	CIDEr	74.9	—	Unverified
5	COSA	CIDEr	74.7	—	Unverified
6	VALOR	CIDEr	74	—	Unverified
7	MaMMUT (ours)	CIDEr	73.6	—	Unverified
8	VideoCoCa	CIDEr	73.2	—	Unverified
9	RTQ	CIDEr	69.3	—	Unverified
10	HowToCaption	CIDEr	65.3	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	MaMMUT	CIDEr	195.6	—	Unverified
2	VLAB	CIDEr	179.8	—	Unverified
3	COSA	CIDEr	178.5	—	Unverified
4	VALOR	CIDEr	178.5	—	Unverified
5	mPLUG-2	CIDEr	165.8	—	Unverified
6	HowToCaption	CIDEr	154.2	—	Unverified
7	HiTeA	CIDEr	146.9	—	Unverified
8	Vid2Seq	CIDEr	146.2	—	Unverified
9	VIOLETv2	CIDEr	139.2	—	Unverified
10	RTQ	CIDEr	123.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VAST	BLEU-4	18.2	—	Unverified
2	UniVL + MELTR	BLEU-4	17.92	—	Unverified
3	UniVL	BLEU-4	17.35	—	Unverified
4	VideoCoCa	BLEU-4	14.2	—	Unverified
5	VLM	BLEU-4	12.27	—	Unverified
6	E2vidD6-MASSvid-BiD	BLEU-4	12.04	—	Unverified
7	TextKG	BLEU-4	11.7	—	Unverified
8	COOT	BLEU-4	11.3	—	Unverified
9	COSA	BLEU-4	10.1	—	Unverified
10	HowToCaption	BLEU-4	8.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VALOR	BLEU-4	45.6	—	Unverified
2	VAST	BLEU-4	45	—	Unverified
3	COSA	BLEU-4	43.7	—	Unverified
4	VideoCoCa	BLEU-4	39.7	—	Unverified
5	IcoCap (ViT-B/16)	BLEU-4	37.4	—	Unverified
6	IcoCap (ViT-B/32)	BLEU-4	36.9	—	Unverified
7	VASTA (Kinetics-backbone)	BLEU-4	36.25	—	Unverified
8	CoCap (ViT/L14)	BLEU-4	35.8	—	Unverified
9	ORG-TRL	BLEU-4	32.1	—	Unverified
10	NITS-VC	BLEU-4	20	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VideoCoCa	BLEU4	14.7	—	Unverified
2	VLTinT (ae-test split) C3D/Ling	BLEU4	14.5	—	Unverified
3	VLCap (ae-test split) - Appearance + Language	BLEU4	13.38	—	Unverified
4	COOT (ae-test split) - Only Appearance features	BLEU4	10.85	—	Unverified
5	MART (ae-test split) - Appearance + Flow	BLEU4	10.33	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	CEN	CIDEr	49.87	—	Unverified
2	GIT	CIDEr	32.43	—	Unverified
3	SEM-POS	CIDEr	26.01	—	Unverified
4	AKGNN	CIDEr	25.9	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	CEN	CIDEr	63.51	—	Unverified
2	GIT	CIDEr	45.63	—	Unverified
3	SEM-POS	CIDEr	37.16	—	Unverified
4	AKGNN	CIDEr	35.08	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	SBD_Keyframe	BLEU4	41.01	—	Unverified
2	V+S-Att-based	BLEU4	36.2	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VAST	BLEU-4	19.9	—	Unverified
2	COSA	BLEU-4	18.8	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	GVT	BLEU4	17.7	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	VNS-GRU (Cross-Lingual)	BLEU-4	58.68	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Shot2Story	CIDEr	37.4	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Vid2Seq	CIDEr	120.5	—	Unverified