Logical Reasoning

Papers

Recently Added Most Hyped Most Active Needs Verification Most Verified

Showing 76–100 of 747 papers

Title	Date	Tasks	Status	Hype
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models	Oct 7, 2024	GSM8KLogical Reasoning	CodeCode Available	1
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning	Oct 1, 2024	Logical Reasoning	CodeCode Available	1
VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning	Sep 3, 2024	Chart Question AnsweringData Visualization	CodeCode Available	1
LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models	Aug 28, 2024	BenchmarkingLogical Reasoning	CodeCode Available	1
CHECKWHY: Causal Fact Verification via Argument Structure	Aug 20, 2024	Fact VerificationLogical Reasoning	CodeCode Available	1
Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding	Jul 11, 2024	EEGLanguage Modeling	CodeCode Available	1
R^2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning	Jul 8, 2024	Logical Reasoning	CodeCode Available	1
ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models	Jul 7, 2024	FairnessGeneral Knowledge	CodeCode Available	1
LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts	Jul 6, 2024	Logical ReasoningMathematical Reasoning	CodeCode Available	1
PUZZLES: A Benchmark for Neural Algorithmic Reasoning	Jun 29, 2024	Decision MakingLogical Reasoning	CodeCode Available	1
VideoVista: A Versatile Benchmark for Video Understanding and Reasoning	Jun 17, 2024	Anomaly DetectionLogical Reasoning	CodeCode Available	1
A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners	Jun 16, 2024	Logical Reasoning	CodeCode Available	1
LogiCode: an LLM-Driven Framework for Logical Anomaly Detection	Jun 7, 2024	Anomaly DetectionBinary Classification	CodeCode Available	1
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models	Apr 23, 2024	Logical ReasoningQuestion Answering	CodeCode Available	1
LeanReasoner: Boosting Complex Logical Reasoning with Lean	Mar 20, 2024	Automated Theorem ProvingLogical Reasoning	CodeCode Available	1
SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials	Feb 22, 2024	Chart Question AnsweringLanguage Modeling	CodeCode Available	1
OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models	Feb 21, 2024	General KnowledgeLogical Reasoning	CodeCode Available	1
Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs	Feb 18, 2024	Logical Reasoning	CodeCode Available	1
The Quantified Boolean Bayesian Network: Theory and Experiments with a Logical Graphical Model	Feb 9, 2024	Information RetrievalLanguage Modelling	CodeCode Available	1
Conditional and Modal Reasoning in Large Language Models	Jan 30, 2024	Logical Reasoning	CodeCode Available	1
Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions	Jan 17, 2024	Arithmetic ReasoningCode Generation	CodeCode Available	1
LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models	Jan 1, 2024	Code GenerationIn-Context Learning	CodeCode Available	1
TEILP: Time Prediction over Knowledge Graphs via Logical Reasoning	Dec 25, 2023	Knowledge GraphsLogical Reasoning	CodeCode Available	1
Advancing Abductive Reasoning in Knowledge Graphs through Complex Logical Hypothesis Generation	Dec 25, 2023	Knowledge GraphsLogical Reasoning	CodeCode Available	1
Modeling Complex Mathematical Reasoning via Large Language Model based MathAgent	Dec 14, 2023	Language ModelingLanguage Modelling	CodeCode Available	1

Show:10 25 50

← PrevPage 4 of 30Next →

All datasets LingOly BIG-bench (Formal Fallacies Syllogisms Negation)BIG-bench (Penguins In A Table)BIG-bench (Reasoning About Colored Objects)BIG-bench (Temporal Sequences)BIG-bench (Logic Grid Puzzle)BIG-bench (StrategyQA)RuWorldTree Winograd Automatic BIG-bench (Logical Fallacy Detection)

Benchmark Results

#	Model	Metric	Claimed	Verified	Status
1	Claude Opus	Delta_NoContext	28.8	—	Unverified
2	GPT-4o	Delta_NoContext	25.1	—	Unverified
3	Gemini 1.5 Pro	Delta_NoContext	23.4	—	Unverified
4	GPT-4	Delta_NoContext	21.5	—	Unverified
5	Command R+	Delta_NoContext	11.6	—	Unverified
6	GPT-3.5	Delta_NoContext	11.2	—	Unverified
7	Mixtral 8x7B	Delta_NoContext	6.4	—	Unverified
8	Llama 3 8B	Delta_NoContext	4.9	—	Unverified
9	Llama 3 70B	Delta_NoContext	2.9	—	Unverified
10	Gemma 7B	Delta_NoContext	2.2	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	PaLM 2 (few-shot, k=3, Direct)	Accuracy	64.8	—	Unverified
2	PaLM 2 (few-shot, k=3, CoT)	Accuracy	57.2	—	Unverified
3	OPT 66B (few-shot, k=3)	Accuracy	54	—	Unverified
4	PaLM 540B (few-shot, k=3)	Accuracy	53.6	—	Unverified
5	GPT-NeoX 20B (few-shot, k=3)	Accuracy	52.8	—	Unverified
6	BLOOM 176B (few-shot, k=3)	Accuracy	52.8	—	Unverified
7	Chinchilla-70B (few-shot, k=5)	Accuracy	52.1	—	Unverified
8	Bloomberg GPT 50B (few-shot, k=3)	Accuracy	50.8	—	Unverified
9	Gopher-280B (few-shot, k=5)	Accuracy	50.7	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	PaLM 2 (few-shot, k=3, CoT)	Accuracy	84.9	—	Unverified
2	PaLM 2 (few-shot, k=3, Direct)	Accuracy	65.8	—	Unverified
3	Chinchilla-70B (few-shot, k=5)	Accuracy	48.7	—	Unverified
4	PaLM 540B (few-shot, k=3)	Accuracy	44.5	—	Unverified
5	Gopher-280B (few-shot, k=5)	Accuracy	40.6	—	Unverified
6	BLOOM 176B (few-shot, k=3)	Accuracy	40.41	—	Unverified
7	Bloomberg GPT (few-shot, k=3)	Accuracy	37.67	—	Unverified
8	GPT-NeoX (few-shot, k=3)	Accuracy	33.56	—	Unverified
9	OPT 66B (few-shot, k=3)	Accuracy	28.08	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	PaLM 2 (few-shot, k=3, CoT)	Accuracy	91.2	—	Unverified
2	PaLM 2 (few-shot, k=3, Direct)	Accuracy	61.2	—	Unverified
3	Chinchilla-70B (few-shot, k=5)	Accuracy	59.7	—	Unverified
4	Gopher-280B (few-shot, k=5)	Accuracy	49.2	—	Unverified
5	PaLM 540B (few-shot, k=3)	Accuracy	38	—	Unverified
6	BLOOM 176B (few-shot, k=3)	Accuracy	36.8	—	Unverified
7	Bloomberg GPT (few-shot, k=3)	Accuracy	34.8	—	Unverified
8	OPT 66B (few-shot, k=3)	Accuracy	31.2	—	Unverified
9	GPT-NeoX (few-shot, k=3)	Accuracy	26	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	PaLM 2 (few-shot, k=3, CoT)	Accuracy	100	—	Unverified
2	PaLM 2 (few-shot, k=3, Direct)	Accuracy	96.4	—	Unverified
3	PaLM 540B (few-shot, k=3)	Accuracy	39.6	—	Unverified
4	BLOOM 176B (few-shot, k=3)	Accuracy	36.8	—	Unverified
5	Chinchilla-70B (few-shot, k=5)	Accuracy	32	—	Unverified
6	Bloomberg GPT (few-shot, k=3)	Accuracy	29.2	—	Unverified
7	OPT 66B (few-shot, k=3)	Accuracy	23.6	—	Unverified
8	GPT-NeoX (few-shot, k=3)	Accuracy	21.2	—	Unverified
9	Gopher-280B (few-shot, k=5)	Accuracy	19	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Chinchilla-70B (few-shot, k=5)	Accuracy	44	—	Unverified
2	PaLM-540B (few-shot, k=5)	Accuracy	42.4	—	Unverified
3	PaLM-62B (few-shot, k=5)	Accuracy	36.5	—	Unverified
4	Gopher-280B (few-shot, k=5)	Accuracy	35.1	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	PaLM-540B (few-shot, k=5)	Accuracy	73.9	—	Unverified
2	Chinchilla-70B (few-shot, k=5)	Accuracy	68.3	—	Unverified
3	PaLM-62B (few-shot, k=5)	Accuracy	65.4	—	Unverified
4	Gopher-280B (few-shot, k=5)	Accuracy	61	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Human benchmark	Accuracy	83.7	—	Unverified
2	RuGPT-3 Large	Accuracy	40.7	—	Unverified
3	RuGPT-3 Medium	Accuracy	38	—	Unverified
4	RuGPT-3 Small	Accuracy	34	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Human benchmark	Accuracy	87	—	Unverified
2	RuGPT-3 Small	Accuracy	57.9	—	Unverified
3	RuGPT-3 Medium	Accuracy	57.2	—	Unverified
4	RuGPT-3 Large	Accuracy	55.5	—	Unverified

#	Model	Metric	Claimed	Verified	Status
1	Chinchilla-70B (few-shot, k=5)	Accuracy	72.1	—	Unverified
2	Gopher-280B (few-shot, k=5)	Accuracy	58.9	—	Unverified