Large Language Models Can Self-Improve
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han
Unverified — Be the first to reproduce this paper.
ReproduceAbstract
Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| GSM8K | PaLM 540B (Self Improvement, Self Consistency) | Accuracy | 82.1 | — | Unverified |
| GSM8K | PaLM 540B (Standard-Prompting) | Accuracy | 17.9 | — | Unverified |
| GSM8K | PaLM 540B (Self Improvement, Standard-Prompting) | Accuracy | 32.2 | — | Unverified |
| GSM8K | PaLM 540B (CoT Prompting) | Accuracy | 56.5 | — | Unverified |
| GSM8K | PaLM 540B (Self Improvement, CoT Prompting) | Accuracy | 73.5 | — | Unverified |
| GSM8K | PaLM 540B (Self Consistency) | Accuracy | 74.4 | — | Unverified |