Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/facebookresearch/codellamaOfficialIn paperpytorch★ 16,338
- github.com/BohdanPetryshyn/code-llama-fim-fine-tuningpytorch★ 63
Abstract
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.
Tasks
Benchmark Results
| Dataset | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| MBPP | Code Llama - Python 70B (3-shot) | Accuracy | 65.5 | — | Unverified |
| MBPP | Code Llama 70B (3-shot) | Accuracy | 62.4 | — | Unverified |
| MBPP | Code Llama - Instruct 70B (3-shot) | Accuracy | 62.2 | — | Unverified |
| MBPP | Unnatural Code Llama 34B (3-shot) | Accuracy | 61.2 | — | Unverified |
| MBPP | Code Llama - Instruct 34B (3-shot) | Accuracy | 57 | — | Unverified |
| MBPP | Code Llama - Python 34B (3-shot) | Accuracy | 56.2 | — | Unverified |
| MBPP | Code Llama 34B (3-shot) | Accuracy | 55 | — | Unverified |
| MBPP | GPT-3.5 Turbo | Accuracy | 52.2 | — | Unverified |
| MBPP | Code Llama - Instruct 13B (3-shot) | Accuracy | 49.4 | — | Unverified |
| MBPP | Code Llama - Python 13B (3-shot) | Accuracy | 49 | — | Unverified |
| MBPP | Code Llama - Python 7B (3-shot) | Accuracy | 47.6 | — | Unverified |
| MBPP | Code Llama 13B (3-shot) | Accuracy | 47 | — | Unverified |
| MBPP | Code Llama - Instruct 7B (3-shot) | Accuracy | 44.4 | — | Unverified |
| MBPP | Code Llama 7B (3-shot) | Accuracy | 41.4 | — | Unverified |