Code Completion
Papers
No papers found.
Benchmark Results
| # | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| 1 | deepseek-coder-33b-base | Average | 69.01 | — | Unverified |
| 2 | deepseek-coder-6.7b-base | Average | 63.4 | — | Unverified |
| 3 | starcoderbase | Average | 55.54 | — | Unverified |
| 4 | gpt-4-1106-preview | Average | 53.28 | — | Unverified |
| 5 | CodeLlama-13b-hf | Average | 52.78 | — | Unverified |
| 6 | deepseek-coder-1.3b-base | Average | 52.63 | — | Unverified |
| 7 | CodeLlama-34b-hf | Average | 49.66 | — | Unverified |
| 8 | CodeLlama-7b-hf | Average | 45 | — | Unverified |
| 9 | gpt-3.5-turbo-0301 | Average | 40.86 | — | Unverified |
| 10 | incoder-6B | Average | 33.79 | — | Unverified |
| # | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| 1 | CodeGPT-adapted | Accuracy (token-level) | 77.13 | — | Unverified |
| 2 | CodeT5+ 770M | EM (line-level) | 37.9 | — | Unverified |
| 3 | CodeT5+ 220M | EM (line-level) | 35.17 | — | Unverified |
| # | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| 1 | CodeGPT-adapted | Accuracy (token-level) | 75.11 | — | Unverified |
| 2 | CodeT5+ 770M | EM (line-level) | 44.86 | — | Unverified |
| 3 | CodeT5+ 220M | EM (line-level) | 43.42 | — | Unverified |
| # | Model | Metric | Claimed | Verified | Status |
|---|---|---|---|---|---|
| 1 | SantaCoder-MGD | Compilation Rate | 73.03 | — | Unverified |
| 2 | SantaCoder | Compilation Rate | 59.97 | — | Unverified |
| 3 | SantaCoder | Compilation Rate | 59.79 | — | Unverified |