| Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet' | Oct 29, 2024 | Code CompletionCode Generation | CodeCode Available | 1 | 5 |
| HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation | Dec 30, 2024 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks | Oct 16, 2024 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| How to Select Datapoints for Efficient Human Evaluation of NLG Models? | Jan 30, 2025 | HumanEvalMachine Translation | CodeCode Available | 1 | 5 |
| CYCLE: Learning to Self-Refine the Code Generation | Mar 27, 2024 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning | Feb 14, 2024 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct | Jul 8, 2024 | Code GenerationCode Summarization | CodeCode Available | 1 | 5 |
| InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models | Mar 11, 2024 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| ReCode: Robustness Evaluation of Code Generation Models | Dec 20, 2022 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair | Dec 25, 2023 | HumanEvalparameter-efficient fine-tuning | CodeCode Available | 1 | 5 |
| ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation | Aug 3, 2023 | Class-level Code GenerationCode Generation | CodeCode Available | 1 | 5 |
| Getting the most out of your tokenizer for pre-training and domain adaptation | Feb 1, 2024 | Code GenerationDomain Adaptation | CodeCode Available | 1 | 5 |
| HALO: Hierarchical Autonomous Logic-Oriented Orchestration for Multi-Agent LLM Systems | May 17, 2025 | Arithmetic ReasoningCode Generation | CodeCode Available | 1 | 5 |
| CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion | Oct 17, 2023 | Code CompletionHumanEval | CodeCode Available | 1 | 5 |
| Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding | Feb 19, 2024 | HumanEvalLanguage Modeling | CodeCode Available | 1 | 5 |
| How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data | Sep 5, 2024 | Code GenerationDiversity | CodeCode Available | 1 | 5 |
| Better & Faster Large Language Models via Multi-token Prediction | Apr 30, 2024 | HumanEvalmbpp | CodeCode Available | 1 | 5 |
| Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models | Feb 24, 2024 | HumanEvalMemorization | CodeCode Available | 1 | 5 |
| CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models | Feb 23, 2025 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
| ContraCLM: Contrastive Learning For Causal Language Model | Oct 3, 2022 | Code GenerationCode Search | CodeCode Available | 1 | 5 |
| Multiple-Choice Questions are Efficient and Robust LLM Evaluators | May 20, 2024 | GSM8KHumanEval | CodeCode Available | 1 | 5 |
| ANPL: Towards Natural Programming with Interactive Decomposition | May 29, 2023 | ARCCode Generation | CodeCode Available | 1 | 5 |
| Multi-lingual Evaluation of Code Generation Models | Oct 26, 2022 | Code CompletionCode Generation | CodeCode Available | 1 | 5 |
| How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark | Jun 10, 2024 | HumanEvalProgram Synthesis | CodeCode Available | 1 | 5 |
| RepoST: Scalable Repository-Level Coding Environment Construction with Sandbox Testing | Mar 10, 2025 | Code GenerationHumanEval | CodeCode Available | 1 | 5 |
