Awesome-Code-LLM

👨‍💻 Awesome Code LLM

🔆 How to Contribute

Contributions are welcome! If you have any resources, tools, papers, or insights related to Code LLMs, feel free to submit a pull request. Let's work together to make this project better!

News

• 🔥🔥🔥 [2024-11-12] Qwen2.5-Coder series are released, offering six model sizes (0.5B, 1.5B, 3B, 7B, 14B, 32B), with Qwen2.5-Coder-32B-Instruct now the most powerful open-source code model.
• 🔥🔥 [2024-11-08] OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models is released.

🧵 Table of Contents

• 🧵 Table of Contents
• 🚀 Top Code LLMs
• 💡 Evaluation Toolkit
• 🚀 Awesome Code LLMs Leaderboard
• 📚 Awesome Code LLMs Papers
  • 🌊 Awesome Code Pre-Training Papers
  • 🐳 Awesome Code Instruction-Tuning Papers
  • 🐬 Awesome Code Alignment Papers
  • 🐋 Awesome Code Prompting Papers
  • 🐙 Awesome Code Benchmark & Evaluation Papers
• 🙌 Contributors
• Cite as
• Acknowledgement
• Star History

🚀 Top Code LLMs

Sort by HumanEval Pass@1

💡 Evaluation Toolkit:

• bigcode-evaluation-harness: A framework for the evaluation of autoregressive code generation language models.
• code-eval: A framework for the evaluation of autoregressive code generation language models on HumanEval.
• SandboxFusion: A secure sandbox for running and judging code generated by LLMs.

🚀 Awesome Code LLMs Leaderboard

📚 Awesome Code LLMs Papers

🌊 Awesome Code Pre-Training Papers

Title	Venue	Date	Code	Resources
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

                        | `Preprint` | `2024.11` | [Github](https://github.com/OpenCoder-llm/OpenCoder-llm)   | [HF](https://huggingface.co/infly/OpenCoder-8B-Instruct)                          |

|
Qwen2.5-Coder Technical Report
| Preprint | 2024.09 | Github | HF | |
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
| Preprint | 2024.06 | Github | HF | |
StarCoder 2 and The Stack v2: The Next Generation
| Preprint | 2024.02 | Github | HF | |
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
| Preprint | 2024.01 | Github | HF | |
Code Llama: Open Foundation Models for Code
| Preprint | 2023.08 | Github | HF | | Textbooks Are All You Need
| Preprint | 2023.06 | - | HF | |
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
| Preprint | 2023.05 | Github | HF | |
StarCoder: may the source be with you!
| Preprint | 2023.05 | Github | HF | |
CodeGen2: Lessons for Training LLMs on Programming and Natural Languages
| ICLR23 | 2023.05 | Github | HF | |
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X
| Preprint | 2023.03 | Github | HF | | SantaCoder: don't reach for the stars!
| Preprint | 2023.01 | - | HF | |
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
| ICLR'23 | 2022.03 | Github | HF | |
Evaluating Large Language Models Trained on Code
| Preprint | 2021.07 | Github | - |

🐳 Awesome Code Instruction-Tuning Papers

Title	Venue	Date	Code	Resources
Magicoder: Source Code Is All You Need

                                                             | `ICML'24`  | `2023.12` | [Github](https://github.com/ise-uiuc/magicoder)       | [HF](https://huggingface.co/ise-uiuc/Magicoder-DS-6.7B)        |

|
OctoPack: Instruction Tuning Code Large Language Models
| ICLR'24 | 2023.08 | Github | HF | |
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
| Preprint | 2023.07 | Github | HF | |
Code Alpaca: An Instruction-following LLaMA Model trained on code generation instructions
| Preprint | 2023.xx | Github | HF |

🐬 Awesome Code Alignment Papers

                                                                                                                          | `Preprint`   | `2024.11` | -                                                             | -         |

| PLUM: Preference Learning Plus Test Cases Yields Better Code Language Models
| Preprint | 2024.06 | - | - | | PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback
| Preprint | 2023.07 | - | - | |
RLTF: Reinforcement Learning from Unit Test Feedback
| Preprint | 2023.07 | Github | - | |
Execution-based Code Generation using Deep Reinforcement Learning
| TMLR'23 | 2023.01 | Github | - | |
CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
| NeurIPS'22 | 2022.07 | Github | - |

🐋 Awesome Code Prompting Papers

Title	Venue	Date	Code	Resources
From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging

            | `Preprint` | `2024.10` | [Github](https://github.com/YerbaPage/MGDebugger)                      | -         |

|
Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs
| AAAI'25 | 2024.06 | Github | - | |
Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step
| ACL'24 | 2024.02 | Github | - | | SelfEvolve: A Code Evolution Framework via Large Language Models
| Preprint | 2023.06 | - | - | |
Demystifying GPT Self-Repair for Code Generation
| ICLR'24 | 2023.06 | Github | - | | Teaching Large Language Models to Self-Debug
| ICLR'24 | 2023.06 | - | - | |
LEVER: Learning to Verify Language-to-Code Generation with Execution
| ICML'23 | 2023.02 | Github | - | |
Coder Reviewer Reranking for Code Generation
| ICML'23 | 2022.11 | Github | - | |
CodeT: Code Generation with Generated Tests
| ICLR'23 | 2022.07 | Github | - |

🐙 Awesome Code Benchmark & Evaluation Papers

Dataset	Title	Venue	Date	Code	Resources
CodeArena
Evaluating and Aligning CodeLLMs on Human Preference

                                                        | `Preprint`   | `2024.12` | [Github](https://github.com/QwenLM/Qwen2.5-Coder/tree/main/qwencoder-eval/instruct/CodeArena) | [HF](https://huggingface.co/datasets/CSJianYang/CodeArena)                                                          |

| FullStack Bench |
FullStack Bench: Evaluating LLMs as Full Stack Coders
| Preprint | 2024.12 | Github | HF Github | | GitChameleon |
GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models
| Preprint | 2024.11 | Github | - | | Evalperf |
Evaluating Language Models for Efficient Code Generation
| COLM'24 | 2024.08 | Github | HF | | LiveCodeBench |
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
| Preprint | 2024.03 | Github | HF | | DevBench |
DevBench: A Comprehensive Benchmark for Software Development
| Preprint | 2024.03 | Github | - | | SWE-bench |
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
| ICLR'24 | 2024.03 | Github | HF | | CrossCodeEval |
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
| NeurIPS'23 | 2023.11 | Github | - | | RepoCoder |
Repository-Level Code Completion Through Iterative Retrieval and Generation
| EMNLP'23 | 2023.10 | Github | - | | LongCoder |
LongCoder: A Long-Range Pre-trained Language Model for Code Completion
| ICML'23 | 2023.10 | Github | - | | - | Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation
| Preprint | 2023.08 | - | - | | BioCoder |
BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models
| ISMB'24 | 2023.08 | Github | - | | RepoBench |
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
| ICLR'24 | 2023.06 | Github | HF | | Evalplus |
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
| NeurIPS'23 | 2023.05 | Github | HF | | Coeditor |
Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing
| ICLR'24 | 2023.05 | Github | - | | DS-1000 |
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
| ICML'23 | 2022.11 | Github | HF | | MultiPL-E |
MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
| Preprint | 2022.08 | Github | HF | | MBPP |
Program Synthesis with Large Language Models
| Preprint | 2021.08 | Github | HF | | APPS |
Measuring Coding Challenge Competence With APPS
| NeurIPS'21 | 2021.05 | Github | HF |

🙌 Contributors

This is an active repository and your contributions are always welcome! If you have any question about this opinionated list, do not hesitate to contact me [email protected].

Cite as

@software{awesome-code-llm,
  author = {Binyuan Hui, Lei Zhang},
  title = {An awesome and curated list of best code-LLM for research},
  howpublished = {\url{https://github.com/huybery/Awesome-Code-LLM}},
  year = 2023,
}

Acknowledgement

This project is inspired by Awesome-LLM.

Star History

⬆ Back to ToC