For paper "From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging".

🔥Update: MGDebugger achieves 100% accuracy on HumanEval with the DeepSeek-R1 model in our latest experiments! Results have been uploaded in /supplementary/dsr1_results.jsonl

Table of Contents

• MGDebugger: Multi-Granularity LLM Debugger
  • Table of Contents
  • Introduction
  • Getting Started
    • Prerequisites
    • Configuring the vLLM Server
  • Usage
    • Running the Demo
    • Running Experiments
    • Log Management
  • Performance
  • Contributing

Introduction

MGDebugger is a hierarchical LLM code debugging method designed to isolate, identify, and resolve errors at various levels of granularity. Using a hierarchical bottom-up debugging approach, MGDebugger systematically progresses from individual subfunctions to the overall system, enabling precise error detection and correction.

With MGDebugger, developers can efficiently debug complex codes and functions by performing granular analysis, reducing debugging time, and improving the success rate of resolving complex issues.

Getting Started

Prerequisites

Before running MGDebugger, ensure your environment meets the following requirements:

• Python: Version 3.8 or later.

• vLLM: Version 0.6.0 or later. Required for model loading and inference. You can follow the official installation guide to set it up.

• Additional dependencies: Install all necessary Python packages using the following command:

  There are some packages not supported on Mac such as auto_gptq and autoawq, you can remove these requirements if you won't need them to load quantized models.

  pip install -r requirements.txt
  

Configuring the vLLM Server

To launch the vLLM server with the DeepSeek-Coder-V2-Lite-Instruct model, execute the following command:

python -m vllm.entrypoints.openai.api_server \
    --model deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct \
    --trust-remote-code \
    --dtype auto \
    --api-key token-abc123s \
    --port 18889

This will initialize the model and start the server on port 18889.

Usage

All the codes for our main experiments are in the src folder.

Running the Demo

We've prepared a demo code snippet to showcase MGDebugger's debugging capabilities. You can run the demo by executing the following command after starting the vLLM server:

python demo.py

Running Experiments

Once the vLLM server is up and running, start MGDebugger by executing:

python main.py

Tip: You can modify the MODEL and input_seeds parameters in the config.py file to test different models and input configurations.

Log Management

MGDebugger automatically stores all debugging and error logs in the output_data directory. You can review these logs to gain deeper insights into debugging details and performance analysis.

Performance

The table below highlights the performance of different methods compared to the baseline (No-Debugging) on the HumanEval and MBPP datasets using the DeepSeek-Coder-V2-Lite model.

| Method | HumanEval Acc. (%) | Δ Acc. (%) |