A unified sandbox that tests LLM-generated code in 8 programming languages with comprehensive analysis
π https://arxiv.org/abs/2410.23074
π― Original Problem:
Current sandbox tools for code generation only support single programming languages and lack comprehensive code analysis capabilities, making it hard to improve LLM performance on multi-language code tasks.
-----
π οΈ Solution in this Paper:
β MPLSandbox: A unified multi-language sandbox with three core modules:
β Multi-Programming Language Sandbox Environment: Isolated sub-sandboxes for 8 programming languages (Python, Java, C++, C#, Go, JavaScript, TypeScript, Bash)
β Code Analysis Module: Integrates tools for code smell, bugs, efficiency, unit tests, and basic code structure analysis
β Information Integration Module: Combines compiler feedback and analysis results to enhance LLM performance
-----
π‘ Key Insights:
β First sandbox tool to support multiple programming languages simultaneously
β Provides comprehensive code analysis beyond just compilation
β Can be deployed as standalone or distributed system
β Easy integration with LLM training pipelines
β Extensible architecture for adding new languages and tools
-----
π Results:
β Tested with multiple LLMs including DeepSeek-Coder (6.7B) and Qwen2.5-Coder (7B)
β Best performance with DeepSeek-Coder-V2-Lite (16B):
- Python: Pass@1: 29.6%, Pass@10: 50.2%
- Java: Pass@1: 26.8%, Pass@10: 47.7%
- C++: Pass@1: 25.1%, Pass@10: 44.6%










