"Enhancing Code Generation for Low-Resource Languages: No Silver Bullet"
Below podcast on this paper is generated with Google's Illuminate.
https://arxiv.org/abs/2501.19085
The performance of LLMs in code generation significantly drops for low-resource programming languages due to limited training data. This paper explores methods to bridge this performance gap.
This paper proposes and compares in-context learning and fine-tuning strategies to enhance LLM code generation for low-resource languages.
-----
📌 Smaller LLMs benefit from fine-tuning due to their limited parameter space. Even small datasets can significantly update relevant weights for low-resource code generation.
📌 In-context learning with translation examples acts as a bridge. It enables LLMs to leverage high-resource language understanding for low-resource code by demonstrating analogous structures.
📌 In-context learning's consistent performance boost, especially translation examples, offers a practical, low-overhead method. It enhances existing LLMs for immediate use in low-resource code scenarios.
----------
Methods Explored in this Paper 🔧:
→ This paper investigates five distinct techniques to improve code generation for low-resource languages, specifically R and Racket, using various LLMs.
→ First, classic fine-tuning is explored, where models are further trained on low-resource language code generation datasets.
→ Second, a pre-training step is added before fine-tuning. This pre-training teaches models to translate code from a high-resource language, Python, to the target low-resource language, aiming to improve the subsequent fine-tuning for code generation.
→ Third, in-context learning with translation examples is examined. Prompts are designed to include pairs of Python functions and their translations to the low-resource language, guiding the model through examples.
→ Fourth, in-context learning using translation rules is tested. Prompts provide explicit rules for translating Python code constructs into the target low-resource language.
→ Fifth, few-shot in-context learning is assessed. Prompts contain examples of function descriptions paired with correct code implementations in the low-resource language.
-----
Key Insights 💡:
→ Smaller LLMs, around 1 billion parameters, benefit most from fine-tuning approaches for low-resource code generation.
→ In contrast, larger LLMs, such as 33 billion parameter models, show greater performance gains from in-context learning methods compared to fine-tuning.
→ Fine-tuning very large LLMs on limited low-resource data can sometimes degrade performance relative to their baseline capabilities.
→ In-context learning with translation examples consistently improves performance across most LLMs and low-resource languages tested.
→ Among in-context learning techniques, providing explicit translation rules generally performs worse than using translation or few-shot examples.
-----
Results 📊:
→ DeepSeek Coder 1B: Fine-tuning increases pass@1 score to 18.4% for Racket, significantly improving over the 7.0% baseline.
→ DeepSeek Coder 33B: In-context learning achieves pass@1 score improvements of +12.5% on R and +8.3% on Racket compared to fine-tuning.
→ GitHub Copilot: Few-shot in-context learning boosts pass@1 score on R by +8.4% over the baseline.
→ In-context learning with translation examples consistently shows performance improvements across various models and languages, ranging from +0.9% to +6.3% pass@1 score increase.