"Model Editing for LLMs4Code: How Far are We?"

This paper presents a systematic study on applying state-of-the-art model editing approaches to repair the inaccuracy of LLMs4Code

https://arxiv.org/abs/2411.06638

🎯 Original Problem:

LLMs4Code often contain outdated or incorrect code knowledge due to limited training data timeframes and noisy training data. Retraining these models is extremely expensive and impractical. Model editing techniques exist but lack comprehensive evaluation for code-specific tasks.

-----

🛠️ Methods applied in this Paper:

→ Created CLMEEval benchmark with CoNaLa-Edit (21K+ code generation samples) and CodeSearchNet-Edit (16K+ code summarization samples)

→ Evaluated six state-of-the-art model editing techniques across three categories: External Memorization, Global Optimization, and Local Modification

→ Tested on three LLMs4Code: CodeLlama (7B), CodeQwen1.5 (7B), and Stable-Code (3B)

→ Developed A-GRACE, an enhanced version of GRACE with added encoder and contrastive learning

-----

🔍 Key Insights:

→ External Memorization technique GRACE achieves best effectiveness and specificity

→ Most techniques perform poorly on LLMs4Code compared to general LLMs

→ All techniques struggle with generalization

→ Performance varies by task type - worse in NL2PL than PL2NL editing