"How to Merge Your Multimodal Models Over Time?"

TIME framework makes model merging work in real-world sequential scenarios.

This paper introduces TIME (Temporal Integration of Model Expertise), a framework for merging multiple expert models sequentially over time instead of all at once.

-----

https://arxiv.org/abs/2412.06712

🤔 Original Problem:

→ Current model merging methods assume all expert models are available simultaneously, but in reality, new tasks emerge progressively over time, requiring continuous model adaptation.

-----

🔧 Solution in this Paper:

→ TIME framework defines temporal model merging across three key axes: Initialization Phase (how to select starting weights), Deployment Phase (how to produce final model), and Merging Technique (how to combine weights).

→ The framework introduces "Best-in-TIME" strategy that uses exponential moving average (EMA) for both initialization and deployment phases.

→ For each new task, the system initializes from previous merged weights, trains on current task, and deploys using EMA-based merging.

-----

💡 Key Insights:

→ Standard offline merging techniques perform poorly in temporal settings

→ Complex merging techniques provide minimal benefits over simple weighted averaging

→ Initialization and deployment strategies matter more than specific merging techniques

→ Temporal merging scales well with larger models and compute budgets

-----

📊 Results:

→ Best-in-TIME outperforms standard sequential fine-tuning by 15% on knowledge retention

→ Scales effectively from 62.3M to 1.37B parameters

→ Matches or exceeds multitask training performance at higher compute budgets