HaKT, proposed in this paper, efficiently transfers knowledge from multiple source models to expand sensing systems with minimal overhead.
This paper introduces HaKT (Heterogeneity-aware Knowledge Transfer), a framework that efficiently expands deep learning-based sensing systems to new domains while handling limited labeled data and heterogeneity challenges. It selects high-quality source models, fuses their knowledge, and adaptively injects it into target models.
-----
https://arxiv.org/abs/2412.04060
🔍 Original Problem:
Expanding deep learning sensing systems to new domains faces three key challenges: limited labeled data, data heterogeneity across domains, and device heterogeneity constraints. Existing approaches like federated learning and transfer learning fail to address these challenges holistically.
-----
🛠️ Solution in this Paper:
→ HaKT first employs a two-stage model selection protocol to identify high-quality source models efficiently without excessive communication overhead.
→ It uses an attention-based mixer to fuse knowledge from multiple source models by assigning sample-specific weights based on feature similarity.
→ The framework includes a knowledge dictionary to store fused predictions and an adaptive learner to optimize knowledge injection into target models.
→ A low-cost joint training scheme simultaneously updates the mixer, target model, and partially frozen source models to reduce computational overhead.
-----
💡 Key Insights:
→ Knowledge from source models can help mitigate label scarcity in target domains
→ Sample-wise weighting is crucial for resolving knowledge conflicts between source models
→ Selective storage and adaptive injection of fused knowledge improves target model training
-----
📊 Results:
→ Achieves up to 16.5% accuracy improvement over state-of-the-art baselines
→ Reduces communication traffic by up to 39%
→ Demonstrates superior performance across various tasks and modalities
Share this post