Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer

Contrastive learning improves category distinction for better text classification.

This paper enhances Transformer for text classification by combining multi-level attention and contrastive learning, boosting accuracy and efficiency. It addresses Transformer's limitations in capturing deep semantic relationships and computational cost.

The proposed model outperforms BiLSTM, CNN, standard Transformer, and BERT on IMDB dataset.

-----

Paper - https://arxiv.org/abs/2501.13467

Original Problem 🤔:

→ Transformers struggle with deep semantic relationships and high computational costs in text classification.

→ Existing improvements haven't fully addressed these challenges for diverse text data.

-----

Key Insights 💡:

→ Multi-level attention effectively captures both global and local semantic information.

→ Contrastive learning improves category distinction by maximizing feature differences.

→ Lightweight modules reduce computational cost without sacrificing performance.

-----

Solution in this Paper :

→ This paper introduces a multi-level attention mechanism combining global and local attention to capture both overall semantics and key details.

→ A contrastive learning strategy with positive and negative sample pairs enhances category distinction.

→ A lightweight module optimizes feature transformation, reducing computational complexity.

-----

Results 👍:

→ Achieves 92.3% accuracy, 92.1% F1-score, and 91.9% recall on IMDB dataset.

→ Outperforms BiLSTM (85.6% accuracy), CNN (86.8% accuracy), standard Transformer (88.5% accuracy), and BERT (90.2% accuracy).