"SecEncoder: Logs are All You Need in Security"

A language model that speaks the native language of security logs

SecEncoder, proposed in this paper, proves security logs alone can train better security-focused language models than general text

https://arxiv.org/abs/2411.07528

🎯 Original Problem:

General LLMs trained on natural language struggle with domain-specific security tasks, especially when handling security logs, which have unique patterns and terminology different from natural text.

-----

🛠️ Solution in this Paper:

→ SecEncoder is a specialized small language model pretrained exclusively on security logs (1TB, reduced to 270GB after deduplication).

→ It uses DeBERTa-v2 architecture with disentangled attention and can handle 48,000 tokens as input.

→ The model employs a customized masked language modeling loss focusing on content tokens rather than delimiters.

→ SecEncoder comes in different sizes: base (110M), large (350M), xlarge (700M), and xxlarge (1.1B) parameters.

-----

💡 Key Insights:

→ Security logs alone can be sufficient for training effective security-focused language models

→ Domain-specific pretraining outperforms general language models in security tasks

→ The model shows strong generalization beyond log analysis to tasks like incident prioritization

→ Optimal performance achieved with large/xlarge variants rather than the largest model

-----

📊 Results:

→ Achieved perplexity of 2.16 and accuracy of 0.90 on in-distribution test data

→ Outperformed BERT-large, DeBERTa-v3-large and OpenAI's text-embedding-ada-002 in security tasks

→ Showed 0.46 vs 0.10 improvement in log similarity compared to natural language models