This paper explores the practice of AI red teaming, a method for assessing the safety and security of generative AI systems, drawing from the experience of red-teaming over 100 GenAI products at Microsoft.
Red teaming reveals that simpler attacks on Artificial Intelligence often succeed, urging a shift from benchmarks to real-world scenarios.
-----
https://arxiv.org/abs/2501.07238
Methods discussed in this Paper: 💡
→ The paper introduces a threat model ontology that structures the red teaming process by considering the system, actor, Tactics, Techniques, and Procedures, weakness, and impact.
→ It emphasizes a system-level perspective that goes beyond model-level assessments, taking into account the broader application context and potential downstream effects.
→ The paper advocates for a combination of manual and automated testing methods, including the use of the PyRIT framework to achieve broader coverage of the risk landscape.
→ It stresses the importance of understanding both system capabilities and where it is applied to prioritize testing scenarios.
-----
→ Real-world attackers often use simple techniques rather than complex, gradient-based methods to exploit Artificial Intelligence systems.
