""Ghost of the past": identifying and resolving privacy leakage from LLM's memory through proactive user interaction"

Ever wondered what ChatGPT remembers about you? Now you can see and change it

MemoAnalyzer, proposed in this paper, reveals and controls private information hidden in LLM's memory systems

A privacy-first approach to managing what LLMs remember about users.

📚 https://arxiv.org/abs/2410.14931

Original Problem 🔍:

LLMs store user interactions indefinitely in their memory systems, including past inputs and retrieval-augmented generation (RAG), creating significant privacy risks. Users remain unaware of this memory mechanism and lack control over their private information.

-----

Solution in this Paper 🛠️:

→ Developed MemoAnalyzer: A browser plugin that visualizes inferred private information from aggregated past inputs/memories

→ Uses background color temperature and transparency to map inference confidence and sensitivity levels

→ Highlights source keywords that contributed to privacy inference

→ Enables one-click modification of sensitive content

→ Implements prompt-based inference method without training on user data

→ Provides hierarchical interface design to balance control and efficiency

-----

Key Insights 💡:

→ Users lack awareness of long-term memory mechanisms (only 5/40 participants understood it)

→ Privacy awareness increases with transparent visualization of inference sources

→ Balancing user control and system efficiency is crucial for privacy management

→ Distinct roles for users and AI enhance agency while maintaining performance

-----

Results 📊:

→ Achieved 22.3% reduction in private information leakage vs GPT baseline

→ Maintained comparable task completion times (460.3s vs 426.2s for GPT)

→ Received significantly higher ratings for privacy protection and user control

→ 96% coverage in identifying privacy-sensitive content by Day 3

→ Reduced cognitive load across all NASA-TLX metrics