Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment

AIPR assessment

Problem difficulty is high: black-box RAG membership inference is a crowded, adversarial, and fast-moving area with many groups trying to optimize stealth, cost, and query efficiency. The paper’s strengths compound well, because the attack is simple, the empirical gains are large, and the cost analysis supports the practical claim that the method is cheaper than prior work. The weaknesses also interact, because the strongest claims are about stealth and defense-agnostic robustness, yet those are

Abstract

Retrieval-augmented generation (RAG) has become central to large language model (LLM) deployments, grounding responses in enterprise or proprietary data to reduce hallucinations. However, this design introduces a new privacy risk: model outputs may signal the presence of specific documents in the retrieval corpus, enabling membership inference attacks (MIAs) that leak sensitive information. Existing MIAs are feasible, but they often rely on easily detected templated queries or require many non-templated yet costly and repetitive queries, limiting practicality. We ask: Can an adversary launch a limited-budget, surrogate-free, stealthy, and defense-agnostic membership inference attack using non-templated queries? We present MEntA (Membership Entailment Attack), a query-efficient MIA that leverages natural-language entailment to maximize information gained per query. By asking low-cost, broad, information-seeking questions and measuring entailment between model responses and candidate documents, MEntA eliminates the need for costly shadow models and large query budgets. Across NFCorpus, SCIDOCS, and TREC-COVID, MEntA achieves up to 0.991 AUC with only 5 queries, outperforming prior methods by up to 0.42 AUC under equivalent conditions. It remains effective under state-of-the-art (SOTA) RAG defenses, while current detectors either miss MEntA or flag benign queries at high rates. Regarding cost, MEntA reduces total attack cost by up to 65$\times$ lower compared to SOTA attacks under the same attack setting. Our findings expose the feasibility of realistic, low-cost privacy leakage in RAG systems and highlight the urgent need for privacy-aware retrieval and defense mechanisms.

Score Breakdown

More from this week

• Optimus: Elastic Decoding for Efficient Diffusion LLM Serving
• TubiFM: Unified Item, Carousel, and Search Ranking for Streaming Discovery
• Context Features Are Cheap: Rank-Aware Decomposition for Efficient Feature Interaction in Recommender Systems
• Rethinking Continual Anomaly Detection on the Edge: Benchmarking Under Realistic Industrial Conditions
• Learning High-Frequency Continuous Action Chunks in Latent Space

More in AI

• GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines
• The CIFAR Synthetic Evidence Corpus for Detecting AI-Generated Evidence
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
• SCOPE: Cost-Efficient Model Selection for Compound AI Systems under Quality Constraints
• Context Features Are Cheap: Rank-Aware Decomposition for Efficient Feature Interaction in Recommender Systems