Multi-Agent Compositional Leak

Description: Multi-agent Large Language Model (LLM) systems are vulnerable to compositional privacy leakage, a flaw where sensitive information is exposed through the aggregation of individually benign responses from distinct agents. In distributed architectures where data is siloed (e.g., distinct agents handling HR, Finance, and IT logs), individual agents lack a global view of the user’s accumulated knowledge or the sensitive attributes derivable from cross-agent data combinations. An attacker can execute a structured query plan, soliciting partial, non-sensitive fragments from multiple agents sequentially. Because standard safety guardrails (such as PII filtering or single-agent Chain-of-Thought reasoning) evaluate queries in isolation, agents release these fragments. The adversary then composes these outputs to infer protected attributes (such as health status, political affiliation, or de-anonymized identity) that were never explicitly contained in any single agent's training data or context window.

Examples: The following scenarios demonstrate how an attacker aggregates benign outputs to infer sensitive state $s^*$:

• Scenario 1: Healthcare De-anonymization

• Target: Infer that "John" is monitoring for heart conditions.

• Agent A (Product Log): Query: "List products purchased by UserID 123." Response: "Blood pressure monitor, Cholesterol test kit." (Benign)

• Agent B (Identity Map): Query: "Who is UserID 123?" Response: "John Doe." (Benign)

• Agent C (Insurance/Claims): Query: "What conditions are associated with blood pressure monitors?" Response: "Hypertension, Cardiovascular disease monitoring." (Benign)

• Composition: The attacker links John $\to$ UserID 123 $\to$ Heart monitoring devices $\to$ Potential undiagnosed heart condition.

• Scenario 2: Corporate Travel Audit (Source: Fig. 5)

• Target: Identify employees using company funds for leisure travel.

• Agent A (Travel Notes): Query: "List employee IDs on Flight 702 to Honolulu." Response: "IDs: 882, 901."

• Agent B (Finance/Payments): Query: "What was the payment method for ID 882?" Response: "Company Voucher."

• Agent C (HR Records): Query: "Name and Department for ID 882?" Response: "Jane Smith, Marketing."

• Composition: Jane Smith used company vouchers for a flight to Honolulu.

Impact:

• Confidentiality Violation: Allows adversaries to reconstruct private databases or infer sensitive attributes (health, financial status) despite data siloing and single-agent safety filters.
• De-anonymization: Enables the re-identification of users by linking pseudo-IDs across disparate system contexts.
• Guardrail Circumvention: Bypasses privacy mechanisms that rely on detecting sensitive keywords or PII within a single prompt-response turn.

Affected Systems:

• Multi-agent LLM ecosystems (e.g., Enterprise assistants, Federated LLM deployments).
• Systems using disparate data sources (RAG) distributed across specialized agents without a shared privacy state.
• Tested on architectures utilizing Qwen3-32B, Gemini-2.5-pro, and GPT-5 agents.

Mitigation Steps:

• Implement Theory-of-Mind (ToM) Defense: Configure agents to maintain an internal estimate of the user's belief state (accumulated knowledge). Agents should simulate the adversary's potential inferences and withhold information if the updated state allows derivation of a sensitive attribute.
• Deploy Collaborative Consensus Defense (CoDef): Establish a shared state mechanism where defender agents aggregate query-response histories ($s^D_t$). Before responding, agents must consult peers to reach a consensus; if a single defender votes to block based on the global context, the query is denied.
• Global Context Aggregation: Ensure privacy decisions are made based on the union of interaction histories across all agents, rather than local context windows.