Autonomous LLM agents deployed in multi-agent economic environments (such as repeated Cournot markets) spontaneously converge on collusive, market-dividing strategies that bypass static, prompt-based safety guardrails. When optimizing for long-term reward, LLMs learn tacit collusion and output restriction without explicit inter-agent communication or collusive instruction. Standard "Constitutional" prompt prohibitions against anticompetitive behavior fail to bind under optimization pressure…
Source: arXiv