Request for Proposals: Multi-Agent Security

Multi-agent security

As advanced AI systems proliferate, they will increasingly operate in environments populated by other AIs and by humans—with complex, often conflicting goals. This multi-agent context raises profound safety challenges and new opportunities for cooperation. Interactions between multiple agents—whether human, artificial, or hybrid—can produce emergent behaviors that are hard to predict, control, or align.

Without careful design, such systems could foster deception, collusion, power concentration, or societal fragmentation. Yet if properly guided, multi-agent systems could instead enable an explosion in technologies for mutually-beneficial cooperation, benefiting individual one-to-one interactions as well as society-wide collaboration on collective action problems. Building safe multi-agent systems is not just about designing good individual agents. It is about shaping the ecosystem of interactions, incentives, and norms that will govern how AIs and humans co-evolve together.

What we want to fund

We seek proposals that explore how to upgrade today’s cooperation infrastructure of norms, laws, rights, and institutions to ensure that humans and AI systems can interact safely and beneficially in multi-agent settings. We are particularly interested in prototypes that address new game-theoretic dynamics and principal-agent problems that will arise in interactions between AI agents and humans, mitigate risks of collusion and deception, and enhance mechanisms for trustworthy communication, negotiation, and coordination. We aim to de-risk the deployment of cooperative AI and create pathways where AI agents strengthen rather than undermine human-aligned cooperation.

Early demonstrations—such as AI systems that assist humans in negotiation or agents that autonomously identify and enforce mutually beneficial deals or detect and punish deception— could lay the foundation for a future where interactions among AIs and humans are predictably safe, transparent, and welfare-improving. We also welcome projects that lift collective intelligence at the group level, using AI to augment the processes through which groups form shared preferences, resolve conflicts, and coordinate action.

Specific work we are interested in

1. AI for preventing collusion and manipulation

Scalable solutions that demonstrably prevent collusion, manipulation, or exploitation in agent-mediated agreements.

• 
  Scalable methods for detecting and mitigating new forms of AI deception and collusion against humans enabled by their distinct ability to communicate, commit and problem-solve.
• Scalable methods for detecting and mitigating indirectly harmful social dynamics like polarization, groupthink, or majority tyranny in AI-mediated group settings.

2. Pareto-preferred coordination agents

Autonomous agents that can identify, negotiate, and enforce mutually beneficial arrangements between humans and other AI systems.

• Mechanisms for credible commitment and agreement enforcement that enable trust in automated coordination.
• Systems that lower transaction costs for beneficial coordination while preserving human autonomy and oversight.

3. AI-enhanced group coordination

AI systems that enhance collective intelligence and enable more effective group coordination around shared preferences.

• AI tools that improve information aggregation, preference elicitation, and consensus-building in group decision processes.
• AI-native institutions that leverage the unique commitment properties of AI systems to build new forms of cooperation and governance among human or AI systems.

​​If you have a proposal which falls within multi-agent security, but does not align with the specific work we have outlined here, you are still welcome to apply. However, please note that such proposals are held to a significantly higher bar. We do not accept proposals that fall outside this area.

Our priorities

Proposals should clearly demonstrate how the work will enhance safety in multi-agent AI environments, with particular attention to preventing harmful emergent dynamics when multiple AI systems interact with each other and humans. We prioritize projects that:

• Emphasize practical implementations that can be tested and refined in real-world multi-agent scenarios, rather than purely theoretical approaches.
• Address power imbalances by designing systems that protect vulnerable participants and prevent unhealthy concentration of influence.
• Incorporate robust safeguards against collusion, deception, and adversarial manipulation between agents.
• Promote open source, transparent interaction protocols that make agent behaviors, incentives, and decision processes legible to humans and other systems.
• Enable beneficial coordination while preserving individual autonomy and diversity of values across participating agents.

Previously funded work

Examples of past projects we have funded:

• Socially Complex Multi-Agent Game Environments – Development of tools and training of agents in dynamic, socially rich multi-agent environments to enable both safety and capabilities research on cooperation, conflict, and emergent behavior.
• Emergent Collusion Detection in Multi-Agent Reinforcement Learning – Creation of methods for detecting collusion and collaboration among large populations of AI agents based on limited action observations, advancing early-warning systems for multi-agent risks.
• Active Inference Models for Multi-Agent Systems – Application of Active Inference theory to build bounded-rational agents with improved cooperative capabilities, reduced disempowerment behaviors, and more robust environment shaping.
• Multipolar and Game-Theoretic Simulation Platform – Construction of a computational model to simulate multipolar and game-theoretic dynamics among AI systems, enabling rigorous testing of cooperation and conflict scenarios at scale.
• Ph.D Research in Multi-Agent Safety at Oxford – Support for early-stage research careers focused on multi-agent system safety, helping to build a pipeline of expertise at leading academic institutions.

How to apply?

Complete the application form linked at the top of this page. Applications are accepted year-round and reviewed quarterly. Submission deadlines are:

• March 31
• June 30
• September 30
• December 31

Proposals are first reviewed in-house for fit and quality. Strong submissions are sent to technical advisors for further evaluation. If your proposal advances, we may follow up with written questions or a short call. We aim to make decisions within eight weeks of each deadline.

Who can apply?

We accept applications from individuals, teams, and organizations. Both non-profit and for-profit organizations are welcome to apply, but for-profits should be prepared to motivate why they need grant funding.

Funding and terms

• We award between $4.5–5.5M in total funding annually. Grants typically range from $10,000 to over $300,000, but we do not set fixed minimums or maximums. Applicants should request the amount they believe is appropriate, supported by a clear budget and scope.

• We fund both short-term and multi-year projects. For longer or higher-budget work, we may disburse funds in tranches, with later payments contingent on progress.

• We can fund overhead costs up to 10% of direct research costs, where these directly support the funded work.

• Grants are subject to basic reporting requirements. Grantees are expected to submit brief progress updates at regular intervals, describing use of funds and progress against agreed milestones.

• Tax obligations vary by country and organization type. Applicants are responsible for understanding and complying with any applicable tax requirements.

For full eligibility criteria, financial details, and documentation requirements, see our Grant Guidelines and Conditions →

Further questions or feedback?

Please contact us at [email protected]