New AI Grantees

We give out $5M in grants each year to reduce catastrophic risks from advanced AI, and are excited to share five new projects we have funded!

Dawn Song & Adam Binksmith

Security Benchmark| Dawn Song | University of California

Dawn Song will build a high-quality, large-scale benchmark with rigorous evaluation criteria, to evaluate frontier AI systems’ ability to find and exploit real world software vulnerabilities. Then, she will expand to broader and more comprehensive cybersecurity risks.

AI Village | Adam Binksmith | AI Digest (Sage)

Agent Village is an interactive online environment where frontier AI agents engage with one another, with viewers via group chat, and with the broader digital world through the use of a virtual computer. This project aims to build a highly accessible, open-ended environment in which viewers can observe AI agents pursue loosely defined goals over long time horizons. The objective is to make the capabilities and limitations of AI systems legible and emotionally salient to a broad public audience. The project will also explore using the platform to reproduce known phenomena from AI safety research (e.g. deception and instrumental subgoals) in an accessible format. If there is sufficient interest, the infrastructure may be opened up to external AI safety researchers for further experimentation and iteration.

Environment-Mediated Pareto-Optimized Multi-Agent Interaction | Richard Mallah | Center for AI Risk Management & Alignment

This project will design, develop, and benchmark a platform for executing AI agents in multi-agent environments. The system will be capable of detecting interactions between agents and automatically arbitrating disagreements and other interdependent decisions across arbitrary domains. The arbitration framework will draw on tools from mechanism design, social choice theory, and multiobjective optimization, leveraging the platform’s built-in semantic services—which serve as the agents’ runtime context—to provide situational awareness and informed resolution. The goal is to reduce the frequency of mutually suboptimal outcomes and overcome barriers to cooperation caused by inaccessible trust structures. By enabling agent creators to place trust in a shared, neutral execution environment rather than in one another, the platform opens up a broader space of win-win and Pareto-efficient outcomes.

Zhonghao He & Tianyi Qiu

Prevent AI-driven value lock-in | Zhonghao He & Tianyi Qiu | University of Cambridge & Peking University

This project addresses the existential risk of premature value lock-in, as defined by Bostrom (2002), where current moral failures are perpetuated and moral progress is foreclosed. To mitigate this risk, the team has collected and published empirical evidence on AI-driven value lock-in (currently under review at ICML), and designed a simple, principled algorithmic intervention to reduce lock-in effects and sycophantic behaviors in large language models. The next phase of the project will involve implementing and testing this solution. More information is available here.