As AI systems grow more capable, they also become more critical—and more vulnerable—components of an already insecure modern computing infrastructure. Whether embedded in decision-making, communications, or control systems, advanced AI introduces new attack surfaces and amplifies the consequences of security failures.
Traditional security paradigms, often reactive, piecemeal and human-driven, cannot scale to match the speed, scale, and complexity of AI-supported attacks. We urgently need next-generation security technologies designed to protect and defend civilization, starting with frontier AI systems and our nuclear, energy and other security-critical infrastructure from the ground up.
We seek proposals that use AI and related tools to dramatically improve our ability to secure our digital infrastructure, with a focus on approaches that are high-assurance, privacy-preserving, and resilient to a rapidly changing, emerging threat landscape. We seek to fund work that enables rigorous defense—including AI-automated red-teaming and vulnerability discovery, formal verification of critical infrastructure, and scalable cryptographic technologies that distribute trust, decentralize control, and ensure accountability.
We are especially interested in pragmatic security approaches that work for today’s frontier models and hardware while also laying the groundwork for secure architectures in a world where powerful AI is widely deployed. We welcome ambitious proposals that push the boundaries of formal methods, secure computation, and privacy-preserving coordination, as well as foundational work in areas like theorem proving and backdoor detection.
As AI systems become more autonomous and embedded, their attack surfaces and criticality will only increase. Investing early in scalable, verifiable security technologies is essential to the safe deployment of future AI.
Build tools that use AI to automate red-teaming and detect vulnerabilities
Develop provable guarantees for system behavior and scalable cryptographic infrastructure to support trustworthy AI deployment.
Design infrastructure that distributes trust, increases transparency, and enables secure AI operation in adversarial environments.
We prioritize work that strengthens the foundations of AI security through rigorous verification, privacy preservation, and resilient architecture. Projects should demonstrate a clear path toward security paradigms that can scale with increasingly powerful and autonomous AI systems.
We especially welcome proposals that:
Examples of past projects in this area include:
Independent
ETH Zurich
Mileva Security Labs
University of Oxford
Running a small, short workshop focused on coordinating about, understanding, and planning to apply «boundaries» as they ultimately relate to safety.
We plan to formalize appropriate threat models for using cryptography to secure AI applications, e.g., for defending against adversarial examples or for model watermarking. In the process, we will show new attacks on many existing schemes, that were likely overlooked due to a lack of threat modeling.
Professional GenAI Security Training, tailored for securing enterprise LLM services and promoting the safe integration of public GenAI services for in-house operations. This training adopts a Capture The Flag (CTF) style and adversary simulation exercises, covering a spectrum from the fundamentals of LLM security to the application of custom data for developing AI-based security agents. Attendees will be provided with a playground application to try out the labs and CTFs.
We want to automatically evaluate offensive cyber capabilities of large language models (LLMs) in a stateful and realistic manner by leveraging capture-the-flag scenarios. This will tell us both about the level of risk from misuse of LLMs by cybercriminals, as well as about potentially extreme risks from misaligned advanced LLMs which may attempt to evade human control by hacking their own servers.
To support our work with the UK AI Safety Institute (UK AISI). Our technical partnership was announced as part of their second progress report.
The UK AISI has signed an MOU & NDA with OpenMined to deploy technical infrastructure to facilitate AI safety research across AI labs, AI research organizations, the UK government, and, in the future, governments around the world.
