AI for Science & Safety Nodes RFP:
II. Coordination and accountability
The previous section talked about hardware and how access to compute is important for the future. This section will focus on something less physical. We need ways for increasingly capable AI systems to coordinate well with each other, with the people using them, and with the wider world, without handing all of the power over to a single model, lab, or government.
The three areas below explore 1. how humans and AI agents can collaborate, monitor each other, and cooperate; 2. how humans should retain control and stay empowered as delegation to AI increases; and 3. how we can harness incentive mechanisms and build a stronger independent ecosystem for assessing AI risk.
Application deadline: 31st October 23:59 PDT
2. Supercollaboration and decentralized alignment
The frontier of AI is not happening in a chat window anymore but increasingly looks like many agents working together and acting across systems. The world of single player AI is behind us and this call is for work that takes the many agent world seriously.
The positive version of this is AI could let much larger groups of humans and AIs coordinate, deliberate, and work together than was previously possible. At Foresight, we have been exploring one part of this through our Supercooperation workshops. There is a growing body of work on how AI can help people deliberate, make decisions, and cooperate better. We would like to see more of it.
But systems that make cooperation easier can also make collusion and manipulation easier. So to us, collaboration and multi-party alignment are both important parts of the same coin.
What we’re looking for: On the collaboration side, this means coordination and credit-assignment protocols, tooling for massive human-AI collectives, and incentive designs that stay robust to gaming and manipulation. On the security side, it means aligning highly capable AI without relying on a single central authority: mutual monitoring and cross-checking among agents, scalable verification and proof techniques, and checks and balances that no one party controls (hat tip to the decentralized-AI research map maintained by the Institute for Decentralized AI for pointing out interesting areas of work).
The best proposals will show how the two reinforce each other, since the structures that let agents cooperate are also the natural place to build in the safeguards that keep cooperation beneficial. One promising foundation for such protocols developed in Andrew Critch’s Boundaries sequence is the idea of boundaries as Schelling points for coordination. Rather than aligning many agents by aggregating their preferences, agents can coordinate on which boundaries to mutually respect. From the physical and informational membranes around cells, individuals, communities, and systems, every entity has boundaries. Mutually respecting those boundaries gives every party a protected fallback, making cooperation among different entities with different minds tractable without a central authority. We welcome proposals that formalize, test, or build coordination infrastructure on this foundation.
Why it matters: We want to fund the cooperation and decentralized alignment efforts that let a multipolar world become more capable while staying safe. Concentrating alignment and oversight in one institution or model creates risks of its own. Distributed systems have their own complex failure modes, but worlds in which catastrophic multi-agent risks are prevented while rich cooperation is unlocked are possible. We would like to fund projects working on this.
Example projects and deliverables:
- Tools that use AI to improve collective deliberation, decision-making, or cooperation.
- Distributed oversight, agent monitoring, and verification projects.
- Open coordination platforms and protocols that build on the concept of boundaries as Schelling points for multi-agent coordination
3. Human Empowerment
Where human control belongs is an empirical as well as normative question. Control may be worth keeping in three cases: Where it demonstrably improves outcomes; where it is intrinsically warranted (e.g. on grounds of responsibility, legitimacy, contestability, transparency, self-governance) even at a performance cost; and where relinquishing it is irreversible. In other cases, insisting on control may waste the beneficial gains of AI. The boundaries between these cases can be argued for, mapped, measured, and built into tools, and we want to fund projects aimed at that. The following areas are illustrative rather than exhaustive, and strong proposals that advance our mission in adjacent directions are welcome.
What we’re looking for: Evidence, decision rules, and working tools that identify when human control genuinely empowers people or when AI delegation is preferable. We are especially interested in mechanisms that help allocate, preserve, or adjust the appropriate division of control between humans and AI systems in practice.
Why it matters: Where we concede control that empowers us, we lose the capacity to steer our own future; where we insist on control that adds nothing, we forgo the benefits of AI and risk installing oversight theatre that looks like protection but leaves no one actually in charge. This call funds the work of telling these cases apart, understanding when human rather than AI control genuinely empowers us, and building the tools to keep control where it does.
Example projects and deliverables:
- Evidence-based map and decision rules for when human oversight adds value versus degrades it;
- A “when to defer control” decision framework with a working demonstration;
- A theory and design for which decisions humans should retain, including measurement techniques of the value we forgo by insisting on control
- Mechanisms to detect and fix hollow oversight
4. AI Insurance & Open Governance
The summer of 2026 has been a wake-up call for AI risk. Sometimes an unreleased model hacks into another company during an eval, and no incident report was required of anyone. Other times the Department of Commerce steps in and export-controls two models with little transparency or notice.
Right now there are potent AI risks and a lot of what we learn comes down to what the labs decide to do, what they decide to report, and who they let in to evaluate them. The main alternative we have seen so far is for governments to step in, but that can end up like the Mythos situation where researchers lost access to models overnight with limited transparency. Overall, this is a pretty difficult incentive landscape.
We think AI insurance can work as a type of open governance. It creates an additional source of evaluation about AI risk outside of the labs and government. The hard part is making insurance work for AI, because in the worst cases the harm is much larger than anything an insurer could pay for. But there may still be a lot to price along the way.
AI insurance has a long intellectual history, but we have not seen it really implemented in the real world. There is a lot of groundwork on how liability and insurance could reduce AI risk, from Robin Hanson’s work on foom liability to more recent work by Gabriel Weil. We would like to see some of this move into the world.
What we’re looking for:
We are looking for work that helps build a middle layer between labs doing self governance and governments doing regulation. We imagine this could look like how insurance works in other fields. We’re interested in work that provides verifiable and effective mechanisms for coordination. We are looking for work thoughtfully considering AI insurance, independent technical assessment, and how these systems should interact with the labs, government, and the rest of humanity. We are especially interested in work that can create better incentives for reducing risk around increasingly capable AI.
Why it matters:
The road to catastrophic risk will, if we are lucky, be littered with more near-misses, security failures, and other real-world incidents before any full loss of control.
AI alignment is only one part of the alignment problem. By that we mean, AI is built by people inside organizations, and those organizations have their own incentive structures. Outside of those organizations there are larger industry and social incentive structures. These are the outer layers AI development happens inside. And if we’re going to try to align the AIs, we should at least try to improve the incentive landscape around building them too.
We probably do not have time to invent entirely new incentive systems from scratch. We live on Earth, after all. Insurance is one of the systems we already have for reducing risk when people are doing things that can cause a lot of damage. There are lots of open questions about how to make that work for AI. How do we estimate damages? Who pays? Who makes them pay? We’d like to fund work that tries to figure this out.
Example projects and deliverables:
We would love to see work filling in these gaps.
- Build a way for insurers to judge the risk of a frontier AI lab and decide what coverage to offer and at what price. This could include capability evaluations, security practices, internal model use, deployment conditions, incidents, and near-misses.
- Build ways for people outside a lab to assess risks. This might look like incident reporting and investigation. It might also look like figuring out ways to give assessors the information they need without leaking lab secrets.
- What happens when the potential for damage is larger than anything an insurer could cover. How should insurance mechanisms like contracts, developer mutuals, catastrophe bonds, or liability for near-misses work, and what incentives do they create?
- This report seems like a good start: Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack
FAQ
Proposals are first reviewed in-house for fit and quality. Strong submissions are sent to technical advisors for further evaluation. If your proposal advances, we may follow up with written questions or a short call. If you choose to opt into Lightcone Commons sharing, Foresight may also share selected applications with Lightcone Commons for evaluation, and possible additional funding consideration.
Unfortunately, due to the number of applications we receive, we are unable to provide individual feedback to unsuccessful applicants.
- Alignment with the specific RFP focus area: the degree to which the project addresses the selected focus area as outlined in the RFP
- Impact on reducing existential risks from AI: the extent to which the project can reduce existential risks associated with AI, focusing on achieving significant advancements within short timelines.
- Feasibility within short AGI timelines: the project’s ability to achieve meaningful progress within the anticipated short timeframes for AGI development. We prioritize projects that can demonstrate concrete milestones and deliverables in the next 1-3 years.
- AI-first work: Instead of large team budgets, this means smart allocation of resources and compute to automate workflows.
- Capability to execute: the qualifications, experience, and resources of the applicant(s) to successfully carry out the proposed work. Strong teams with proven expertise in the field will be prioritized.
- High-risk, high-reward potential: the level of risk involved in the project, balanced with the potential for substantial, transformative impact on the future of AI safety. We encourage speculative, high-risk projects with the potential to drive significant change if successful.
- Requirement for open source: We require the work product (code, data, and outputs) of the funding to be open-sourced.
- We fund both short-term and longer projects. Grants are typically paid in one lump sum. However, for larger projects spanning multiple years, payments may be made in tranches, with each subsequent tranche contingent upon the successful completion and reporting of previous milestones.
- We can fund overhead costs up to 10% of direct research costs, where these directly support the funded work.
- Successful applicants must pass our due diligence process, which includes confirming your connections to Foresight Institute, sharing any ongoing criminal proceedings, bankruptcy, tax documents etc., and sharing an itemized budget, project plan and organizational documents.
- By accepting funding, grantees agree that we may list their project on our website and share the project title and project lead on for example social media. If you prefer for your project to remain private, please inform us.
- Grants are subject to basic reporting requirements. Grantees are expected to submit brief progress updates at regular intervals, describing use of funds and progress against agreed milestones.
- Tax obligations vary by country and organization type. Applicants are responsible for understanding and complying with any applicable tax requirements.