Self-operating Calculations of Risk for Yielding Accurate Insights will develop an open-source forecasting bot powered by AI-driven research tools to enhance forecasting accuracy and volume. The bot and tools will support human reasoning while providing a foundation for developers working on forecasting automation, fostering innovation and further advancements in the field.

This project aims to automate at least one part of the AI Safety research pipeline, such as generating or refining research ideas or writing research code. It also investigates how AI Safety researchers use AI tools and designs solutions to address any blockers identified.

Currently, imaging activity throughout entire mammalian brains is not possible; if this were possible, then radical advances in understanding how mammalian brains compute, including perhaps human brains, would be possible. To enable this, we will develop highly biocompatible silicon nanoparticles that will drastically reduce light scattering in living mammalian brain tissue, enabling much deeper and higher-resolution optical imaging and neural interfacing.

Lo-fi approaches for uploading: developing a Turing test for cloning of biological organisms using High-precision Multi-modal Behavior Modeling (HMBM)

Funding for a comprehensive report on brain emulation at human scale, aiming to bring the topic into wider discourse. The project will synthesize cutting-edge research and insights from top labs to create a high-quality, accessible resource that highlights the potential of brain emulation technology, its societal implications, and the progress that could be achieved within the next decade. The report will serve as a catalyst for talent and funding in the field, helping to scale interest and engagement in this transformative technology.

Seed funding for a much larger project that involves the “Complete Neuronal Input-Output Functions Project” primarily aims to revolutionize our understanding of brain computation through detailed mapping and modeling of neuronal Input-Output Functions (IOFs). The project’s approach to enhancing AI safety lies in its potential to inform whole brain emulation strategies, which promise to make biological intelligence faster, more interconnected, and safer. By starting with the model organism C. elegans and aiming to control and measure the IOFs of all its neurons, the project plans to build a comprehensive understanding of the causal interactions between neurons. This detailed mapping will provide critical insights into the fundamental computational processes of the brain, which are essential for developing more predictable and controllable AI systems. Moreover, the project proposes a structured collaboration between three key groups focusing on accelerating electrophysiology, advancing molecular data generation, and integrating these with computational models. This collaborative effort is designed not only to push the frontiers of neuroscience but also to create a foundation for safely scaling up AI technologies. By aligning the understanding of neuronal computation with AI development, the project aims to contribute directly to AI safety, ensuring that advances in AI are grounded in a deep, biologically informed understanding of intelligent behavior.

I propose development of a novel method (expansion x-ray microtomography) which could put human brain connectomics within reach by reaching nanoscale resolution while vastly decreasing the imaging speed bottleneck. This could lead to advances in understanding of AI-relevant cognitive processes such as empathy, sociality, motivation, and decision making and thus facilitate design of AI with closer alignment to human interests.

We aim to investigate a neglected prior for AI Alignment called self-other overlap: the model having similar internal representations when it reasons about itself and when it reasons about others. We want to test the hypothesis that inducing self-other overlap is a general, scalable, and transferable method of reducing the risk of deception in RL agents and large language models. We hope that methods that align the behavior of models in distribution such as RLHF, together with methods for honesty like self-other overlap and pressures for coherence, would increase the likelihood that the models are aligned with human values.

I will demonstrate a novel approach for WBE by employing a computational model of neural development. To this end, I will simulate how a mature brain can be reproduced by simulating its development from a single precursor cell.

This project explores the potential of brain-computer interfaces (BCIs) to enhance AI safety, particularly in the context of AGI and/or ASI within a 5-year horizon. It will assess the technical and economic feasibility of BCIs for real-time human oversight, faster intervention, and improved AI alignment. The goal is to produce a comprehensive review and roadmap, identifying promising pathways for further research and development. By leveraging advances in neural interfaces, this work aims to improve human-AI interaction, reducing risks from misaligned AI behavior in high-stakes scenarios.

Professional GenAI Security Training, tailored for securing enterprise LLM services and promoting the safe integration of public GenAI services for in-house operations. This training adopts a Capture The Flag (CTF) style and adversary simulation exercises, covering a spectrum from the fundamentals of LLM security to the application of custom data for developing AI-based security agents. Attendees will be provided with a playground application to try out the labs and CTFs.

We want to automatically evaluate offensive cyber capabilities of large language models (LLMs) in a stateful and realistic manner by leveraging capture-the-flag scenarios. This will tell us both about the level of risk from misuse of LLMs by cybercriminals, as well as about potentially extreme risks from misaligned advanced LLMs which may attempt to evade human control by hacking their own servers.

Running a small, short workshop focused on coordinating about, understanding, and planning to apply «boundaries» as they ultimately relate to safety.

We plan to formalize appropriate threat models for using cryptography to secure AI applications, e.g., for defending against adversarial examples or for model watermarking.  In the process, we will show new attacks on many existing schemes, that were likely overlooked due to a lack of threat modeling.

To support our work with the UK AI Safety Institute (UK AISI). Our technical partnership was announced as part of their second progress report.  

The UK AISI has signed an MOU & NDA with OpenMined to deploy technical infrastructure to facilitate AI safety research across AI labs, AI research organizations, the UK government, and, in the future, governments around the world.

Assess the ability of AI agents to engage in steganographic collusion and AI-AI manipulation within adversarial oversight environments, and develop evaluations to assess how AI models strategize and enact exploitation within negotiation scenarios.

This research aims to explore the potential of safe multipolar AI scenarios, with a focus on multi-agent game simulations, game theory, scenarios avoiding collusion and deception, and addressing principal agent problems in multipolar systems.

Our mission is to conduct research and facilitate discussions on biologically inspired multi-objective multi-agent AI safety benchmarking. This effort aims to contribute to more concrete standardization, informed policy making, and the development of global safety culture in AI applications.

To meet rising demand for our Intelligence Rising AI scenario workshops, we seek support to maintain and improve the web application developed by Modeling Cooperation that simplifies, automates, and improves their facilitation. Designed by researchers from the Universities of Cambridge, Oxford, and Wichita State, these workshops support decision-makers in governments, industry, academia, and other relevant groups in understanding the possible development paths and risks of transformative AI.

An embedded agent is one whose cognitive machinery is a part of the environment in which it’s acting to achieve goals. Current frontier AI models are not embedded, but superintelligent AI will eventually become embedded whether we like it or not, because understanding your place in the world and gaining some form of back-door access to yourself are convergently instrumental goals for many tasks. If this first happens suddenly and unexpectedly in a domain such as “the Internet” or “the physical world” that would be extremely risky. Therefore I propose to study phenomena of embedded agency in safe, mathematically simple sandbox environments. This could lead to deconfusion and experimental verification of theorized embedded agency phenomena, hopefully long before it becomes a concern in capable general-purpose models.

Using the theory of Active Inference to build realistic models of multi-agent bounded-rational systems. We would like to understand how to improve agent’s cooperative capabilities, propensity to cooperate and ability to shape environments without disempowering others.

This research aims to explore the potential of safe multipolar AI scenarios, with a focus on multi-agent game simulations, game theory, scenarios avoiding collusion and deception, and addressing principal agent problems in multipolar systems.

Our mission is to conduct research and facilitate discussions on biologically inspired multi-objective multi-agent AI safety benchmarking. This effort aims to contribute to more concrete standardization, informed policy making, and the development of global safety culture in AI applications.

We are pioneering a multi-agent computational model that can directly simulate multi-polar and game theoretic behaviours in AI scenarios. The developed model will enable the rigorous testing and refinement of scenarios, their underpinning assumptions, and prospective policy proposals, presenting first-of-its-kind computational analysis for multi-polar and AI safety research.