This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.
Privacy Overview
Strictly Necessary Cookies
Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.
If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.
3rd Party Cookies
This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.
Keeping this cookie enabled helps us to improve our website.
Please enable Strictly Necessary Cookies first so that we can save your preferences!
Watch the proposal from one of the working groups at out recent AI safety workshop. This group explored Systemic Risk of AI.
Participants:
Brandon Sayler, University of Pennsylvania
Colleen McKenzie, AI Objectives Institute
Jeremiah Wagstaff, Humaic Labs
Max Reddel, ICFG
Milan Griffes, Lionheart Ventures
Philip Chen, Lionheart VC
Vassil Tashev, Independent
This working group developed a comprehensive approach to identify and address cascading systemic risks associated with AI development. They created a taxonomy of interconnected risks spanning cybersecurity, economics, geopolitics, and social dynamics, proposing the creation of "risk observatories" to monitor early warning signs of potential problems. This approach aims to centralize and automate risk detection systems, allowing quick identification and addressing of issues as they arise. Key risk areas included AI-driven job displacement, erosion of trust due to deepfakes, cybersecurity threats, and potential AI-enabled totalitarianism. Their analysis culminated in a flow chart illustrating how issues could lead to three major endpoints: extinction, excessive state control, or anarchy. By implementing their proposed risk observatory system, the group aims to mitigate these systemic risks and prevent cascading negative outcomes.
Watch here:
foresight.org/summary/group-8-systemic-risk-of-ai-intelligent-cooperation-workshop-2024/ ... See MoreSee Less
Divya Siddarth from the Collective Intelligence Project
presents at our recent AI safety workshop, advocating to empower people to have more control and decision-making power in their interactions with AI, particularly as transformative technologies become more central in society.
See her full talk and summary here: foresight.org/summary/divya-siddarth-collective-intelligence-artificial-intelligence-ic-workshop-... ... See MoreSee Less
Thank you to our Vision Weekend keynote speakers, including Eric Drexler, Juan Benet, Andrew Steele,
and PK Douglas who pointed to paths to progress in Secure AI, Neurotechnology, Molecular Nanotechnology, and Longevity.
Remember to get your early bird tickets to Vision Weekend USA today, foresight.org/vw2024us/\ ... See MoreSee Less
Watch the proposal from one of the working groups at out recent AI safety workshop. This group explored international evaluation standards.
They proposed conducting theoretical research to provide different levels of probability for safety properties based on proof theory and model theory. The team aims to build a solid foundation for multiple AI systems and develop a domain-specific language for specifying and proving safety properties. They plan to use category theory, type theory, and linear logic to monitor resource usage during the proof process. This approach represents a new systematic way to conduct safe AGI research, with the goal of detecting unsafe models early in development or deployment. The team believes that proof is key to understanding AI behavior and establishing trust. Their midterm and final goal is to investigate the safety dynamics of state-of-the-art large language models, though they currently lack experts familiar with both mathematical logic and safe AGI. The risks associated with this approach include the potentially longer learning curve for people to understand and invent new models.
Watch here:
foresight.org/summary/group-6-international-evaluation-standards-intelligent-cooperation-workshop... ... See MoreSee Less
Watch Dean Tribble's talk from our recent AI Safety workshop, exploring of the intersection between smart contracts and AI sheds light on how these technologies are evolving to transform interactions in the digital space. By defining smart contracts as code-based agreements that automatically enforce their terms, Tribble highlights their role in facilitating secure and reliable interactions among strangers. This is particularly significant in blockchain environments, where trust and security are paramount.
Blockchains are important because they can provide a decentralized and transparent platform for these contracts to operate, ensuring that the terms are executed impartially and without interference. This is crucial in settings where participants may not have mutual trust.
See the full summary and talk here:
foresight.org/summary/dean-tribble-smart-contracts-and-ai-intelligent-cooperation-workshop-2024/ ... See MoreSee Less
Watch the proposal from one of the working groups at our recent AI safety workshop. This group explored potential approaches to preventing AI Misuse.
Participants:
Brandon Goldman, Lionheart VC
Dmitrii Usynin, Imperial College London
Josh Tan, MetaGov
Mimee Xu, New York University
Niccolo Zanichelli, University of Parma
Richard Mallah, Center for AI Risk Management & Alignment
This working group proposed a public-private partnership platform to address misaligned incentives in AI development, focusing on data bottlenecks and evaluations. The platform, using secure multi-party computation, would enable privacy-preserving model evaluations, data appraisal, and evaluation of evaluations without exposing model weights. It includes secure evaluation of data on models, secure model evaluation for private benchmarks, and auditable data with confidential computations. The project aims to provide secure oversight through safety probes and real-time evaluations to inform policy, funded by bonds and a data tax. The group emphasized aligning incentives between stakeholders and addressing the current lack of effective oversight mechanisms, particularly for frontier models. Their scalable solution is designed to facilitate collaboration between large businesses and governments in the rapidly evolving AI landscape.
Full video and summary here: foresight.org/summary/group-5-better-incentives-intelligent-cooperation-workshop-2024/ ... See MoreSee Less
Watch Evan Miyazono discuss formal scalable oversight at our recent Intelligent Cooperation workshop.
Evan's approach to reducing AI risk emphasizes user-centric strategies that involve users, engineers, and regulators directly in defining the objectives of AI systems. This model allows stakeholders to focus more on the desired outcomes rather than the intricate processes of AI operation. By incorporating safeguarding tools, the system assists in clearly outlining the desired properties of AI outputs. These specifications then guide proof generators in producing objective evidence that verifies the AI’s solutions align with all the predefined criteria.
Full talk and summary here: foresight.org/summary/evan-miyazono-formal-scalable-oversight-intelligent-cooperation-workshop-2024/ ... See MoreSee Less
Watch the proposal from one of the working groups at out recent AI safety workshop. This group explored potential approaches to preventing AI Misuse.
Participants:
Brian Behlendorf, Mozilla
Fazl Barez, Oxford University
Janna Lu, Mercatus Center
The team proposed working within existing institutional regulatory frameworks, rather than attempting to regulate AI models' capabilities directly. They suggested three strategies: extending the current regulatory regime, open-sourcing guardrails to detect and prevent AI misuse, and using AI for objective evaluations of AI models and their end use. The group emphasized the importance of creating legislation that outlaws specific types of AI misuse, similar to Connecticut's ban on deepfakes. They proposed simulating potential AI misuse scenarios to help regulators anticipate and prevent catastrophic outcomes. The team highlighted that success would involve making it difficult to use AI for creating weapons or other harmful applications. They also recognized the need for a responsive team to quickly address vulnerabilities as they are discovered, suggesting regular checkpoints for such teams.
Watch here: foresight.org/summary/group-4-preventing-ai-misuse-intelligent-cooperation-workshop-2024/ ... See MoreSee Less
Watch Fazl Barez talk about “unlearning” in AI at our recent Intelligent Cooperation workshop. This means removing harmful or unnecessary data from models without retraining them completely. He highlights challenges like the model’s tendency to relearn unwanted behaviors and proposes solutions like pruning neurons to prevent this. Barez’s research underscores the complexity of AI unlearning, revealing its potential and limitations in enhancing model safety and ethical governance.
Discussion and summary here: foresight.org/summary/fazl-barez-unlearning-in-large-language-models-intelligent-cooperation-work... ... See MoreSee Less
Thank you so much to everyone who joined us for Vision Weekend Europe 2024 at the magnificent Bückeburg Palace!
Together we explored "Paths to Progress" across our most critical technologies. From the latest research on longevity, nanotech, neurotech, and AI to strategies for funding and advancing these pivotal areas.
We hope you were inspired by the diverse program, including art installations, piano and violin performances, the Cyber Futures Dinner, engaging unconference sessions, in-depth technology deep dives, and late-night poetry readings.
A HUGE thank you to our partners Coordination Network, SPRIN–D, Ani Biome, Protocol Labs, and Ocean Protocol. Your support made Vision Weekend Europe not just possible, but truly exceptional!
We look forward to seeing you all at Vision Weekend USA in December 2024!
Learn more about Vision Weekend here:
foresight.org/vw2024us/ ... See MoreSee Less
Watch the proposal from one of the working groups at out recent AI safety workshop. This group explored potential modeling and simulation capabilities that can help with AI safety.
foresight.org/summary/group-3-modeling-simulation-capabilities-intelligent-cooperation-workshop-2... ... See MoreSee Less
Watch the opening remarks by Allison Duettmann, CEO of Foresight Institute, at our recent AGI: Cryptography, Security & Multipolar Scenarios Workshop.
foresight.org/summary/introduction-by-allison-duettmann-intelligent-cooperation-workshop-2024/ ... See MoreSee Less
Watch the proposal from one of the working groups at out recent AI safety workshop. This group explored potential approaches to international AI governance and coordination mechanisms to address catastrophic and existential risks from AI.
Participants:
Colleen McKenzie, AI Objectives Institute
Fazl Barez, Oxford University
Janna Lu, Mercatus Center
José Andrade, Genpact
Lisa Thiergart, MIRI
Max Reddel, ICFG
Vassil Tashev, Independent
Yonatan Ben Shimon, MatchBoxDAO
Full talk and summary here: foresight.org/summary/group-1-robust-international-coordination-mechanisms-intelligent-cooperatio... ... See MoreSee Less
Jeffrey Ladish recently addressed critical issues in AI threat models at the AGI: Cryptography, Security & Multipolar Scenarios Workshopy. His presentation focused on the advancing capabilities of deepfake technology and its implications for security—from manipulating elections to generating unauthorized media content.
Ladish also highlighted how Palisade Research is at the forefront of developing countermeasures to mitigate these risks. For a comprehensive overview of these discussions, view the full presentation here: foresight.org/summary/jeffrey-ladish-ai-threat-models-hacking-deception-and-manipulation-ic-works... ... See MoreSee Less
“Embedded Agency Playgrounds” led by Keenan Pepper, funded by Foresight Institute, paves the way for AI systems embedded in their environments—anticipating a safer AI future. Preliminary report upcoming.
See all our AI safety grantees here:
foresight.org/ai-safety/ ... See MoreSee Less