Controllable AI with Open Agency Systems @ Intelligent Cooperation Workshop

Evan Miyazono presented a project on controllable AI and coordination systems, addressing challenges in specifying desired outcomes for AI systems. The focus is on controllability to mitigate risks associated with general intelligence by modifying constraints to bind AI behavior. The proposed Open Agency Architecture involves participants interacting with an LLM to generate outcome specifications, using reinforcement learning for controllable policies. Miyazono reminds us that collaboration among entities and research labs is crucial for building collaborative AI systems, and discusses how we could generate specifications, formal verification, and software patching. Finally, the potential of LLMs generating exploits and loopholes underscores the importance of aligning AI with the legal system.