Mapping The Jagged Edges Of AI With The Tools Of Cognitive Science

Current artificial intelligence systems demonstrate surprising amount of heterogeneity in their abilities, displaying superhuman competence in some tasks but puzzling limitations in others. I will argue that the tools we need for understanding this heterogeneity can be found in cognitive science, where researchers have spent decades developing theoretical and empirical methods for making sense of the capabilities of intelligent systems. Work by cognitive scientists suggests two strategies for mapping the jagged edges of AI: identifying general properties of neural networks that might translate into limitations for current AI systems, and considering cases where human minds might provide a guide to problems that pose a challenge for AI. I will present examples of both strategies, discussing some surprising cases where large language models perform poorly in predictable ways and recent results using the limits of human cognition to predict cases where large language models and vision language models fail.