New Directions in Cooperative AI

Most models of intelligence begin with individual agents, seeking to achieve objectives, and approach the question of cooperation as a strategic challenge, and opportunity, that is layered on when such individual agents meet up. In this view, the individual is the foundation and cooperation is built on top. In this talk I’ll argue that to build human-like intelligence (and beyond) we need to reverse this picture. Cooperating in groups in valuable for many species but it is foundational for humans: humans wouldn’t exist but for their evolved capacity to cooperate. I’ll argue that cooperative intelligence is the core of human intelligence. And at that core is the cognitive capacity to read and participate in the maintenance of the normative environment—the rules that maintain the stability and coordinated behavior of a group in distinctive patterns. I’ll explore what this insight could mean for the research agenda on cooperative AI.