Foundation

Mission

Importance, Neglectedness, and Tractability


Scientific progress is often hard to predict. Great research is frequently driven by intrinsic curiosity, and it can be difficult to say in advance what the most important research directions for a field will turn out to be. On the other hand, we think that there are features which make some research directions more promising than others. Following an increasing number of philanthropic organisations (see here, for example), CAIF is guided by the importance, neglectedness, and tractability of potential activities, while maintaining an awareness of the value of curiosity and serendipity for great science.

‍Importance. Is this research area likely to make an important contribution to the cooperative intelligence of advanced AI systems for the benefit of all?

Neglectedness. Is this work likely to be done anyway?

‍Tractability. Does the research area lend itself especially well to making progress?

While tractability is a common dimension along which to assess potential research, our emphasis on the qualities of importance and neglectedness differ somewhat from their standard interpretations in academia. For instance, a result may be viewed as particularly important within the researcher’s own field, but to warrant the highest prioritisation such work should also be likely to help us build more cooperative AI systems, now and in the future. Likewise, in many fields there may be multiple research groups knowingly racing to solve the same problem, but from the perspective of counterfactual impact this may not be the kind of research that’s essential to prioritise, because it’s likely to happen with or without our support.

Differential Progress


In the course of building AI systems to be more cooperatively intelligent, those systems might also gain capabilities that could be used to harm others rather than contribute to improvements in social welfare. The alignment problem is one example of this: as AI systems become more generally intelligent, divergences in their goals and humans’ become more dangerous for humans. In the context of Cooperative AI, the ability to understand other agents can lead to improvements in cooperation, but also in deception and manipulation, and the same abilities that allow one to commit to honouring mutually beneficial agreements could also be used to commit to coercive threats.

With these risks in mind, CAIF is interested in supporting differential progress on cooperative intelligence. That is, we want to support research that leads to significant progress on cooperative capabilities – capabilities that lead to increases in social welfare in a wide range of environments – relative to progress on capabilities that are dual-use (e.g., useful for deception, manipulation, disempowering other agents) and therefore may not robustly improve social welfare. This idea is discussed in further detail in a recent seminar from the New Directions in Cooperative AI series.