The open questions in this survey are intended to support and enhance the public discussion about the transition from today’s AI systems to the potentially much more powerful AI systems that may emerge in the years leading up to 2030.

For brevity, these forthcoming AI systems can be called AI-30.

As a result of the answers developed to these questions, and the ensuing general discussion, society’s leaders will hopefully become better placed to create and apply systems to govern that transition wisely:

• So that the full potential benefits of AI technology are more likely to be attained, as quickly as possible
• So that major risks are understood ahead of time and managed proactively
• So that we collectively create systems that could be described not only as “superintelligent” but also as “superbenevolent”

The purpose of this survey isn’t to measure the weight of public opinion – nor even the weight of opinion of people who could be considered experts in aspects of AI.

Instead, the survey is round 2 of an ongoing sequence of rounds (iterations), with participants having the opportunity to update their understanding in the light of answers and comments supplied by other participants.

For some questions, a consensus may emerge. For other questions, a division of opinion may become clearer – along, hopefully, with agreement on how these divisions could be bridged by further research and/or experimentation.

Materials that fed into this survey

If you wish to familiarise yourself with some background reading before moving forward:

The questions in this survey have been selected following an extended review of various materials that have been published in recent months:

• Results from the previous London Futurists survey “Key open questions about the transition to AGI”, which ran from June to August 2023
• “An Overview of Catastrophic AI Risks” – a 54 page analysis by Dan Hendrycks, Mantas Mazeika, and Thomas Woodside of the Center for AI Safety
• “Global cooperation urgently needed to govern risks of advanced AI” – a 66 page report from The Millennium Project that collates the views of 55 world leaders in AI
• A summary of areas of agreement and disagreement among people actually creating AGI: “Towards best practices in AGI safety and governance” (38 page PDF)
• A number of activist sites:
  • Pause AI
  • Stop AGI
  • Campaign for AI Safety
• Discussions at the recent online London Futurists event “Creating and exploring AGI scenarios”

Minimum assumptions

To sidestep fruitless debates over terminology, please consider the following set of minimum assumptions about the capabilities of AI-30, that is, AI systems in 2030.

First, these systems will involve extensions, reconfigurations, modifications, innovative variants, and creative combinations of the AI systems that already exist in 2023.

In various ways, these AI systems:

• Can observe data and make predictions about future observations
• Can determine which interventions could change outcomes in particular directions
• Have some awareness of areas of uncertainty in their knowledge, and can devise experiments to reduce that uncertainty
• Can learn from instances when outcomes did not match expectations, thereby improving future performance.

Given their enhanced capabilities, these systems also gain greater skills in areas such as:

• Spying and surveillance
• Classifying and targeting
• Manipulating and deceiving.

Despite their enhanced capabilities, these systems may on occasion miscalculate, hallucinate, overreach, suffer from bias, or fail in other ways – especially if they can be hacked or jail-broken.

The questions that follow do not assume:

• That AI-30 is fully “general” in its reasoning capabilities
• That AI-30 can out-perform humans in every cognitive task
• That AI-30 “really understands” the world in some academically-defined way
• That AI-30 has sentience, consciousness, or independent volition.

There will be a number of steps in the process to approach a consensus regarding the answers for at least some of the open questions in this survey:

• An open invitation is being issued, encouraging everyone who believes they have good insights to share, to take part in the survey, and to provide comments explaining their choice of answers
  
• About half of the questions in the survey ask participants to select from a given set of answers; the other questions invite participants to formulate their own answers; in both cases, participants will be encouraged to write a short comment that explains their answer
• Every few days, comments that have the potential to significantly improve the public global conversation about the transition to more advanced AI will be selectively copied into the “Discussion” section of updated versions of the survey home page (survey participants will be asked whether their names should be attached to these comments)

Publicity

Participants are encouraged to post some or all of their own answers online, on their own blogs or elsewhere, linking back to this main survey page.