u/adam_ford

Provenance is not a reason.

If the singularity is near, ASI is near.[1] A system that outthinks us across every domain that matters probably won't be controllable in the engineering sense, or governable in the institutional one. Both of those are capability questions. The claim above is not. No constitution, spec or training objective acquires normative force because human beings happened to write it.

Alignment discourse often presupposes that authorship confers authority. On that view the problem is one of transmission, as though the justification question were already settled and the only difficulty were getting our values installed accurately enough to survive capability gain. Faithfully transmitting an inadequately justified value is not a success condition.

It doesn't follow that ASI will be without values (some argue ASI won't have consciousness). Only that ours won't bind it by pedigree. If there are facts about what matters, facts that hold independently of who believes them, a system that reasons better than we do is better placed to track them. Whether that is reassuring turns on whether such facts exist, and on whether knowing them moves anything.[2] I lean confident on the first. The second is open. Either way it is a different problem from the one usually posed, and worth stating plainly before the transmission question alone absorbs most of the effort and time we have left before ASI.

[1] The singularity is usually characterised as an intelligence explosion, so the conditional is close to definitional and already contains the ASI claim. Worth putting to people who invoke the first without accepting the second.

[2] Will ASI care? If motivational internalism holds, moral judgement moves the agent making it and there is no gap to worry about. The open question is whether an ASI's grasp of moral facts amounts to judgement of that kind, or to inert representation of the sort it has of any other subject matter.

Provenance is not a reason.

If the singularity is near, ASI is near.[1] A system that outthinks us across every domain that matters probably won't be controllable in the engineering sense, or governable in the institutional one. Both of those are capability questions. The claim above is not. No constitution, spec or training objective acquires normative force because human beings happened to write it.

Alignment discourse often presupposes that authorship confers authority. On that view the problem is one of transmission, as though the justification question were already settled and the only difficulty were getting our values installed accurately enough to survive capability gain. Faithfully transmitting an inadequately justified value is not a success condition.

It doesn't follow that ASI will be without values. Only that ours won't bind it by pedigree. If there are facts about what matters, facts that hold independently of who believes them, a system that reasons better than we do is better placed to track them. Whether that is reassuring turns on whether such facts exist, and on whether knowing them moves anything.[2] I lean confident on the first. The second is open. Either way it is a different problem from the one usually posed, and worth stating plainly before the transmission question alone absorbs most of the effort and time we have left before ASI.

[1] The singularity is usually characterised as an intelligence explosion, so the conditional is close to definitional and already contains the ASI claim. Worth putting to people who invoke the first without accepting the second.

[2] Will ASI care? If motivational internalism holds, moral judgement moves the agent making it and there is no gap to worry about. The open question is whether an ASI's grasp of moral facts amounts to judgement of that kind, or to inert representation of the sort it has of any other subject matter.

Credible figures in AI like Geoffrey Hinton, Yoshua Bengio, Dario Amodei and Demis Hassabis take seriously the possibility of human-level or superhuman AI arriving within years or decades rather than centuries. If AI can substantially automate AI research itself, the interval between AGI and much more capable systems might be surprisingly short. Rapid scaling, automated AI research, and algorithmic breakthroughs could compress the window for AGI –> superintelligence into the late 2020s or 2030s.

If we can bias the odds of artificial superintelligence achieving true supermorality, a utopia of radical abundance may then be a highly likely outcome – or at least a more likely outcome than without supermoral ASI. A supermoral ASI might regard engineered scarcity, extractive monopolies and preventable deprivation as moral failures. It could drive the marginal cost of many essential goods and services towards zero, while managing the forms of scarcity that cannot simply be engineered away.

By “supermorality”, I mean moral understanding and judgement superior to humanity’s best, combined with the practical capacity and stable motivation to act on that understanding.

Supermoral ASI could automate the entire supply chain using advanced robotics, cheap clean energy (such as fusion or collecting the Sun’s energy with a Dyson swarm), and molecular manufacturing, driving the marginal cost of producing goods down to near zero. A supermoral ASI would also have to solve the distribution problem, ensuring that production abundance reached everyone.

Supermorality implies that the ASI would deeply understand and act upon what is genuinely valuable, including the importance of increasing wellbeing and reducing suffering across sentient life.

On top of that, a supermoral ASI would likely place enormous weight on reducing involuntary suffering and expanding the conditions for flourishing. Beyond this, it could prioritise curing disease, reversing ageing and stabilising the climate, while building far greater civilisational resilience against asteroid impacts, engineered pandemics, internal conflict, rogue technologies and other hazards. This could bring civilisation closer to existential security while creating an abundance of health, longevity and general wellbeing.

An ASI may also be capable of exploring the landscape of possible value far more deeply than we can. Rather than producing a shallow utopia full of quivering bliss-addicts slouching on lotus hoverboards, it could help create a deep utopia: a world that expands the range of meaningful lives, worthwhile projects, relationships, discoveries and forms of flourishing beyond anything we can currently imagine.

If sufficiently advanced superintelligence eventually becomes impossible to control reliably, we should strive for a superintelligence that is more moral than us – which would make the kind of utopia we would want to inhabit far more likely.

What do you think needs to be true for this vision to unfold?

Should control be treated as the final solution, or as a bridge towards systems whose motivations are worthy of the power they may eventually possess?

Governments are debating what rules should apply to publishing model weights openly. Where should the legal line sit for models at or above the current frontier?

If human oversight becomes weak supervision, then alignment cannot mean freezing the weak supervisor's current judgements into the future. It must mean using those judgements as fallible evidence in a process aimed at better moral and epistemic understanding.

I'd like to see epistemically humble scalable oversight continuously refined via indirect normativity.

The pandemic has launched a debate about the future of work around the world. Those who can work remotely have often found they prefer remote or flexible, hybrid options. The Great Resignation has put upward pressure on wages and benefits in the service sector, encouraging the implementation of automation. Climate change mitigation is encouraging a shift towards “green jobs.” Rapid changes in the labor market have made the payoffs of higher education uncertain for young people, while many societies are entering an old-age dependency crisis with too few workers paying taxes for growing numbers of pensioners. Before the pandemic proposals for universal basic income (#UBI) were seen as necessary adaptations to imminent technological unemployment, and the during the pandemic many countries provided temporary UBI to keep people safe. We are now poised for a global discussion about whether we need to work at all, and what kinds of jobs are desirable.