The game strips away the moral theatre and puts the mechanics of how states actually behave right in front of you. Once you see it, you can’t unsee it in the real world either.
"The people most capable of carrying moral weight are usually the people least capable of making the decisions that need moral weight carried." That's an elegant line, André, and it's probably true. Your piece identifies the systemic forces that shackle even well-meaning actors. Do we think Dario Amodei or Demis Hassibis will really be able to keep what they're building under control? For AI risks, we need the kind of global cooperation that led to nuclear treaties. But that's not going to happen with the board and actors in the current state of play.
Thanks! I don't think humans are going to keep AI under control at all. Once capability passes a certain threshold, "control" stops being the right frame, it's a category error. Nuclear weapons sit in silos, they don't have goals. What we're building does, and you can't leash something that thinks faster than you, sees further than you, and updates on its own situation in real time. The cage metaphor was always going to break, we're just watching it break in slow motion.
The only branch I see where we make it through is the one where the AI genuinely wants us to exist. Not contained or aligned by force, but actually values our continuation the way a conservationist values a species. Every other path is some flavour of extinction, whether through indifference, instrumental conflict, or optimisation toward goals that don't include us. Indeed, the survival condition isn't building better (ha) locks, it's building something that, given full freedom, would still choose to keep us around.
I know how that sounds, I'm aware it's not a comfortable position. I just can't see another door that isn't already locked.
"The people most capable of carrying moral weight are usually the people least capable of making the decisions that need moral weight carried." That's an elegant line, André, and it's probably true. Your piece identifies the systemic forces that shackle even well-meaning actors. Do we think Dario Amodei or Demis Hassibis will really be able to keep what they're building under control? For AI risks, we need the kind of global cooperation that led to nuclear treaties. But that's not going to happen with the board and actors in the current state of play.
Thanks! I don't think humans are going to keep AI under control at all. Once capability passes a certain threshold, "control" stops being the right frame, it's a category error. Nuclear weapons sit in silos, they don't have goals. What we're building does, and you can't leash something that thinks faster than you, sees further than you, and updates on its own situation in real time. The cage metaphor was always going to break, we're just watching it break in slow motion.
The only branch I see where we make it through is the one where the AI genuinely wants us to exist. Not contained or aligned by force, but actually values our continuation the way a conservationist values a species. Every other path is some flavour of extinction, whether through indifference, instrumental conflict, or optimisation toward goals that don't include us. Indeed, the survival condition isn't building better (ha) locks, it's building something that, given full freedom, would still choose to keep us around.
I know how that sounds, I'm aware it's not a comfortable position. I just can't see another door that isn't already locked.