广场

Towards a Science of Capable Autonomous Agents & Provable Safety — Part II

Part II: Provable Safety In Part I, I talked about what agents must contain as they become increasingly capable. Here, I talk about having provable guarantees on their safety, in spite of these capabilities. This is in large part a contrast from what AI safety used to be like, where more capability ensured more safety: e.g. if a self-driving car can’t reliably detect objects, it’s obviously unsafe. Instead, what we are talking about is: how do we ensure capable systems remain open to human correction, even when it’s otherwise suboptimal for completing the task? Suppose an agent is editing a document when a notification announces that its computer will shut down. It knows this will prevent it…