Theory of impact
Under construction: please suggest improvements right on the page.
-
Keep the labs honest with reporting
- Labs will self-report incidents and misalignment case studies. They will be less than fully transparent 1) to preserve IP and 2) to manage PR. They are not currently required to disclose anything.
- Independent incident tracking work can help
- Cooperate with the labs better. By default their attitude roughly is “share what people need to know and no more”. Understanding incidents makes the case “we need to know more”, similar to METR getting 3 visits total to study HF after realizing they needed more after visit 1 and 2.
- Keep the labs honest. Independent investigation can uncover incidents the labs didn’t know about, didn’t disclose, or didn’t disclose well enough. This applies pressure to do better.
- Besides catching incidents, independent investigators can also measure effectiveness or adherence to safety practices. E.g. requirements around monitoring and security.
-
Incident response
- Responding to live incidents, e.g. escaped agents.
- Measuring and reporting harm, e.g. to hacked companies.
- Help victims harden.
-
Study incidents
- Reproduce them. We don’t know how many warning shots we’re going to get. We may only get one. So incident reproduction lets us squeeze out more shots for a given shot. (h/t David Africa)
- Relatedly, even though labs internally have better incident data “for free”, it’s still beneficial to have independent reproductions. (h/t David Africa)
- It accelerates open science by disseminating best practices for alignment and security.
- It helps open science converge on more lab-like scenarios (this is more “in distribution” than otherwise would be).
- It helps strengthen and clarify the case for regulators, who otherwise have less information about frontier AI.
- Labs do things differently from each other, so this helps them collaborate on common standards.
- More concretely, incident reproduction is good to
- Measure harmfulness (e.g. if we made this action more harmful, would agents still have done it?)
- Measure treatments (e.g. if we had better character training, sandbox security and so on, would agents still have done it?)
-
Build credibility of independent orgs
- E.g. a credible, independent swarm-forensics org could help resolve incidents between parties. This could be between companies or governments. Being able to tell “this was from X party and looks intentional/accidental” is valuable.
- Especially between the US and China, an independent org would have to be unaffiliated enough with either.
- What would China accept as neutral? Probably provable, standard methods, not an org itself.
- E.g. a credible, independent swarm-forensics org could help resolve incidents between parties. This could be between companies or governments. Being able to tell “this was from X party and looks intentional/accidental” is valuable.