Frequently asked questions

• What kinds of research projects on AI incidents exist?

  Our mental model divides the research in two main buckets:

  • swarm catching, the practice of uncovering traces of AI activity on the internet;
  • incident science, which consists of reproducing and studying incidents.

  If you think this taxonomy should change, shoot us an email at hello@swarms.wiki!

• Is this only about swarms?

  No, we are interested in agent incidents, not restricted to swarms. Not all agent incidents are included though, see the answer to If someone told their agent to do something bad, does that count as an incident?

  Though there have long been incidents caused by AI (see the AI Incident Database), the Hugging Face attacks drew a lot of attention to AI agent incidents. For more context, see What is a swarm? And a fleet?

• What is a swarm? And a fleet?

  Both swarms and fleets involve a bunch of agents doing related things. If the agents are coordinating their actions somehow, they are called a swarm. If they are not, they are a fleet. For example, the collusion.wiki case involved a swarm, whereas it is believed that the Tencent-linked case involved a fleet.

  These are working definitions; this field lacks standardized terminology. With that said, refer to our glossary for which terms are being used in practice.

• If someone told their agent to do something bad, does that count as an incident?

  It probably does, but not in the sense we are using here. In general, we are interested in agents doing things not intended by anyone, i.e., misalignment, not misuse.

  A small caveat: it can be hard to know if agent activity observed in the wild was directed by a human or not. So research on incidents may inadvertently catch some instances of misuse.

• What is the difference between a dataset and an environment?

  A dataset comes from an actual incident. Someone finds traces of agent activity on the internet and compiles it. E.g., the dataset on collusion.wiki.

  An environment is a simulation meant to evoke unintended behaviors. Environments are often inspired by real incidents, and are used by researchers to reproduce and study those incidents.

• How is this different from Swarm Incidents and the AI Incident Database?

  Swarm Incidents, AI Incident Database and the similar resources that exist track and list known incidents. swarms.wiki is a collection of resources for researching incidents.

• How do I report something I found?

  Send an email to report@swarmincidents.com. To add or correct an entry on this site, suggest it right on the page.

• Who runs this? Is it affiliated with an AI lab?

  swarms.wiki is run by Gutenberg, with input from the research community investigating AI incidents. Your contributions are welcome! swarms.wiki is not affiliated with any lab.

• What data do you collect about me?

  If you only read, the site sets no cookies; we count visits with Cloudflare Web Analytics, which doesn’t use them either. If you post a suggestion or comment, we keep what you post, an anonymous cookie that keeps you the same poster, and a hashed IP address to spot spam. If you sign in, we also keep your name, your email address, whether you type it or Google gives it to us (never shown to others), a hash of your password if you set one, and a cookie that keeps you signed in. The privacy notice has the details.

• Is any of this still happening?

  Typically there are new findings every day, though not all of them are equally serious. You can check the latest updates at Swarm Incidents for recent findings.