Methodology
StupidLLM uses a CVSS-inspired severity scoring system to rate every documented AI agent failure on a 0-10 scale. Each incident is verified against source evidence and categorized by failure mode and root cause.
Severity Scoring (0-10)
Causes irreversible damage: data loss, security breach, production outage.
Significant impact: corrupted codebase, API key exposure, system instability.
Moderate impact: logic errors, wasted resources, incorrect but recoverable output.
Minor issues: cosmetic bugs, harmless hallucinations, non-production impact.
Verification
Every incident includes a source URL (GitHub PR, tweet, blog post, news article) used to verify the claim. Verified incidents are confirmed against their source evidence. Unverified incidents are marked as such and may be updated when sources become available.
Source taxonomy
Every incident cites a source type, and the type informs how much weight the incident carries independent of its verification status. Direct artifacts (a pull request, a benchmark run) are trusted more than secondhand narration, and single first-person reports are weighted lowest until corroborated. Current counts across the 97-incident corpus:
Moderate trust — reviewed by a publication, but relies on the outlet's own sourcing.
Weighted case-by-case against the evidence it provides.
Highest trust — a pull request or commit is the failure itself, not a description of it.
High trust — produced by a standardized, repeatable evaluation rather than a single anecdote.
Highest trust — published by researchers with disclosure obligations and technical detail to verify.
Lower trust on its own — a first-person account without independent corroboration; verified before being treated as confirmed.
Moderate trust — first-person account, weighed on the specificity of evidence given.
Moderate trust — public discussion thread, weighed against corroborating detail.
Sample size and ranking
An agent must have at least 3documented incidents before it appears in any ranked list. Mean severity computed over a single report is not a measurement, and presenting it as one would be misleading. Agents below that threshold are published and browsable, but they are shown in a separate “insufficient data” tier.
Entries such as multiple-agents, multiple-llms, and unknown-agent describe multi-agent or unattributed failures. They are collection buckets, not products, and are never counted toward any vendor.
Known limitations
- Reporting bias. Incidents are collected from public reports. Widely-used agents attract more scrutiny, so incident counts reflect attention as well as reliability and must not be read as a defect rate.
- No exposure denominator. We do not know how many total tasks each agent ran, so these counts cannot be converted into a failure probability.
- Long tail. Most tracked agents have only one or two documented incidents, which is why the ranking threshold exists.
- Severity is a judgment. Scores are assigned by reviewers against the rubric above. They are reproducible in intent, not machine-derived.