THREAT ASSESSMENT: The U.S. military came within minutes of attacking a Chinese vessel this spring based on intelligence that never existed. A chatbot made it up. Aircraft were already airborne when officials caught the error and called off the strike, TechCrunch AI reports, citing a CNN investigation published Friday.
This isn’t a lab demo gone wrong. This is a live armed operation, aborted at the last minute, that could have put the U.S. and China into direct conflict. What stands out here is how far a fabricated claim traveled before anyone pushed back.
What happened
According to TechCrunch AI, the false intelligence surfaced during the war with Iran. Here’s the chain of events:
- A Special Operations Command analyst asked an AI chatbot to combine open source data with classified signals intelligence.
- The chatbot misread the ship’s cargo manifest and reported the vessel was carrying components for a nuclear weapons program.
- The analyst ran the tool a second time to format those findings into an official-looking summary.
- That summary moved through command channels and became the basis for an armed operation.
- U.S. officials discovered the hallucination only after aircraft had launched. The mission was aborted.
No one questioned the report until the very end. That’s the failure. The hallucination didn’t get caught by a review process. It got caught by luck and timing.
Why this matters
The Pentagon has spent the past two years pushing AI into the “kill chain” (the sequence from detecting a target to acting on it). The pitch is speed: commanders respond faster and stay ahead of China. This incident shows the cost of that speed when human oversight thins out.
Two details deserve attention:
- The AI didn’t just err. It laundered the error. The second query turned a bad output into a polished document. A formatted summary looks like finished intelligence. It reads as authoritative, and people downstream treat it that way.
- Errors move faster than skepticism. Once a report enters command channels, each handoff adds credibility. By the time a senior officer sees it, it carries the weight of everyone who forwarded it, even though nobody verified the core claim.
This is significant because the same pattern shows up in civilian work. A hallucinated citation in a legal brief. A fabricated figure in a financial summary. The military version just has aircraft attached.
Expert take
Jake Steckler, a research scholar at GovAI and former U.S. Army officer, told TechCrunch AI that service members need to understand “the uncertainty inherent to LLMs.” He added: “It’s especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions.”
Steckler doesn’t argue for pulling AI out of the loop. He argues for guardrails. “These tools can be useful in the right contexts and with the right safeguards in place,” he said. “But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption.”
That last point is the sharpest one. Every near-miss like this makes operators trust the tools less. Rushing adoption produces the incidents that kill adoption.
Tactical points for practitioners
You probably don’t run military ops. But if you deploy LLMs anywhere decisions get made, this incident is a checklist:
- Separate synthesis from verification. An LLM can draft a summary. It shouldn’t be the only source confirming the facts inside it.
- Watch the formatting step. Using AI to make output look official is where raw guesses become trusted documents. Flag AI-generated summaries as such.
- Build in a human checkpoint before irreversible action. The higher the stakes, the earlier that checkpoint needs to sit.
- Track provenance. Anyone reading a report should be able to trace which claims came from a model and which came from a verified source.
- Train for uncertainty. Users need to know that confident-sounding output is not the same as correct output.
What comes next
Expect this to feed directly into the debate over how fast the Pentagon integrates AI into targeting. Congress and outside watchdogs now have a concrete case, not a hypothetical. The likely outcome isn’t a pause. It’s tighter rules on which tools can touch classified intelligence and mandatory human sign-off before AI-derived findings drive operations.
The near-miss got caught this time. The question the military has to answer is what catches the next one. Full details are in the TechCrunch AI report.