A wave of self-represented litigants are walking into courtrooms with cases built by AI, and judges are scrambling to figure out the rules. According to MIT Tech Review, courts across the US are now split on a basic question: when you prep your case with ChatGPT or Claude, are those conversations protected? The answer, so far, depends entirely on which judge you draw.
That uncertainty is the real story here. The legal system is being forced to make calls in real time, with no settled precedent and no clear law to lean on.
The privilege split nobody saw coming
The contradictions are stark. In February, MIT Tech Review reports, a federal court in Michigan ruled that a self-represented person’s ChatGPT conversations counted as work product, meaning they’re shielded from the opposing side. The same day, a federal court in New York ruled the opposite: documents a criminal defendant made with Claude were not privileged, because Claude isn’t an attorney and AI companies can hand user data to third parties.
Then in March, Judge Braswell landed somewhere in the middle. Yes, she acknowledged, systems like ChatGPT, Claude, and Gemini collect user data for training. “But that does not eliminate all expectations of privacy,” she wrote.
Three rulings, three different directions. If you’re using AI to prepare anything legal, assume those chats could surface in court. Privilege is not guaranteed.
Malpractice without a pulse
The second problem is quality. Judges are watching people get confidently wrong advice from chatbots.
Judge Allison Goddard, a federal magistrate in California, described a plaintiff who slipped and fell in a store and demanded $700,000, far more than the case was worth. “Where are you getting the idea that you’re getting $700,000? Did you go to ChatGPT?” she asked. The person mumbled. She then walked them through the actual law. Her verdict on AI legal advice: “It’s like Dr. Google went to law school.”
That raises the liability question. Who pays when the chatbot is wrong? In March, Nippon Life Insurance Company sued OpenAI, alleging ChatGPT practiced law without a license and helped a woman reopen an already-settled lawsuit, flooding the court with frivolous filings. OpenAI moved to dismiss in May, arguing “ChatGPT is not a person and neither has nor uses any degree of legal knowledge or skill.” The case is still pending.
Where the law is heading
Legislators are starting to move, though slowly. New York introduced a bill in March to bar chatbots from impersonating lawyers, even with a disclaimer. In Congress, several bills would ban chatbots from posing as lawyers, doctors, and other licensed professionals. None have gained real traction yet.
Here’s my read on the next year or two. Expect this to land in three waves:
- Near term: the privilege split keeps widening until an appeals court or two forces consistency. Until then, treat AI legal chats as discoverable.
- Medium term: “no impersonating a professional” laws pass somewhere first, likely a state, and others copy the language. Disclaimers won’t be enough to dodge liability.
- Watch closely: the Nippon Life v. OpenAI ruling. If a court accepts that an AI company can be on the hook for bad legal advice, every consumer chatbot’s terms of service gets rewritten overnight.
What to do now
If you build or deploy AI products, the lesson is sharp. Anything that edges toward licensed advice, legal, medical, financial, is becoming a liability magnet. Tighten your disclaimers, log your guardrails, and watch the state bills.
And yet the demand isn’t going anywhere. Judge Braswell noticed that litigants who once mumbled now answer her questions with confidence, having rehearsed with a chatbot. “This is a really tough system to navigate. With AI, though, it gets a little less complex,” she says. People will keep using AI as their lawyer because, for them, the upside still beats the risk. The courts just haven’t caught up. Full details are at MIT Tech Review.