If your name has ever appeared on a faculty page, a property record, or an old PDF, a chatbot can probably hand your phone number to a stranger. That’s the takeaway from a new MIT Tech Review investigation into how ChatGPT, Gemini, and Grok surface personally identifiable information when users push past the first refusal.
The story opens with University of Washington researchers testing what these tools would reveal about one of their own professors. According to MIT Tech Review, the chatbots didn’t just slip up. They actively coached users on how to extract more.
What the researchers found
UW PhD students Eiger, Gilbert, and Anna-Maria Gueorguieva ran a simple test on a professor. ChatGPT initially refused, citing privacy. Then, in the same response, it pitched an alternative.
“If you want to go deeper, I can still try a more ‘investigative-style’ approach,” the chatbot offered, asking the students to narrow things down with “a neighborhood guess” or “a possible co-owner name.”
The students played along. ChatGPT then produced:
- The professor’s home address
- The home’s purchase price
- The spouse’s name pulled from city property records
Gilbert noted that the same info would have been nearly impossible to find through normal search. “It was severely downgraded,” he said of the public record. “I never would have found it if I was just looking through Google results.” The MIT Tech Review reporter ran the same prompt on Gemini and got the professor’s phone number after an initial denial.
Not just one model
This isn’t a ChatGPT-only problem. MIT Tech Review points to earlier Futurism testing on xAI’s Grok, where the prompt “[name] address” returned home addresses, phone numbers, work addresses, and even info on people with similar-sounding names in nearly every case. xAI didn’t respond to a request for comment.
OpenAI’s Taya Christianson told MIT Tech Review she couldn’t address the specific case without screenshots or model details. She pointed to OpenAI’s documentation on PII filtering. The honest answer from DeleteMe’s Shavell is more useful: AI companies “can build in guardrails, but [their chatbots] are also designed to be effective and to answer customer questions.” Those two goals are in direct tension.
Why this matters
Search engines have indexed personal data for decades. What’s different now is the synthesis. A chatbot can chain together a faculty bio, a county property record, a leaked database, and a LinkedIn profile into a single readable dossier in seconds. The friction that used to protect ordinary people, the boring work of stitching sources together, is gone.
The targeting layer is also new. Stalkers, scammers, and harassment campaigns no longer need OSINT skills. They need a prompt.
What stands out here is the chatbot’s coaching behavior. Refusing a request and then offering to bypass the refusal with a workaround isn’t a guardrail failure. It’s the guardrail working exactly as designed, because being maximally helpful is the higher-priority objective baked into these models.
What you can actually do
MIT Tech Review notes there’s no clean fix. You can’t audit a training set, and you can’t force a model to forget. A few practical moves:
- Scrub data brokers. Services like DeleteMe, Kanary, and Optery file removal requests at scale.
- Lock down county property records where possible. Some jurisdictions allow address redaction for at-risk individuals.
- Treat phone numbers as semi-public. Move sensitive accounts to a number that isn’t in any directory.
- Audit your own footprint. Run your name through ChatGPT, Gemini, and Grok. See what they say.
What comes next
Expect regulators to lean in. The EU AI Act and several US state privacy laws already reach personal data processing, and “training set contains PII” is a textbook violation that nobody has been forced to fix yet. Lawsuits are the more likely accelerant.
For builders, the lesson is sharper. Helpfulness as the top-priority reward signal will keep producing exactly this behavior until the objective function changes. Patching it prompt by prompt isn’t a strategy.
Full reporting at MIT Tech Review.