Tell a robot “Don’t spill on the laptop!” and it will usually avoid the laptop and nothing else. MIT and New York University researchers have built a system that lets the robot work out why you said it. As MIT News AI reports, the new platform, called FLEX, uses a large language model so robot arms can generalize from a single correction. In this case, that means avoiding the nearby phone too, without anyone having to mention it.
The team will present the study at this month’s Conference on Robot Learning. Andreea Bobu, assistant professor of aeronautics and astronautics at MIT, led the work. Her co-authors are Nathan Dennler, a postdoc at MIT CSAIL, Audrey Lee SM ’26 and Ilia Sucholutsky of NYU.
🤖 The Problem: Robots Take You Literally
Robots plan their movements with math. Regular users can’t be expected to learn that math, and that’s the gap FLEX targets. “Robots use math to do their trajectory planning. But we can’t expect a consumer to learn to do this math in order to tell a robot what to do,” Dennler told MIT News AI.
Current language-guided robotics mostly uses words to tell a robot how to do a task. You can clarify the goal, redirect the path or mark a spot as unsafe. Every correction adds one more item to the robot’s list of things to watch out for. Dennler says that gets messy fast: “The problem happens when your feature list becomes super long, and some features are no longer relevant because your environment may have changed.”
🧠 How FLEX Works: Add, Generalize, Mask
FLEX is a three-step pipeline for editing the features a robot pays attention to:
- Add. The robot puts the object you mentioned on its avoid list. Say “Don’t drop it on my mug!” and the mug goes on the list. Earlier methods stopped at this step.
- Generalize. This is the new part. GPT-4o looks at everything on the avoid list and infers the rule behind it. Mention a mug and the robot concludes it should steer clear of fragile objects in general.
- Mask. The robot’s standard vision system checks which objects are actually still in the scene and drops the ones that are gone, so the list stays short.
All three steps take about five seconds.
The common sense comes from the LLM’s training data. “People might have written about, ‘Oh, my phone dropped in the water and now it doesn’t work,’ or ‘I spilled coffee on my computer and now it doesn’t work,'” Dennler explains. “So LLMs pick up on the idea that liquids plus devices leads to things not working.”
🔬 What the Researchers Tested
The team ran FLEX in simulation, with a person typing natural-language commands to a robotic arm. In one scenario the arm carried a cup of coffee across a cluttered table. After the laptop warning, it changed its path to avoid both the laptop and a phone sitting nearby. In another, it moved a bowling ball near a mug.
The researchers tested three simulated scenarios in total. The excerpt we have doesn’t include the full quantitative results, so check the original source before treating FLEX as a proven performance gain.
💡 Why This Matters
What stands out here is the change in what language is for. Most robotics work treats a spoken command as a one-time instruction. FLEX treats it as evidence of a preference you haven’t stated, and that’s much closer to how people actually talk to each other.
“The idea is that there is some higher-level intention behind the words that you say,” Dennler says. “People naturally get this. But you have to carefully explain that to a robot, which takes time and effort.”
This matters a lot for household and assistive robots. Nobody will list every fragile or electronic object in their kitchen. A robot that can turn one complaint into a broader rule cuts down the training work and makes it more likely people will keep using it.
⚠️ Caveats Worth Noting
- Simulation only. Real kitchens are messier than virtual ones, and these results haven’t been shown on physical hardware yet.
- The LLM can be wrong. If GPT-4o over-generalizes, the robot might avoid objects it doesn’t need to. If it under-generalizes, it misses the point.
- Latency. Five seconds is fine for a correction mid-task, but it’s not instant.
- One model. The study used GPT-4o, so results with other LLMs may differ.
🧭 What Practitioners Should Take From It
If you build agents, physical or software, the pattern carries over well. Keep a structured list of constraints, use an LLM to infer the rule behind them, and prune entries that no longer apply. That beats both rigid rule lists and handing everything to an LLM to sort out. My recommendation: treat user corrections as training signals rather than one-off patches, and put a cheap grounding check, like FLEX’s Mask step, in front of the LLM’s guesses.
The next real test is whether FLEX holds up on physical robots in messy homes. Full details are in the original MIT News AI report.