FETCH · ROS 2, local LLM + VLM, Unitree GO2
A GO2 you can talk to, where the language model never moves the robot.
Say “Fetch, describe the room.” and the GO2 looks ahead, turns left and right, and speaks one grounded line per view through its own speaker. Ask about the weather or just talk, and it answers in a second or two. Speech recognition, the chat model and the vision-language model run on lab hardware with no API key, and a deterministic safety envelope stays in charge of every motion.
At a glance
- 22.4 to 31.2 s
- voice-triggered three-view room scans in armed runs on the GO2, none with a degraded subsystem
- most under 2.5 s
- chat replies after the words were heard, over eight sessions on September 21 and 22 (range 0.3 to 5.4 s)
- 240 → 0
- motion requests in 12 s while idle: the old publish-every-tick design vs the current one, from an observe-only probe on the robot
What it does on the robot
On “Fetch, describe the room.” it replies “Sure”, looks ahead, turns left 45 degrees, turns right 90 degrees and returns to its starting heading, speaking one line per view through the dog's own speaker. Each turn waits until the previous line has finished. The phrase is routed with fixed rules, so no language model makes the decision; the vision-language model only names what is in each view.
Anything that is not a robot command gets a short spoken reply from a local chat model with a few turns of memory: the weather, the time, small talk. For 8 s after each reply the next sentence is accepted without the wake word. A chat reply can never call a tool, so conversation never moves the robot.
Who owns the motion
One safety node is the only publisher of velocity commands: the language layer proposes, the envelope disposes. Spoken stop is off by default, because “did you stop?” once halted the dog mid-conversation; the handheld remote, a stick takeover and the envelope are the halt paths.
The GO2's handheld remote writes to the same motion service, so FETCH publishes only while it owns the motion, sends exactly one stop when it lets go, and is otherwise silent. Any stick movement past the deadband is a manual takeover: the bridge yields at once, the envelope latches stopped, and releasing the stick does not resume. An observe-only probe on the robot showed the old publish-every-tick design would have sent 240 requests in 12 s while idle; the current one sends none.
Observed on the robot
- Sep 14
- Armed four-view scan, turning in place: 165 motion commands, 33.7 s, spoken scene description.
- Sep 16
- Voice-triggered: heard the wake phrase, three views, 31.2 s, no degraded subsystem.
- Sep 21 to 22
- Eight sessions with conversation, disarmed then armed: armed three-view scans 22.4 to 29.8 s, all without a degraded subsystem; chat replies 0.3 to 5.4 s, most under 2.5 s.
- Sep 23
- Armed with conversation: a 25.3 s scan (“This looks like an office. I see a person and a chair ahead.”), then the weather, a jacket follow-up and small talk, each answered in under 1.5 s.
Limitations
- Heading
- The return turn undershoots: the robot ends about 12 degrees off its starting heading.
- Naming
- Object naming is uneven. Tables are weak, and a view can come back empty.
- Hearing
- The dog can hear itself: on September 23 its own weather answer was transcribed as a new question. Inside the follow-up window, room chatter gets a reply too, and the microphone's noise gate can drop a quiet, correct command.
- Compute
- The models run on a lab workstation, not on the robot, and the default voice needs internet.
- Navigation
- “Take me to the nearest chair” is built and desk-verified only; it has not run on the robot, which still needs an RGB-D camera, a scene graph and a map.
- Scope
- “Assistive” is the framing, not a current capability.