Project
Yumi TTS
Say "hey robot" and a physical ABB YuMi moves: a fully local voice pipeline wired to RAPID.
A fully local wake-word voice interface that drives a physical ABB YuMi (IRB 14000). Speech is transcribed on-device, parsed into a deterministic command, confirmed, and dispatched over a socket protocol to RAPID running on the robot. Running on the physical arm, backed by 44 green Python tests.
Demo
Demo video coming soon
Recording of the physical YuMi responding to a spoken command.
What it is
Say "hey robot, ..." at an Ubuntu box and the physical YuMi in the ABB Singapore office does it. The system is speech-to-command only: the robot moves rather than talks back. Two features run over one socket: a coffee order that triggers a preloaded recipe, and a gesture library where a spoken word runs taught two-arm poses.
This project is the voice front end, the socket wire, and the gesture dispatch. It does not touch the robot's proprietary recipe routines, which are treated as out of scope and left untouched.
How it works
A wake-word listener runs a local faster-whisper transcription with a Silero voice-activity filter to kill silence hallucinations. Recognized speech is parsed into a deterministic command, confirmed, then sent over a TCP socket to the controller. The robot is the client and the voice box is the server, so the exact same stack drives a RobotStudio virtual controller and the real robot. Text lines flow both ways: a command number or a gesture name goes out, an acknowledgement and a done signal come back. New gestures late-bind by name, so adding one means teaching two poses and adding a single keymap line, with no RAPID edit.
Status
| Piece | State |
|---|---|
| Wake word and local speech-to-text | Working |
| Physical YuMi responding to a spoken command | Working on the real robot |
| Two-arm move test and a taught gesture | Validated |
| Ubuntu voice box (self-bootstrapping launcher, USB mic auto-detect) | Set up and running |
| Python test suite | 44 green |
Honest scope: additional gestures are wired but still need their poses taught, and a full coffee brew depends on the robot's native gripper. The core claim on this page is exactly what it says: a spoken command moves the physical arm today.
Stack
- Robot: ABB IRB 14000 (YuMi), IRC5 controller, socket messaging over the service port.
- Voice box: Ubuntu, on-device faster-whisper with a Silero VAD, USB mic, running the full Python stack on CPU.
- Software: a Python voice and dispatch stack over a locked text socket protocol, mock-first development, 44 green tests.
- Robot side: RAPID that accepts commands and late-binds gesture names to taught procedures.
Why it matters
This is the human-to-robot application layer on real industrial hardware: wiring human intent through wake-word speech, a deterministic parse, a confirmation step, socket dispatch, and RAPID, with the software discipline of a locked protocol and a green test suite. It is a clean demonstration of the application and integration end of the stack.