Demo: the NASA Perseverance Mars landing with an AI-generated ASL inset (bottom right). Video footage: NASA/JPL-Caltech.
Why it matters
Captions help many people, but for many Deaf people ASL is the first language, and captions are a second-language experience. Hiring interpreters for every video isn't possible, so almost no video has ASL. The goal here is to make that the default instead of the exception.
How the prototype works
1. Transcript to gloss
The spoken script is converted to ASL gloss, the written form of ASL word order.
2. Generated signer
A video model renders a consistent signer from a single reference photo.
3. Timed to the video
Signing starts when the first word is spoken, with a natural idle pose between lines.
What's next
- Smoother transitions between signs.
- Review and correction of generated signs.
- A larger sign vocabulary.
- Better timing against the spoken audio.
- Expand to multiple signing languages.