A two-way translator between English and American Sign Language.
Signal takes typed or spoken English and shows it as ASL. Later releases will read a signer's hands through the camera and turn them into words. Everything runs on your device where possible; nothing about your speech or video is uploaded when the recognizer is enabled.
How it works
Between the English you type and the sign that plays, there's a real ML pipeline mapping meaning across two very different languages. Hover the cards to see the foil.
- 01
Compose
Type it, or hold the mic and speak. Whatever gets into the composer is a plain English sentence — nothing else happens yet.
- Text or voice
- Web Speech API (browser-native)
- No server call until you hit send
- 02
Gloss
A local LLM (llama3.1:8b via Ollama) rewrites the sentence in ASL word order. Articles drop, wh-words move to the end, proper nouns become fingerspelling.
enWhat is your name?aslNAME YOU WHAT- Topic-comment order
- FS:X for out-of-vocab
- Falls back to rules if Ollama's down
- 03
Sign
Each gloss token indexes a real MSASL keypoint clip. The avatar plays them back with smoothstep transitions so hand paths don't teleport at token boundaries.
- 100-sign MSASL vocabulary
- 32 frames × 273 dims per clip
- 24 fps SVG stick figure
Built with the Deaf community in mind
Every audio cue has a visual equivalent.
Text-to-speech is opt-in. Recording state is shown, not signalled by a beep. Nothing in the app assumes you can hear.
The signer is the star, not the model.
The avatar and sentence are the largest, most centered things on screen. Controls recede.
Recognition runs on your device.
Keypoints are extracted in your browser via MediaPipe. Video never leaves your machine.
Under-trained is spoken plainly.
At 100 vocabulary words we say so. When the model misfires, we surface confidence, not a confident guess.
Credits & data
The sign clips come from the MS-ASL dataset (Microsoft Research). Signal reduces those videos to MediaPipe keypoint sequences, trains a transformer on 100 signer classes, and reaches 0.80 top-1 accuracy on the validation split via a 12-model ensemble with test-time augmentation.
The English ↔ gloss translator is a pluggable module — Anthropic Claude, local Ollama (default: llama3.1:8b), or a pure-Python rules fallback.
Signal is an in-development research project. It is not affiliated with the MSASL dataset owners.
Links
© 2026 Signal · Made as a research prototype · v0.1