Google DeepMind releases sign-language-to-text model, with ASL dictation coming to Pixel 11
A pose-based translation model shows how privacy-preserving inputs and end-to-end learning can replace hand-labelled intermediate steps.
Google DeepMind on 12 August introduced a sign-language-to-text model, which it calls SL2T, and said it now underpins new features for Deaf and hard of hearing users. The company describes it as a massively multilingual system that converts signing into written text, and presents it as a step change in quality and generality for sign language processing.
Two consumer features use the model at launch. Gboard gains sign-to-text dictation, so a user can sign to search the web, write messages, draft documents or talk to Gemini, much as a hearing user dictates by voice. Live Transcribe gains response features built on the same model. The first supported pair is American Sign Language to English, with more languages promised, and the features arrive first on Pixel 11 phones at no extra cost.
Technically, SL2T does not consume raw video. It works from body pose landmarks extracted from the camera feed, which the company says protects user privacy, and translates directly from pose coordinates to text, skipping the intermediate "gloss" annotations that earlier pipelines relied on, which DeepMind says preserves nuance that gloss tends to flatten.
Training used more than 100,000 hours of data across over 50 sign languages, roughly a quarter of it ASL. On the FLEURS-ASL benchmark the model reports a BLEURT score of 70, which the company says exceeds any previously published result. DeepMind says Deaf perspectives shaped the work throughout, including via an AI Sign Language Advisory Committee, and notes that around 70 million people worldwide use more than 200 sign languages.
Why it matters
Accessibility has often been a secondary target for language models, so a first-party launch built for signers is notable. Working from pose landmarks reflects a wider preference for inputs that are cheaper and less sensitive than raw pixels. Skipping gloss for end-to-end translation mirrors the shift spoken-language translation made a decade ago. If the multilingual pretraining transfers, the approach could reach far beyond ASL.
Accessibility has often been a secondary target for language models, so a first-party launch built for signers is notable.
Camera captures the signer
Video frames are processed on the device rather than uploaded as images
Pose landmarks are extracted
Hands, arms, face and body become coordinate sets; raw pixels are not passed on
SL2T model reads the pose sequence
A single model trained on 100,000+ hours across 50+ sign languages
Text is produced directly
No intermediate gloss step; ASL to English at launch, more languages planned
Apps consume the text
Gboard dictation and Live Transcribe responses, first on Pixel 11
What you can learn from this
- Pose landmarks as an intermediate representation. A pose estimator turns each video frame into a small set of coordinates for hands, arms, face and body, often a few hundred numbers per frame. The translation model then sees only that skeleton, never the pixels. This shrinks the input by orders of magnitude, makes the model largely indifferent to lighting, clothing and background, and means the sensitive image data never needs to leave the device or be stored. The trade-off is that anything the landmark detector misses, such as subtle finger occlusion, is lost to every later stage.
- End-to-end translation versus gloss pipelines. Gloss is a written shorthand for signs, such as writing each sign as an upper-case English word in signing order. Older systems first predicted gloss and then translated gloss to sentences, which meant two models, two sets of errors, and a bottleneck that discards facial grammar, spatial reference and timing. Training a single sequence model from pose to text lets the network learn those features directly from paired examples. It behaves better because nothing forces the meaning through a lossy intermediate vocabulary.
- Multilingual transfer learning. Training one model on more than 50 sign languages, rather than one, lets shared structure such as hand shapes, movement primitives and discourse patterns be learned once and reused. Data-rich languages effectively subsidise data-poor ones, which is why spoken-language systems moved to massively multilingual training years ago. The same mechanism explains why a model trained with only a quarter ASL data can still lead an ASL benchmark. The risk is interference, where languages compete for capacity, so architecture and data balance matter.
- Learned metrics such as BLEURT. Classic translation metrics like BLEU count overlapping words with a reference sentence, which penalises valid paraphrases. BLEURT instead uses a trained model to score how close a candidate is in meaning to the reference, so it correlates better with human judgement. A score on FLEURS-ASL is only meaningful against other systems on the same benchmark, which is why the announcement frames it as a relative gain. Learned metrics can inherit their own biases, so serious evaluation still pairs them with human review.
- Participatory governance for assistive AI. A model can score well and still fail its users if it mishandles regional signing, ignores Deaf norms or is deployed in the wrong context. Advisory committees drawn from the affected community shape data collection, evaluation criteria and product decisions before launch rather than after complaints. This works because the people best placed to spot errors are domain experts who are not usually on the engineering team. It also builds the trust needed for users to hand a camera feed to an app.
We teach this
How to use this in practice
- Run a pose estimator on your own webcam. Install MediaPipe in a fresh Python environment (
pip install mediapipe opencv-python) and run the hand and pose landmark demo against your camera. Watch the coordinate stream printed per frame, note how many numbers describe a hand, and then cover the lens partially to see how the detector behaves when fingers are hidden. Done means you can explain, from your own output, why a downstream model that sees only these coordinates never receives an image, and where its blind spots begin. - Draw the two pipelines side by side. On paper or in a diagramming tool, sketch a gloss-based pipeline (video, pose, gloss, text) and an end-to-end pipeline (video, pose, text). Under each arrow, write what information could be lost at that step, such as facial expression, sign duration or use of the space in front of the signer. Then mark which stage would need its own labelled training data. Done means a colleague can look at the diagram and say which design would drop non-manual grammar and why.
- Score a translation two ways. Take five sentence pairs from any public translation dataset, write a deliberate paraphrase of each reference, and score the paraphrases with BLEU using the
sacrebleupackage and with a learned metric such as BLEURT or COMET from Hugging Face. Record both scores in a table alongside your own judgement of whether each paraphrase is acceptable. Done means you have a concrete example where word-overlap scoring punishes a correct answer that a learned metric accepts, plus a note on any case where both metrics disagree with you. - Audit an accessibility feature you already have. Open the accessibility settings on your phone or laptop, enable live captioning or voice dictation, and use it for ten minutes of real work such as replying to messages. Write down three moments where it misheard or lagged, and what context a model would have needed to fix each. Then note whether the feature processed audio on the device or sent it to a server. Done means you have a short, honest list of failure modes and a privacy note that you can compare against the launch claims for sign-to-text.
Sources
- Putting sign language AI into users' hands — Google DeepMind
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report