Nvidia's open Nemotron 3.5 Transcribe combines transcription, translation, and speaker recognition

Key Points

- Nvidia has released Nemotron 3.5 Transcribe, an open speech model that follows text commands and combines transcription, translation, and speaker recognition for up to 40 languages.

- The model has 30 billion parameters, but only 4 billion are active per step, so it runs on a single GPU. It ranks near the top of benchmarks for accuracy and speaker attribution.

- Nvidia is also releasing Granary v2, a dataset with 844,000 hours of audio, along with tools for efficiently adapting the model to specialized fields like medicine.

Nvidia has released an open speech recognition model that takes text instructions, and the company says it leads several benchmarks. The release also includes a large training dataset and tools for adapting the model to specialized fields.

A plain transcript doesn't tell you who said what. It also won't translate anything or spell technical terms correctly. Nemotron 3.5 Transcribe is designed to handle all of that in a single model, and it can answer questions about a recording's content as well. Nvidia says the model transcribes 40 languages, translates 31 of them to or from English, and identifies speakers in 13. It handles recordings up to an hour long. They can include up to eight speakers.

Users control the model with simple instructions. Depending on the prompt, the same meeting recording can become a transcript with speaker labels, an English translation, or a summary of the decisions made. Users can also pass along technical terms like drug names as hints. The model is only supposed to use them if they actually appear in the recording.

Near-top accuracy on a single GPU

The model has 30 billion parameters, but only about 4 billion are active per processing step. As a result, it runs on a single Nvidia GPU. That covers everything from the compact DGX Spark to data center hardware. Nvidia says the model processes 100 hours of audio in under three minutes on an RTX PRO 6000 Blackwell Server Edition at NVFP4 precision.

On Hugging Face's English Open ASR Leaderboard, Transcribe ranks first among open-weight models with an average word error rate of 5.33 percent. In Artificial Analysis' comparison, it scores 2.81 percent. That's close to the best open model, Voxtral Small at 2.77 percent, but Transcribe uses only about a sixth of the active parameters. According to Nvidia, several widely used commercial speech services trail behind.

The lead is biggest in speaker recognition. VoiceArena tested 14 systems on conversations where speakers sometimes talk over each other, and Transcribe took first place. It also dropped the fewest words, missing 8.32 percent compared to a median of about 31 percent. On David AI's benchmark, which measures speaker attribution across 21 languages, it posted the lowest error rate. It finished just ahead of Nvidia's own dedicated model for the task.

An open dataset lets developers train for their own fields

Alongside the model, Nvidia is releasing the Granary v2 dataset. It contains about 844,000 hours of speech in 48 languages, up from roughly 643,000 hours in 25 languages in the previous version. Each recording comes in several transcription styles. One version is verbatim, with filler words like "uh" and spelled-out numbers. Another is cleaned up, with no fillers and digits instead of words. Nvidia is also sharing the tools it used to build the dataset, so developers can run them on their own recordings. The company says the dataset doesn't fully match the model's training data, though.

Nvidia tested targeted fine-tuning on recordings of Indian doctors and medical students. On the full benchmark, a more precise prompt cut the error rate from 30.8 to 23.8 percent. After fine-tuning, the rate fell to 11.8 percent. The training data was mostly synthetic speech. For speakers the model hadn't heard before, Nvidia separately reports a drop from 22.44 to 12.17 percent, a relative improvement of 45.8 percent. That put the adapted model ahead of an open medical speech model, which scored 13.79 percent. Nvidia says the whole process took about 14 GPU hours on a single Nvidia GB300.

The specialization comes at a cost. The error rate rose slightly on general English speech, and the medical model stayed more accurate on brand-name drugs. Nvidia recommends setting aside at least 100 real, verified recordings for testing when building custom versions.

The base model has limits too. Features vary by language, and Nvidia says the audio question-answering feature was trained in English. The model is also built only for recorded audio. Nvidia says users should spot-check names, speaker changes, and answers.

The model and dataset are on Hugging Face. There's also a demo you can try. For fine-tuning, Nvidia offers guides and an extension that lets an AI coding assistant run the training workflow.

Nvidia only recently introduced Nemotron 3 Diarization, a dedicated speaker recognition model, in September. It can tell apart up to eight speakers, even in live conversations, but it needed a separate model like Nvidia's Parakeet for the actual transcription. Nemotron 3.5 Transcribe combines both jobs and adds translation and audio understanding. It doesn't support real-time processing, though. Nvidia had already used the Granary dataset for its multimodal Nemotron 3 Nano Omni.

Competitors are setting different priorities. Mistral's Voxtral family also combines transcription and audio understanding. With the recently announced Voxtral 3, the company is focusing on real-time voice agents.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Subscribe now