Fine-Tune ASR for Your Domain with NVIDIA Speech Agent Skills

Learn how staged adaptation reduced word error rate by 70.8% on an ATC benchmark—and how to apply the workflow to your domain.

Try the interactive demo: Play with the Nemotron ATC Voice Demo to explore ASR for air traffic control.

A speech recognition model can handle everyday conversation and still struggle with a pilot reading back a clearance. Real world scenarios may include radio interference, unfamiliar callsigns, and compressed aviation phraseology that may change both what the model hears and what it should expect.

In our air traffic control (ATC) study, staged adaptation reduced Nemotron 3.5 ASR word error rate (WER) from 77.00% to 22.46% on a two-hour human-annotated ATCO2 benchmark- a 70.8% relative reduction. WER measures word substitutions, deletions, and insertions relative to the reference transcript; lower being better. General-English WER increased slightly: 3.50% before adaptation and 3.63% afterward. This tutorial uses air traffic control as a worked example of domain adaptation. NVIDIA’s speech agent skills guide a coding agent through data preparation, model-specific training with NeMo speech, and evaluation of domain accuracy alongside general-English retention. Researchers define the goals, choose the experiment budgets, and review the results.

From an adaptation goal to an evaluated checkpoint

A skill is a reusable set of instructions for a coding agent. The nemotron-asr-finetune orchestration skill scopes the goal and routes the work. When training or offline evaluation is needed, it invokes the nemo-speech-asr-finetune specialist. They are complementary parts of one workflow, not two independent training systems.

Figure 1. The orchestration skill routes work to specialists and reviews the returned evidence. This study primarily used the NeMo specialist for training and evaluation. Synthetic-text generation and deployment are separate paths; dedicated acoustic profiling is planned.

Three concrete outputs make that workflow understandable:

1. A prepared experiment. The goal, baseline, data splits, and evaluation criteria are explicit. A vocabulary problem might need word boosting or a language model rather than acoustic fine-tuning.
2. A compatible training configuration. The chosen checkpoint determines the architecture, tokenizer, preprocessing, and loss. The configuration records the data mixture, learning rate, training budget, and checkpoint-selection rule.
3. An evaluated checkpoint. A saved model is accompanied by its configuration, predictions, and domain/general-English scores. A promising training curve alone is not the final result.

Figure 2. The orchestration skill defines the experiment and routes execution to the NeMo specialist. Each stage leaves an inspectable output. Researchers review the evidence before approving another iteration.

Install the skills and the training software

There are two places to prepare: the machine running your coding agent and the Linux GPU host that will run NeMo. They can be the same machine. Installing a skill adds instructions for the agent; it does not install the training framework.

Install both skills where your agent runs

With Node.js/npm and Git installed, open a terminal in your project directory and run:

npx skills@latest add NVIDIA/skills \
  --skill nemotron-asr-finetune

npx skills@latest add NVIDIA-NeMo/Speech \
  --skill nemo-speech-asr-finetune

npx skills@latest list

Choose your coding agent and installation scope when prompted. Install the NeMo specialist separately from its owning repository: the orchestration skill's reference to it does not install it. Start a fresh agent session and confirm that both skills are discoverable. See the NVIDIA skills quickstart for agent-specific installation options.

Prepare NeMo on the GPU host

The fine-tuning skill defaults to a NeMo container, which supplies the Python training environment and its dependencies. The host still needs a compatible NVIDIA driver, Docker, and the NVIDIA Container Toolkit configured for Docker. Installing the skills does not perform this host setup.

This example pins the NeMo Speech 26.07.00 container, also listed in the NeMo Speech installation guide. Check its GPU and driver requirements before use. On the GPU host, choose a project directory with sufficient disk space, then run:

nvidia-smi
mkdir -p data experiments

export NEMO_IMAGE=nvcr.io/nvidia/nemo-speech:26.07.00
docker pull "$NEMO_IMAGE"

docker run --rm -it --gpus all --shm-size=8g \
  -v "$PWD/data:/data:ro" \
  -v "$PWD/experiments:/exp" \
  --workdir /exp \
  "$NEMO_IMAGE" bash

Place your licensed audio, manifests, and checkpoint in the host's data directory, or change that mount to their existing location. Audio paths in manifests must resolve inside the container, for example under /data. The read-only mount protects inputs; files written to /exp remain in the host's experiments directory after the container exits. NeMo is already installed in the image, so a second pip installation is not needed for this setup.

Verify the environment before training

Run this check inside the container:

python - <<'PY'
import torch
import nemo.collections.asr as asr
from importlib.metadata import version

print("NeMo:", version("nemo-toolkit"))
print("ASR module:", asr.__file__)
print("PyTorch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("Visible GPUs:", torch.cuda.device_count())
assert torch.cuda.is_available(), "GPU access is required"
PY

Confirm that the imports succeed and the expected GPUs are visible. Ask the agent to record the container digest, software versions, and checkpoint identity with the experiment. Use training scripts compatible with that NeMo version. These commands prepare an environment; they do not launch training or reproduce every historical setting in this study. You still need separately defined training, development, and test manifests and permission to use the data and model.

A useful starting request is:

Use nemotron-asr-finetune to adapt my ASR checkpoint to air traffic control. Inspect the data and environment, measure the baseline on domain and general-English speech, and propose a compatible recipe. Keep test data out of training and checkpoint selection. Save the configuration, predictions, and before-and-after scores.

Here is a list of possible prompts that you may use showing the full capabilities of the orchestration skills:

1. Choose the model and establish a baseline

The intended application matters. We used Nemotron 3.5 ASR Streaming 0.6B, a cache-aware streaming model with RNN-T decoder, when recognition had to proceed as audio arrived. We also tested Parakeet TDT 0.6B v3, an offline model. Both turn audio features into tokens, but use different transducer designs: RNN-T models token emission and blank transitions, while TDT predicts tokens and durations to advance through the encoded audio.

Start from the pretrained checkpoint, not randomly initialized weights. Preserve its expected tokenizer and feature processing unless changing them is itself the experiment. In our Nemotron experiment, switching from the checkpoint’s original audio preprocessing to per-feature normalization without retraining sharply increased recognition errors. A plausible preprocessing option is not automatically compatible with learned weights.

Before adaptation, measure word error rate (WER): substitutions, deletions, and insertions divided by reference words. Lower is better. Use the same evaluation protocol (text normalization / punctuation & capitalization) on the same evaluation audio blend for each within-experiment comparison. Before scoring, we lowercase references and predictions, retain Unicode letters, digits, and straight apostrophes, replace other punctuation and symbols with spaces, and collapse whitespace. The scorer also removes diacritic marks, although none occur in our frozen evaluation reference transcripts. We measured general English on LibriSpeech test-clean. Nemotron evaluation used native streaming with 320 ms chunks, and greedy RNN-T decoding.

2. Prepare data with distinct roles

ATCO2 provides real radio conditions and specialized language. Filtering the approximately 4,281.9-hour delivery for language, confidence, duration, and usable audio/text produced about 314.7 hours of English Silver training data. Silver means machine-generated labels; Gold means human-annotated labels. Neither term tells you whether the audio belongs in training or evaluation.

Hours describe source pools, not the amount sampled during optimization. We kept Gold splits disjoint by recording and audited their identifiers and airport-date groups. Training checks covered readable audio, duration, transcript style, unknown markers, and overlap. Gold development selected checkpoints; the test set is hold out for evaluation, which did not update weights or select checkpoints.

Reserving two hours for evaluation left little trusted training audio. That trade-off makes the result interesting: can a small, accurate correction set improve a model that was previously trained using a much larger, noisier pool?

3. Adapt broadly, then refine carefully

We asked two questions: does English replay limit forgetting during Gold fine-tuning, and does adapting on Silver first improve the result? Figure 3 assigns each checkpoint an ID: N for Nemotron and P for Parakeet. N0 and P0 are the pretrained starting models.

First, we compared two recipes starting from pretrained Nemotron N0. Gold-only fine-tuning (N1) reduced ATC WER to 26.09%, but English WER rose from 3.50% to 5.16%. Adding English replay (N2) kept English WER closer to baseline at 3.72%, with ATC WER at 26.89%.

Figure 3. Checkpoint lineage and standalone greedy WER. Each row names the model, its starting checkpoint, and the data used in that training stage. ATC uses the full two-hour ATCO2 Gold test; English uses LibriSpeech test-clean. IDs are local to this comparison; N and P always stay within their own model family. Parakeet P0 was evaluated on October 13 using the exact pretrained checkpoint that started these Gold experiments. Recipes and seed repetitions remain in the Gold-refinement appendix.

Next, we tested a broader starting point. N0 → N3 adapts on Silver + English, sampled 80/20. N3 → N4 then refines on Gold + English, sampled 85/15. These ratios control how frequently training examples are sampled from each dataset. Pools repeat as needed; the percentages do not represent audio hours or a fixed split within each batch. English replay is included in both stages. The Gold refinement uses a 2,000-step budget and development-set checkpoint selection.

N4 achieved the strongest ATC/English balance among these recipes: 22.46% ATC WER and 3.63% English WER. Compared with direct Gold + English fine-tuning (N2), the Silver-adapted starting point improved both scores. Keeping Silver in the final Gold mixture was less effective in our tested recipes.

Parakeet followed the same two-stage path: P0 → P3 uses Silver + English, then P3 → P4 uses Gold + English. P4 reached 20.00% ATCO2 WER and 2.31% English WER. The data mixtures match Nemotron’s, but the architectures, execution modes, and training details differ.

We plan to release the Gold-refined Nemotron N4 and Parakeet P4 checkpoints on Hugging Face soon.

A separate experiment: adapting to Jacktol

We used Jacktol ATC-ASR to ask a simple question: how well could our models recognize Jacktol test recordings after learning from its training data? The dataset provides about 5.9 hours of training audio, and published models trained on Jacktol gave us a reference for comparing results.

Both models started from their pretrained checkpoints. We first trained on Jacktol + University of West Bohemia (UWB) ATC + English, then continued with Jacktol + English to focus on the target dataset while retaining general-English examples. The checkpoints we retained reached 5.93% Jacktol test WER for Parakeet and 7.31% for Nemotron. These scores use a different test set from the ATCO2 results above.

Figure 4. The Jacktol curriculum, starting from each pretrained model. A1–A2 mix Jacktol, UWB ATC, and English; R1–R3 focus on Jacktol while retaining English replay. Green cells mark the retained checkpoints. Parakeet R3 made only two fewer Jacktol errors than R2, with slightly worse English and UWB scores, so we retained R2. Nemotron R3 is N-J, used in the LM experiment below. These Jacktol scores use a different test set from Figure 3.

We call this Nemotron checkpoint N-J. In the next section, we test an n-gram language model with Nemotron N-J and Gold-refined Parakeet P4. Their acoustic weights stay fixed: only the way they decode speech changes.

4. Add aviation language model without retraining the ASR model

An n-gram language model estimates likely token sequences from text. It can help the decoder prefer familiar aviation phrasing when acoustic evidence is ambiguous. As a standalone statistical language model, building it does not update the ASR model's acoustic weights.

We trained 3-gram and 4-gram LMs from four text recipes: Gold, Gold + English, Jacktol, and Gold + Jacktol. Gold contributed the 393 training transcripts; Jacktol contributed 6,495 usable training transcripts after excluding two with unknown markers. The Gold + Jacktol recipe repeated Gold text to approximately balance word counts. We did not deliberately add development or test references; known cross-dataset recording overlap is handled separately below.

Using each checkpoint's own tokenizer, we built KenLM ARPA models and converted them to GPU-compatible .bin.nemo artifacts for NeMo's NGPU-LM path. NVIDIA's GPU n-gram LM tutorial explains this construction and integration. During decoding, an LM weight controls how strongly token-history scores influence the next choice. We retained greedy decoding, without beam search, for both models; Nemotron remained native streaming.

For each checkpoint, we screened 44 LM settings and a fresh no-LM control on Gold development. We selected one candidate from each text family, then required that English WER worsen by no more than 0.20 percentage points. Figure 5 shows the development-selected Gold + Jacktol candidate for each checkpoint, not a choice made from test scores.

Figure 5. Fresh no-LM controls versus Gold + Jacktol n-gram fusion, with acoustic weights fixed. Parakeet uses Gold-refined P4 from Figure 3. Nemotron uses N-J, the separate Jacktol curriculum endpoint (historical stage P3), rather than Gold-refined N4. The ATCO2 view excludes 213 known Jacktol-linked segments, leaving 1,695 segments (1.749 hours). Compare each checkpoint with its own no-LM control; Figure 3 uses a different ATCO2 test view.

On this recording-disjoint ATCO2 view, the LM reduced Parakeet WER from 19.72% to 19.14% and Nemotron WER from 19.96% to 19.86%. English moved from 2.30% to 2.33% and from 5.30% to 5.33%, respectively. These are modest gains, especially for Nemotron; they do not replace acoustic adaptation.

The Gold-only LM corpus contains just 4,761 normalized words. A useful future experiment would use the orchestration skill's Data Designer path to generate additional aviation training text, check its quality and overlap, and evaluate it as a separately weighted source. We have not run that experiment or measured a synthetic-text benefit. Another next step is to inspect remaining errors, especially callsigns and aviation terms, and test targeted word boosting during decoding. This can favor relevant terms without retraining the acoustic model. We have not evaluated it in this study.

What to take into your own experiment

Fine-tuning is partly an art because several choices interact: label quality, acoustic coverage, sampling, learning rate, and the point at which training stops. The way to make that art teachable is to change a defined recipe and preserve the evidence needed to explain its outcome.

In this study, broad adaptation followed by trusted-label refinement worked better than simply adding more noisy data to the final stage. English replay limited forgetting. An n-gram LM supplied a smaller, separately measurable decoding gain. None of these findings promises the same improvement on another domain.

For your next project, ask the skills to produce a prepared experiment, a compatible configuration, and an evaluated checkpoint. Keep raw predictions and inspect consequential errors, such as callsigns and numbers, alongside aggregate WER. Which problem should your first experiment isolate: unfamiliar vocabulary, a different audio channel, or loss of general-language accuracy?

Evaluation data and acknowledgments

NVIDIA plans to release the two-hour, human-annotated ATCO2 evaluation subset at a later date, covering the two-hour human-annotated ATCO2 evaluation set, not the licensed Silver training corpus or the Gold training/development splits. Jacktol offers a public starting point under its dataset-card terms, but using it creates a different experiment.

We gratefully acknowledge ELDA for the ATCO2 Project Data used in this study. See the ELRA catalogue, reference ELRA-S0484, for the source corpus.

Ready to adapt ASR to your domain?

• Install the skills: Get the Nemotron ASR orchestration skill and the NeMo Speech fine-tuning skill.
• Explore the experiments: Review the training recipes, results, and evaluation details.
• Find data to get started: Use your own licensed recordings or explore the public Jacktol ATC-ASR dataset under its dataset-card terms. Keep training and evaluation splits separate.