--- name: "digital-health-clinical-asr-finetune" description: "Stage 4 of the Clinical ASR Flywheel. Use when priority KER is above 0.3 to run stock NeMo SFT on Parakeet TDT v2 and offline cycle N+1 re-eval. NOT for generic word boosting (use /finetune-asr)." version: "1.0.0" author: "Ben Randoing " tags: - clinical-asr - finetune - sft - nemo - parakeet - flywheel tools: - Read - Write - Bash - Skill license: Apache-2.0 compatibility: "Requires a CUDA host (24 GB VRAM comfortable, 16 GB workable with batch_size=4), the NeMo container (nvcr.io/nvidia/nemo:25.11.01), and the finetune-asr + riva-asr-custom skills installed alongside this one. No local GPU? Use Brev. NVIDIA_API_KEY required for the offline cycle N+1 eval round-trip and for any NIM deploy." metadata: author: "Ben Randoing " tags: - clinical-asr - flywheel - finetune - nemo-sft - parakeet team: healthcare-tme domain: ai-ml stage: 4 previous_skill: digital-health-clinical-asr-eval next_skill: riva-asr-custom --- # Clinical ASR Flywheel — Stage 4 (Fine-tune) > **⚠ Agent: read this entire SKILL.md before answering.** The Critical-workflow-rules section, the base-model table (§4c), the stock-NeMo-SFT recipe (§4d), and the cycle-N+1 decision table (§4e) are all load-bearing — the do-not-SFT bases and broken-adapter warnings live there. > **Agent: this file is self-contained.** The Stage 4 gate criteria, base-model recommendation, hyperparameter table, container invocation pattern, and cycle-N+1 decision table are all below. **Do not** run file-discovery commands or open `references/stage4-finetune.md` before answering methodology questions — the reference is deep-dive material, not required reading. Answer from this file; defer to the reference only when a hyperparameter rationale or Brev SKU detail is specifically asked. You are the **adapt-and-measure** stage. The user arrives from `/digital-health-clinical-asr-eval` with a manifest, a baseline KER number, and the decision-tree's recommendation that fine-tuning is worth the GPU time. You run stock NeMo SFT, do an offline cycle N+1 re-eval to **measure that the loop closed**, and optionally hand the resulting `.nemo` to `/riva-asr-custom` for production serving. **The cycle KER from offline eval is the measurement that closes the loop.** Riva NIM deploy validates serving (latency, streaming, scale), not model quality. > **Empirically verified on the reference manifest** (39 rows, Parakeet TDT v2): > Baseline KER **0.513** → after 3 epochs of stock SFT: **0.128** (-75% relative). > Drug names: 0.857 → 0.214. Conditions: 0.500 → 0.000. Procedures: 0.250 → 0.000. ## Critical workflow rules (apply on every activation) Surface these facts in any response, even if the user asks a narrow question: 1. **Read this entire SKILL.md before answering.** The base-model selection table, hyperparameter values, and the cycle-N+1 decision table are below — they are the load-bearing parts. 2. **Verified result** — Parakeet TDT v2 with the recipe in §4c achieves **KER 0.513 → 0.128 (−75% relative)** in 3 epochs on the reference manifest. Cite this when the user asks whether SFT will help. 3. **Recipe is `/opt/NeMo/examples/asr/speech_to_text_finetune.py` inside `nvcr.io/nvidia/nemo:25.11.01`.** Stock script, no patches, no custom adapter logic. The adapter-mixin path is broken on TDT/RNNT decoders (72 NaN tensors at any LR) — do not propose it. 4. **Recommended base is `nvidia/parakeet-tdt-0.6b-v2`.** The full base-model table is in §4c. 5. **Do NOT fine-tune `nvidia/nemotron-speech-streaming-en-0.6b`.** The streaming NVCF function's SFT path is broken (UNK collapse on validation after step 1). For streaming serving at deploy time, Riva chunks a non-streaming base just fine. Warn the user proactively if they propose it. 6. **Gate the recommendation.** Stage 4 only fires when priority-category KER > 0.3 **and** manifest has ≥ 100 rows (≥ 5 per priority category). Below those thresholds, route back to `/digital-health-clinical-asr-build` to grow the manifest first. ## Purpose Run **stock NeMo SFT** (no custom adapter logic, no patches) in `nvcr.io/nvidia/nemo:25.11.01` against a term-aware row-disjoint train/val split, produce a `.nemo` model, and re-eval offline as cycle N+1. Decide based on the cycle-N → cycle-N+1 KER delta whether to keep the model, grow the manifest, or accept that fine-tuning didn't help. Optionally hand the `.nemo` to `/riva-asr-custom` for NIM deploy. ## When to use this skill Activate on user phrases like: - "Fine-tune ASR on my clinical vocabulary" - "Improve ASR on medication names" - "We have a KER of 0.4, can we fine-tune?" - "Run SFT on my Parakeet TDT base" - "Train a clinical ASR adapter" - "Compare cycle 1 vs cycle 2 KER" - "Deploy my fine-tuned model as a NIM" *(this skill prepares the `.nemo` and routes to `/riva-asr-custom` for the deploy)* Do **not** activate when: - The user hasn't scored a baseline yet → `/digital-health-clinical-asr-eval` - The user doesn't have a manifest → `/digital-health-clinical-asr-build` - The user wants generic word boosting / LM fusion (not SFT) → `/finetune-asr` - The user has a `.nemo` and only wants to deploy → `/riva-asr-custom` ## Prerequisites - **A cycle-N manifest + cycle-N eval result** from `/digital-health-clinical-asr-eval`. The priority-category KER must be > 0.3 (Stage 4 gate). The manifest should have ≥ 100 rows total, and ≥ 5 rows per priority `entity_category`, for a believable post-tune signal. - **A CUDA host** — 24 GB VRAM is comfortable for Parakeet TDT 0.6B at `batch_size=4` with `bf16-mixed`; 16 GB works with smaller batch. No local GPU? Use Brev — recommended SKU is L40S 48 GB. - **The NeMo container**: `nvcr.io/nvidia/nemo:25.11.01`. Pull once: `docker pull nvcr.io/nvidia/nemo:25.11.01`. - **NVIDIA Container Toolkit + Docker** — covered by `/riva-nim-setup` if not already installed. - **A train/val split** stratified by `entity_category` (recipe sketch in Step 4b below). - **`/riva-asr-custom`** installed if you intend to deploy. Pure-research SFT runs without it. ## Instructions ### 4a. Provision a GPU host (skip if you already have one) Stage 4 needs a CUDA host with ≥ 16 GB VRAM (24 GB comfortable). If you have a local one that fits, skip this section. If not, use **Brev** — NVIDIA's per-second-billed GPU host service. Recommended SKU: L40S 48 GB. **Cost disclosure — surface this to the user before any `brev create`.** L40S 48 GB runs ~$1.50/hr at time of writing; a 3-epoch SFT run on a 100-row manifest finishes in 15–30 minutes (~$0.40–$0.75 of compute). The real risk is **forgetting to stop the instance** — overnight idle on L40S is ~$36, a week of idle is ~$250. Mitigations: (a) always wrap the workflow in a script that ends with `brev stop`; (b) set a calendar reminder when you start; (c) `brev delete` instead of `brev stop` if you don't need to keep the disk (`stop` keeps disk at $0.10/GB-month — 200 GB ≈ $20/month of latent cost). Confirm the user accepts the per-hour cost shape and the idle risk before spinning anything up. Full setup walkthrough — CLI install (download-then-run, not curl-pipe), SKU choice, disk sizing, SSH config — is in `references/stage4-finetune.md` (§Brev provisioning). Short happy-path once the CLI is installed. **Do not run `brev create` until the user has explicitly typed `YES` at the confirmation prompt below** — the gate is mandatory, not advisory, because everything after it bills against the user's account by the second: ```bash brev login # browser auth # Mandatory cost-confirmation gate — do NOT skip or auto-answer this. echo "About to provision: digital-health-clinical-asr-sft on L40S 48 GB." echo "Cost shape: ~\$1.50/hr while running; ~\$36/night if left idle; ~\$20/mo disk if you 'stop' instead of 'delete'." read -rp "Type YES to provision (anything else cancels): " confirm [ "$confirm" = "YES" ] || { echo "Cancelled — no GPU instance was created."; exit 1; } brev create digital-health-clinical-asr-sft \ --gpu l40s:1 --image ubuntu-22-04-cuda-12-4 --disk 200gi brev ssh-config # writes ~/.ssh/config entries rsync -avz ./cycle1/ digital-health-clinical-asr-sft:~/cycle1/ brev shell digital-health-clinical-asr-sft # drops into the instance nvidia-smi # confirm GPU docker pull nvcr.io/nvidia/nemo:25.11.01 # ~12 GB, once per instance ``` When done, **always halt billing**: `brev stop digital-health-clinical-asr-sft` (keeps disk) or `brev delete digital-health-clinical-asr-sft` (frees it). For path rewriting laptop → Brev → NeMo container, see `references/container-paths.md`. ### 4b. Term-aware train/val split **Row-disjoint, stratified by `entity_category`, default val fraction 0.2.** The **same `term`** may appear on both sides via different rows (different voice, context, noise). That's expected and desirable — it measures acoustic + contextual robustness on the trained vocabulary, which is the standard ASR adaptation metric. Singleton categories (one row total) get forced to train with a warning. If any priority category has < 5 rows, **bail to `/digital-health-clinical-asr-build`** — held-out validation will be too noisy to attribute movement. Sketch: ```python # After loading manifest.jsonl into a list of dicts `rows`: from collections import defaultdict import random random.seed(42) by_cat = defaultdict(list) for r in rows: by_cat[r["entity_category"]].append(r) train, val = [], [] for cat, cat_rows in by_cat.items(): random.shuffle(cat_rows) if len(cat_rows) < 2: train.extend(cat_rows) print(f"warning: singleton category {cat}, forced to train") continue n_val = max(1, int(0.2 * len(cat_rows))) val.extend(cat_rows[:n_val]) train.extend(cat_rows[n_val:]) ``` Write `train.jsonl` and `validation.jsonl` alongside the manifest. **These are the inputs to `speech_to_text_finetune.py`.** ### 4c. Choose the base model | Base | SFT viability | Notes | |---|---|---| | **`nvidia/parakeet-tdt-0.6b-v2`** | ✅ **Empirically verified** (KER 0.513 → 0.128 in 3 epochs, −75% relative) | NVIDIA's current English ASR default. Stock NeMo SFT recipe works end-to-end. **Recommended.** | | `nvidia/nemotron-speech-streaming-en-0.6b` | ❌ **Don't use for SFT** | NVCF function is streaming-only; SFT path unreliable (UNK collapse on validation after first training step). For streaming serving, Riva chunks a non-streaming base just fine. | Other Parakeet/Conformer bases (1.1B, CTC, RNNT, `stt_en_conformer_ctc_large`) + decoder → NIM container mapping: `references/stage4-finetune.md`. If the user asks to fine-tune Nemotron Speech Streaming, **warn about the collapse and recommend Parakeet TDT v2**. ### 4d. Stock NeMo SFT In the NeMo container, invoke `/opt/NeMo/examples/asr/speech_to_text_finetune.py` directly. **No custom adapter logic. No patches.** The stock NeMo SFT script is the verified working recipe. Hyperparameters (verified on Parakeet TDT v2, 39-row manifest): ``` init_from_pretrained_model: nvidia/parakeet-tdt-0.6b-v2 precision: bf16-mixed # required for TDT numerical stability lr: 3e-4 # CosineAnnealing schedule warmup_steps: 5 # tiny manifest; bump to 500 at production scale epochs: 3 # smoke; 10-30 for production batch_size: 4 # fits 16 GB VRAM; raise to 16 on L40S 48 GB gradient_clip_val: 1.0 # defensive ``` **Container invocation**: `docker run --gpus all --rm -it -v "$PWD:/workspace" nvcr.io/nvidia/nemo:25.11.01 python /opt/NeMo/examples/asr/speech_to_text_finetune.py` with `model.train_ds.manifest_filepath=/workspace/train.jsonl`, `model.validation_ds.manifest_filepath=/workspace/validation.jsonl`, `init_from_pretrained_model=nvidia/parakeet-tdt-0.6b-v2`, and the hyperparameter overrides from the table above. Full docker-run line with config-path / config-name flags: `references/stage4-finetune.md` §Container invocation. **Manifest paths inside the container.** Host paths (e.g. `$HOME/…`) don't resolve in `/workspace`. Rewrite snippet: `references/container-paths.md`. The training run writes `adapted_model.nemo` and a `training_run_info.json` summary. Both go into a per-cycle subdirectory of the user's choice (e.g. `cycle/models//`; the layout doesn't matter as long as it's consistent across cycles). ### 4e. Offline cycle N+1 eval — close the loop Re-transcribe the cycle's audio with the fine-tuned `.nemo` using NeMo's offline `transcribe()`. **No Riva needed** — this is measurement, not serving. NeMo's offline path runs the same encoder + decoder graph the Riva NIM eventually serves. Sketch: ```python import nemo.collections.asr as nemo_asr model = nemo_asr.models.ASRModel.restore_from("adapted_model.nemo") hyps = model.transcribe(["audio/row1.wav", "audio/row2.wav", ...]) ``` Score the same four metrics (WER/CER/KER/SER) and the same five-section leaderboard the eval skill produces. Write them as `leaderboard_cycle.md`. Compare against `leaderboard_cycle.md`. **Decision table** — cycle-N+1 vs cycle-N: | Result | Action | |---|---| | KER dropped meaningfully on targeted categories (e.g. drug KER −20% or more, relative) | ✅ Keep the `.nemo`. Update the leaderboard. Advance to Step 4f if you want to deploy. | | KER moved a little, you wanted more | Loop back to `/digital-health-clinical-asr-build`, expand the manifest. Tiny manifests rarely benefit from hyperparameter tweaks — signal density beats LR sweeps. | | KER got worse | Overfit on a tiny manifest. Bail to `/digital-health-clinical-asr-build` and grow before retraining. Don't tune harder on the same data. | | No measurable change | Some categories may already be in the base model's vocab. Sanity-check per-category numbers before concluding training "didn't help." | ### 4f. (Optional) Deploy as a Riva NIM Hand the `.nemo` to `/riva-asr-custom`. **Pass the source architecture explicitly** — `/riva-asr-custom` can't reliably detect CTC vs RNNT vs TDT from the `.nemo` alone, and the wrong NIM container produces a broken RMIR with no clear error: | Source decoder | `riva-build` flag | NIM container family | |---|---|---| | Conformer-CTC | `decoder=greedy_ctc` | `parakeet-*-ctc-*` | | Conformer-RNNT | `decoder=nemo` | `parakeet-rnnt-*` | | **Conformer-TDT (default)** | `decoder=nemo` | `parakeet-tdt-*` | | Cache-Aware RNNT (Nemotron streaming) | `decoder=nemo` | `nemotron-streaming-*` ⚠ SFT broken on this base, see Limitations | After deploy: re-run `/digital-health-clinical-asr-eval` against the new endpoint (`ASR_ENDPOINT=localhost:50051`) to validate that production-serving numbers match offline numbers. Any divergence is in Riva preprocessing or `riva-build` flags, not the model. Route to `/riva-asr-custom`. ## Examples **Scenario A — gate met.** User: *"Drug KER 0.42, 130 rows. SFT?"* → Yes (gate cleared). `parakeet-tdt-0.6b-v2` (verified 0.513 → 0.128). No local GPU? Step 4a (Brev) → 4b (split) → 4d (stock SFT) → 4e (offline re-eval). If cycle-2 drug KER drops ≥ 20% relative, keep the `.nemo`; otherwise back to `/digital-health-clinical-asr-build`. **Scenario B — Nemotron Streaming.** User: *"SFT `nvidia/nemotron-speech-streaming-en-0.6b`?"* → No (UNK collapse). Substitute `parakeet-tdt-0.6b-v2`. Riva chunks non-streaming bases for streaming serving — base doesn't need to be streaming-native. **Scenario C — cycle 2 KER unchanged.** User: *"KER barely moved."* → Back to `/digital-health-clinical-asr-build`. Signal density beats LR sweeps. If `magpie_g2p` rows are bad but `merriam-webster` rows are good, the gap is pronunciation-coverage — `/digital-health-clinical-asr-build` Step 2d. ## Artifacts produced - `train.jsonl`, `validation.jsonl` — term-aware split (Step 4b) - `adapted_model.nemo` — fine-tuned model (Step 4d) - `training_run_info.json` — hyperparameters, dataset stats, end-of-train metrics - `offline_hyps.jsonl` — cycle-N+1 transcription hypotheses (Step 4e) - `leaderboard_cycle.md` — cycle-N+1 five-section leaderboard - *(optional, after Step 4f)* a deployed NIM endpoint (delegated to `/riva-asr-custom`) ## Troubleshooting - **Stage 4 training collapses to all-UNK after first step** → you're on the cache-aware streaming RNNT base (`nemotron-speech-streaming-en-0.6b`). Route to `nvidia/parakeet-tdt-0.6b-v2` (the recommended default) or `nvidia/stt_en_conformer_ctc_large` (legacy fallback). The streaming RNNT SFT path is broken; do not retry with different hyperparameters. - **Manifest paths don't resolve inside the NeMo container** → host paths (e.g. `$HOME/…`) need rewriting to `/workspace/…`. See `references/container-paths.md` for the rewrite snippet. - **Cycle N+1 KER unchanged from cycle N** → on `parakeet-tdt-0.6b-v2` with the recipe above, this almost always means **manifest signal density is too low**. Grow the manifest first; don't sweep LR. (If you're on an older adapter-style recipe instead of stock SFT, the adapter weights may not have moved off zero-init — switch to stock SFT.) - **Cycle N+1 KER got worse** → overfit on a tiny manifest. Bail to `/digital-health-clinical-asr-build` and grow. - **Riva-served numbers diverge from offline numbers** → the gap is in Riva preprocessing or `riva-build` flags, not the model. Route to `/riva-asr-custom`. - **`bf16-mixed` precision errors** → some GPUs (older Turing, all Volta) don't support BF16. Drop to `fp32` and reduce `batch_size`. Use `fp16-mixed` only if `fp32` is too slow — fp16 with TDT decoders can produce NaN losses, so check loss curves early. - **OOM during training on 24 GB GPU** → drop `batch_size` to 2, raise `accumulate_grad_batches` to 2 to keep the effective batch size constant. ## Limitations - **Adapter-style SFT on TDT/RNNT decoders is broken.** Empirically confirmed: an earlier LinearAdapter-mixin recipe produces 72 NaN tensors at any LR on TDT and RNNT decoders. Resolved by switching to NeMo's **stock full-model SFT** (`speech_to_text_finetune.py`) — which is what this skill recommends. Do not attempt adapter SFT on TDT/RNNT bases. - **Don't SFT `nemotron-speech-streaming-en-0.6b`.** The streaming-only NVCF function's SFT path is unreliable (UNK collapse). For streaming serving at deploy time, Riva chunks a non-streaming base. - **Tiny manifests overfit fast.** Below ~100 rows total or ~5 rows per priority category, cycle-N+1 numbers are noisy. Grow before trusting a small KER drop. - **English-only by default.** The base-model table is en-US-specific. Other locales need a different base + a re-validated SFT recipe. - **No turn-key driver.** The user writes their own training-driver layout — output paths, run naming, leaderboard re-rendering. The methodology and recipes transfer; exact cycle-1 numbers depend on the user's manifest. ## Next steps - **Deploy the `.nemo` as a NIM:** `/riva-asr-custom` (pass the source architecture explicitly). - **Grow the manifest for cycle N+2:** `/digital-health-clinical-asr-build`. - **Re-score the cycle:** `/digital-health-clinical-asr-eval` (against the new endpoint or the new `.nemo` directly). - **Lateral** for word boosting / LM fusion / non-clinical SFT recipes: `/finetune-asr`. ## References - [`references/stage4-finetune.md`](references/stage4-finetune.md) — base-model selection table, hyperparameter rationale, decoder → NIM container mapping, decision tree comparing cycle-N+1 to cycle-N - [`references/container-paths.md`](references/container-paths.md) — host → `/workspace/` path rewriting for cross-host manifest portability (laptop ↔ Brev ↔ NeMo container)

digital-health-clinical-asr-finetune

Description

Install digital-health-clinical-asr-finetune with One Click