u/CupGlass540

I spent months trying to train a 730M-parameter TTS model on my own hardware. It wouldn't converge, and nothing in my stack would tell me why. Not the loss curve, not TensorBoard, not the checkpoints. Every tool I had showed me numbers. None of them would say "this run is already dead, stop paying for it."

That's the gap I built trainproof for (MIT, `pip install trainproof`). It's a deterministic linter for training runs: it reads the logs you already produce and returns a verdict with an exit code.

No ML judging ML, no confidence scores. Every check is a rule that fires or doesn't, and prints the number it fired on. A reliability tool that hallucinates is worse than no tool, because then you stop trusting your own alarms.

Severity and exit code are separate on purpose:

FAIL -> exit 1 your run is broken

WARN -> exit 0 worth your attention

NOT-CHECKED -> exit 2 I could not judge this

PASS -> exit 0 checked, fine

A tool that can't tell "your run failed" from "I couldn't read your log" is lying to your CI quietly.

Validating a detector means feeding it faults you already know the answer to, so the rules were measured against a controlled fault-injection study: one Qwen2.5-3B QLoRA, six configurations - healthy, 100x LR, lr=0, fp16 NaN, shuffled labels, overfit - three seeds each, 18 runs. The 100x LR spiked grad-norm to ~2,650, about 4,900x its own median, caught in seconds.

The result worth posting is the one that got through. Shuffled labels - a dataset that cannot be learned - REDUCED its loss by 69.8% (18.9 -> 5.7) and looked textbook-healthy on its own curve. It was memorizing the statistics of noise. From a single run's loss curve that's indistinguishable from real training, so it's written into the README as a stated limitation, and it's why `compare` exists: put the run next to a known-good baseline and the relative floor gives it away immediately.

Then the rules went against real fine-tunes I'd already paid for. Both logs ship in evidence/ so you can reproduce the verdicts:

Coqui XTTS v2, 125,000 steps -> FAIL (TP-DIVERGE, TP-THROUGHPUT)

Fish Speech LoRA (Lightning), 2049 -> WARN (TP-OVERFIT)

TP-OVERFIT means eval loss climbed past 1.2x its own minimum while train loss kept falling: your best checkpoint has already gone by, and if you keep only the last one, you kept the wrong one. That XTTS run is read by two independent readers - Coqui's text log and its TensorBoard event file, same run - and they return the same verdict and the same rule set.

Real logs also proved the tool wrong, and that's the part I'd defend hardest. TP-ZERO-GRAD fired whenever every gradient norm was exactly 0.0 and reported a severed backward graph. Coqui writes avg_grad_norm as 0.0 when clipping is off, so a healthy 125k-step run whose loss reached 0.017 got a FAIL from my own tool. The fix was reasoning, not a threshold tweak: a run cannot both learn and receive no gradient, so the check now stands down when the loss improved - and records why it stood down as a visible skip, because a check that didn't run must never look like a check that passed. No test caught that. One real log did, in an afternoon.

Across a run's life:

- before the GPU: dataset + tokenizer lint (malformed JSONL w/ line number,

empty rows, dupes, missing eos_token, pad==eos), plus `env` - does your

entrypoint even import (probed in a subprocess), is the checkpoint intact,

RAM, disk

- during: one-line HF callback; warns, or aborts a diverging run if you opt in

- after: diverged / flatlined / NaN'd / spiked / overfitting

- vs baseline: the relative-floor rules

Reads HF trainer_state.json / Coqui / TensorBoard event files / JSONL / CSV. The tfevents reader is written from the wire format - no tensorflow, no tensorboard, no protobuf, no torch - validated byte-exact against EventAccumulator on a real 2049-step Lightning run. Truncated event files, the normal state of a killed run, are read up to the cut instead of raising. Checkpoints are inspected WITHOUT unpickling, as the ZIP archives they are; torch.load executes arbitrary code by design, which is why torch 2.6 flipped weights_only to True.

Where it is now: 84 stable rule IDs, 230 tests, 17 releases, a written contract in CONTRACTS.md, and every example verdict frozen in 38 golden snapshots - a rule that stops firing and one that fires spuriously both break the build.

Repo: https://github.com/Mormolykos/trainproof PyPI: https://pypi.org/project/trainproof/ Write-up with the full fault-injection results: https://ai.bedvibe.studio/trainproof/ Sibling project it builds on: https://pypi.org/project/ttsproof/ (failure-mode QA for TTS) More of what I've built: https://tts.bedvibe.studio/portfolio/

What failure mode has burned your GPU hours? If a deterministic check would have caught it, tell me and it goes in, with credit.

I ran a matched-content experiment on speaker verification: 4 speakers, identical sentences, controlled changes in pitch, phonation and articulation, fixed microphone position and gain, every utterance through three encoders — ECAPA-TDNN, a ResNet speaker encoder, and WavLM-base-plus-sv.

One speaker sat 0.7 semitones below his normal pitch. That is below the threshold where a listener reliably hears any change at all. His verification score dropped 0.238 across all eight sentences.

The interesting part is that pitch was not what moved him. His harmonics-to-noise ratio fell from 10.63 to 8.39 dB over the same block. The encoder was reacting to phonation, and the pitch number was just the thing that happened to be easy to measure.

Across the whole run, all 30 speaker × condition × encoder cells were negative, 28 of them unanimous across every utterance. These systems are not mainly disturbed by shouting or disguise. They are disturbed by someone speaking slightly differently in a way nobody would notice.

To be clear about credit: the displacement effect itself is not my finding. I had it as my headline until an adversarial prior-art audit turned up Hughes et al. (Interspeech 2023), who established it with six trained phoneticians across seventeen conditions. I withdrew the claim and reframed the paper as a replication and extension across encoder architectures. The audit is published in full, including the claim it killed.

Two things I could not find in prior work:

Jitter beats HNR. Pitch deviation and jitter are independently associated with displacement in all three encoders, and HNR adds nothing once jitter is in the model — it correlates 0.55 with jitter and loses all independent power beside it. HNR is the measure most people reach for, and on its own it will attribute the effect to the wrong thing.

Rough phonation breaks F0 trackers in one direction. 10 of 145 utterances carried impossible pitch values, up to 32.1 semitones — a 6.4× frequency ratio no human produces. All 10 were in rough phonation, none in modal (Fisher exact p = 2.4e-11). Octave errors push upward and essentially never downward, so this is differential measurement error, not noise, and it loaded onto one specific regression coefficient. Cheap fix: validate your F0 tracker per phonation condition and publish the validation.

Everything is up — manuscript, pre-registration, the full prior-art audit, per-utterance tables for all three encoders (233 rows each), the analysis scripts, and the 137 source recordings. Every number reproduces from the CSVs without touching the audio.

Paper and data: https://doi.org/10.5281/zenodo.21921958

Write-up: https://ai.bedvibe.studio/speaker-drift/

Audio licence note: research, benchmarking, evaluation and teaching are permitted; ML training and voice cloning are not. The speakers are identifiable adults.

I ran a matched-content experiment on speaker verification: 4 speakers, identical sentences, controlled changes in pitch, phonation and articulation, fixed microphone position and gain, every utterance through three encoders — ECAPA-TDNN, a ResNet speaker encoder, and WavLM-base-plus-sv.

One speaker sat 0.7 semitones below his normal pitch. That is below the threshold where a listener reliably hears any change at all. His verification score dropped 0.238 across all eight sentences.

The interesting part is that pitch was not what moved him. His harmonics-to-noise ratio fell from 10.63 to 8.39 dB over the same block. The encoder was reacting to phonation, and the pitch number was just the thing that happened to be easy to measure.

Across the whole run, all 30 speaker × condition × encoder cells were negative, 28 of them unanimous across every utterance. These systems are not mainly disturbed by shouting or disguise. They are disturbed by someone speaking slightly differently in a way nobody would notice.

To be clear about credit: the displacement effect itself is not my finding. I had it as my headline until an adversarial prior-art audit turned up Hughes et al. (Interspeech 2023), who established it with six trained phoneticians across seventeen conditions. I withdrew the claim and reframed the paper as a replication and extension across encoder architectures. The audit is published in full, including the claim it killed.

Two things I could not find in prior work:

Jitter beats HNR. Pitch deviation and jitter are independently associated with displacement in all three encoders, and HNR adds nothing once jitter is in the model — it correlates 0.55 with jitter and loses all independent power beside it. HNR is the measure most people reach for, and on its own it will attribute the effect to the wrong thing.

Rough phonation breaks F0 trackers in one direction. 10 of 145 utterances carried impossible pitch values, up to 32.1 semitones — a 6.4× frequency ratio no human produces. All 10 were in rough phonation, none in modal (Fisher exact p = 2.4e-11). Octave errors push upward and essentially never downward, so this is differential measurement error, not noise, and it loaded onto one specific regression coefficient. Cheap fix: validate your F0 tracker per phonation condition and publish the validation.

Everything is up — manuscript, pre-registration, the full prior-art audit, per-utterance tables for all three encoders (233 rows each), the analysis scripts, and the 137 source recordings. Every number reproduces from the CSVs without touching the audio.

Paper and data: https://doi.org/10.5281/zenodo.21921958

Write-up: https://ai.bedvibe.studio/speaker-drift/

Audio licence note: research, benchmarking, evaluation and teaching are permitted; ML training and voice cloning are not. The speakers are identifiable adults.

I spent months trying to train a 730M-parameter TTS model on my own hardware. It wouldn't converge, and nothing in my stack would tell me why. Not the loss curve, not TensorBoard, not the checkpoints. Every tool I had showed me numbers. None of them would say "this run is already dead, stop paying for it."

That's the gap I built trainproof for (MIT, `pip install trainproof`). It's a deterministic linter for training runs: it reads the logs you already produce and returns a verdict with an exit code.

No ML judging ML, no confidence scores. Every check is a rule that fires or doesn't, and prints the number it fired on. A reliability tool that hallucinates is worse than no tool, because then you stop trusting your own alarms.

Severity and exit code are separate on purpose:

FAIL -> exit 1 your run is broken

WARN -> exit 0 worth your attention

NOT-CHECKED -> exit 2 I could not judge this

PASS -> exit 0 checked, fine

A tool that can't tell "your run failed" from "I couldn't read your log" is lying to your CI quietly.

Validating a detector means feeding it faults you already know the answer to, so the rules were measured against a controlled fault-injection study: one Qwen2.5-3B QLoRA, six configurations - healthy, 100x LR, lr=0, fp16 NaN, shuffled labels, overfit - three seeds each, 18 runs. The 100x LR spiked grad-norm to ~2,650, about 4,900x its own median, caught in seconds.

The result worth posting is the one that got through. Shuffled labels - a dataset that cannot be learned - REDUCED its loss by 69.8% (18.9 -> 5.7) and looked textbook-healthy on its own curve. It was memorizing the statistics of noise. From a single run's loss curve that's indistinguishable from real training, so it's written into the README as a stated limitation, and it's why `compare` exists: put the run next to a known-good baseline and the relative floor gives it away immediately.

Then the rules went against real fine-tunes I'd already paid for. Both logs ship in evidence/ so you can reproduce the verdicts:

Coqui XTTS v2, 125,000 steps -> FAIL (TP-DIVERGE, TP-THROUGHPUT)

Fish Speech LoRA (Lightning), 2049 -> WARN (TP-OVERFIT)

TP-OVERFIT means eval loss climbed past 1.2x its own minimum while train loss kept falling: your best checkpoint has already gone by, and if you keep only the last one, you kept the wrong one. That XTTS run is read by two independent readers - Coqui's text log and its TensorBoard event file, same run - and they return the same verdict and the same rule set.

Real logs also proved the tool wrong, and that's the part I'd defend hardest. TP-ZERO-GRAD fired whenever every gradient norm was exactly 0.0 and reported a severed backward graph. Coqui writes avg_grad_norm as 0.0 when clipping is off, so a healthy 125k-step run whose loss reached 0.017 got a FAIL from my own tool. The fix was reasoning, not a threshold tweak: a run cannot both learn and receive no gradient, so the check now stands down when the loss improved - and records why it stood down as a visible skip, because a check that didn't run must never look like a check that passed. No test caught that. One real log did, in an afternoon.

Across a run's life:

- before the GPU: dataset + tokenizer lint (malformed JSONL w/ line number,

empty rows, dupes, missing eos_token, pad==eos), plus `env` - does your

entrypoint even import (probed in a subprocess), is the checkpoint intact,

RAM, disk

- during: one-line HF callback; warns, or aborts a diverging run if you opt in

- after: diverged / flatlined / NaN'd / spiked / overfitting

- vs baseline: the relative-floor rules

Reads HF trainer_state.json / Coqui / TensorBoard event files / JSONL / CSV. The tfevents reader is written from the wire format - no tensorflow, no tensorboard, no protobuf, no torch - validated byte-exact against EventAccumulator on a real 2049-step Lightning run. Truncated event files, the normal state of a killed run, are read up to the cut instead of raising. Checkpoints are inspected WITHOUT unpickling, as the ZIP archives they are; torch.load executes arbitrary code by design, which is why torch 2.6 flipped weights_only to True.

Where it is now: 84 stable rule IDs, 230 tests, 17 releases, a written contract in CONTRACTS.md, and every example verdict frozen in 38 golden snapshots - a rule that stops firing and one that fires spuriously both break the build.

Repo: https://github.com/Mormolykos/trainproof PyPI: https://pypi.org/project/trainproof/ Write-up with the full fault-injection results: https://ai.bedvibe.studio/trainproof/ Sibling project it builds on: https://pypi.org/project/ttsproof/ (failure-mode QA for TTS) More of what I've built: https://tts.bedvibe.studio/portfolio/

What failure mode has burned your GPU hours? If a deterministic check would have caught it, tell me and it goes in, with credit.

I run a small TTS platform and got tired of shipping voice models that sounded fine on "Hello world" and then read "3:30 PM" as "three colon thirty pee em", or looped the same 450 ms forever on short inputs.

So I turned my internal QA harness into a library: ttsproof (pip install ttsproof, MIT).

What it does:

- 817 curated edge cases across 39 categories (Benchmark Corpus 1.0): numbers, decimals, currencies, dates, time zones, phone numbers, URLs, acronyms, single letters, pronunciation torture words (Worcestershire, synecdoche, colonel...), proper names (Reykjavík, Nguyễn, Tchaikovsky...), scientific and medical vocabulary, tongue twisters, homographs, Greek, Norwegian, punctuation abuse, hallucination traps ("buy now buy now buy now"), emoji, SQL/JSON snippets...

- Structural audio checks that need no model at all: empty/truncated audio, duration explosions, long silences, clipping, repeated-chunk loop detection, end-of-clip artifacts.

- Equivalence-aware WER — "May 5, 2026" vs "may fifth twenty twenty-six" is NOT a failure. Plain WER lies about formatting; this canonicalizes both sides to spoken form first (with diacritic folding, so "Reykjavik" matches "Reykjavík").

- Honest scoring policies per category — URLs and currencies have many valid readings, so they're scored by keyword survival instead of exact match. Emoji only has to not break the audio. No fake failures.

- ttsproof benchmark --cmd "yourtts {text} {out}" → per-category scoreboard + a self-contained HTML report with waveforms and audio players for every failure.

- ttsproof regress for CI — fails the build when a fine-tune quietly breaks number pronunciation.

The corpus is versioned independently of the tool (Benchmark Corpus 1.0), so scores stay comparable across releases.

The method comes from a technical report I published: evaluated on 390 samples with a blinded human validation of the ASR-uncertain zone — that's where the "quarantine" verdict comes from. Short utterances where ASR disagrees get flagged for human ears instead of counted as failures, because at that length the ASR is as likely wrong as the TTS.

Repo: https://github.com/Mormolykos/ttsproof

Report (DOI): https://doi.org/10.5281/zenodo.20757553

v0.3, MIT, no telemetry, minimal deps (numpy + soundfile; faster-whisper optional). I want the corpus to grow from real failures, not invented ones — tell me what breaks YOUR models and it goes into Corpus 1.1 with credit.

I run a small TTS platform and got tired of shipping voice models that sounded fine on "Hello world" and then read "3:30 PM" as "three colon thirty pee em", or looped the same 450 ms forever on short inputs.

So I turned my internal QA harness into a library: ttsproof (pip install ttsproof, MIT).

What it does:

- 817 curated edge cases across 39 categories (Benchmark Corpus 1.0): numbers, decimals, currencies, dates, time zones, phone numbers, URLs, acronyms, single letters, pronunciation torture words (Worcestershire, synecdoche, colonel...), proper names (Reykjavík, Nguyễn, Tchaikovsky...), scientific and medical vocabulary, tongue twisters, homographs, Greek, Norwegian, punctuation abuse, hallucination traps ("buy now buy now buy now"), emoji, SQL/JSON snippets...

- Structural audio checks that need no model at all: empty/truncated audio, duration explosions, long silences, clipping, repeated-chunk loop detection, end-of-clip artifacts.

- Equivalence-aware WER — "May 5, 2026" vs "may fifth twenty twenty-six" is NOT a failure. Plain WER lies about formatting; this canonicalizes both sides to spoken form first (with diacritic folding, so "Reykjavik" matches "Reykjavík").

- Honest scoring policies per category — URLs and currencies have many valid readings, so they're scored by keyword survival instead of exact match. Emoji only has to not break the audio. No fake failures.

- ttsproof benchmark --cmd "yourtts {text} {out}" → per-category scoreboard + a self-contained HTML report with waveforms and audio players for every failure.

- ttsproof regress for CI — fails the build when a fine-tune quietly breaks number pronunciation.

The corpus is versioned independently of the tool (Benchmark Corpus 1.0), so scores stay comparable across releases.

The method comes from a technical report I published: evaluated on 390 samples with a blinded human validation of the ASR-uncertain zone — that's where the "quarantine" verdict comes from. Short utterances where ASR disagrees get flagged for human ears instead of counted as failures, because at that length the ASR is as likely wrong as the TTS.

Repo: https://github.com/Mormolykos/ttsproof

Report (DOI): https://doi.org/10.5281/zenodo.20757553

v0.3, MIT, no telemetry, minimal deps (numpy + soundfile; faster-whisper optional). I want the corpus to grow from real failures, not invented ones — tell me what breaks YOUR models and it goes into Corpus 1.1 with credit.

I run a small TTS platform and got tired of shipping voice models that sounded fine on "Hello world" and then read "3:30 PM" as "three colon thirty pee em", or looped the same 450 ms forever on short inputs.

So I turned my internal QA harness into a library: ttsproof (pip install ttsproof, MIT).

What it does:

- 817 curated edge cases across 39 categories (Benchmark Corpus 1.0): numbers, decimals, currencies, dates, time zones, phone numbers, URLs, acronyms, single letters, pronunciation torture words (Worcestershire, synecdoche, colonel...), proper names (Reykjavík, Nguyễn, Tchaikovsky...), scientific and medical vocabulary, tongue twisters, homographs, Greek, Norwegian, punctuation abuse, hallucination traps ("buy now buy now buy now"), emoji, SQL/JSON snippets...

- Structural audio checks that need no model at all: empty/truncated audio, duration explosions, long silences, clipping, repeated-chunk loop detection, end-of-clip artifacts.

- Equivalence-aware WER — "May 5, 2026" vs "may fifth twenty twenty-six" is NOT a failure. Plain WER lies about formatting; this canonicalizes both sides to spoken form first (with diacritic folding, so "Reykjavik" matches "Reykjavík").

- Honest scoring policies per category — URLs and currencies have many valid readings, so they're scored by keyword survival instead of exact match. Emoji only has to not break the audio. No fake failures.

- ttsproof benchmark --cmd "yourtts {text} {out}" → per-category scoreboard + a self-contained HTML report with waveforms and audio players for every failure.

- ttsproof regress for CI — fails the build when a fine-tune quietly breaks number pronunciation.

The corpus is versioned independently of the tool (Benchmark Corpus 1.0), so scores stay comparable across releases.

The method comes from a technical report I published: evaluated on 390 samples with a blinded human validation of the ASR-uncertain zone — that's where the "quarantine" verdict comes from. Short utterances where ASR disagrees get flagged for human ears instead of counted as failures, because at that length the ASR is as likely wrong as the TTS.

Repo: https://github.com/Mormolykos/ttsproof

Report (DOI): https://doi.org/10.5281/zenodo.20757553

v0.3, MIT, no telemetry, minimal deps (numpy + soundfile; faster-whisper optional). I want the corpus to grow from real failures, not invented ones — tell me what breaks YOUR models and it goes into Corpus 1.1 with credit.