ASR.lab

Context

The evaluation and deployment of Automatic Speech Recognition (ASR) systems in production environments face the broad diversity and variability of real-world acoustic conditions. Speech signals are frequently degraded by ambient noise, room reverberation, compression codecs, or significant volume fluctuations. Selecting the most suitable ASR model therefore requires a rigorous, evidence-based approach to precisely measure each engine’s behavior under its own acoustic constraints (e.g., contact center calls, meetings, in-vehicle cabin noise), in order to find the optimal trade-off between accuracy, latency, and infrastructure costs.

Objective

This project aimed to design and implement a systematic, automated evaluation platform for comparing ASR engines under reproducible and configurable acoustic conditions. The expected deliverables included:

1. An audio processing pipeline capable of applying realistic acoustic degradations, running advanced speech enhancement filters, and normalizing loudness levels.
2. A unified adaptation layer facilitating communication with various local and cloud-based ASR engines.
3. A declarative YAML-based configuration system enabling automated combinatorial analyses (grid searches).
4. An interactive, self-contained dashboard for analyzing, filtering, and exporting quality metrics from the benchmark.

Approach

1. Declarative architecture and orchestration: Design of a Python 3.12 orchestrator that parses a declarative YAML configuration to generate a Cartesian product of test scenarios.
2. Upstream audio processing pipeline:
   • Controlled audio degradation: integration of VST3-format audio plug-ins to apply realistic filters (reverberation, additive noise, distortion, etc.) to reference audio files. The pipeline enables testing multiple acoustic degradation scenarios to measure the breaking point — and therefore the resilience — of each model.
   • Optional restoration (Speech Enhancement): optional branching of deep learning-based denoising and source separation models (Demucs and DeepFilterNet) onto degraded signals, in order to assess the effectiveness of these pre-processing steps on speech recognition performance.
   • Loudness normalization: automatic normalization to various loudness levels expressed in LUFS according to the EBU R128 broadcast standard, eliminating loudness bias during the transcription phase.
3. Unified ASR transcription engines: development of a universal adapter unifying access to 7 automatic speech recognition frameworks:
   • Whisper (OpenAI / Faster-Whisper optimized variant), Wav2Vec2 (Meta via Hugging Face Transformers), NVIDIA Parakeet, Vosk (Kaldi offline), SeamlessM4T (Meta), Moonshine, and SenseVoice (Alibaba).
4. Metric computation across two text formats:
   • Computation of industry-standard metrics: Word Error Rate (WER), Character Error Rate (CER), Match Error Rate (MER), Word Information Lost (WIL), and Word Information Preserved (WIP).
   • These metrics are computed on the raw transcribed text, and after applying a standardized text normalization pipeline (punctuation removal, number normalization, lowercasing) to isolate phonetic recognition errors from mere text formatting discrepancies.
5. Standalone report generation: creation of a rendering module generating a single, portable, self-contained HTML report. Interactive visualizations (WER heatmaps, scatter plots, word-level colored text diffs) are rendered client-side with no dependency on Plotly or Pandas, optimizing display speed and offline usability.

Features

• 7 supported ASR frameworks: Whisper, Wav2Vec2, NeMo, Vosk, SeamlessM4T, Moonshine, and SenseVoice, all locally executable.
• Automatic grid search: full combinatorial generation of evaluation scenarios from YAML files.
• Standalone interactive reports: client-side HTML dashboard with no third-party runtime dependencies (Pandas/Plotly), exportable to CSV.
• Plugin-based extensibility: open architecture facilitating the addition of new transcription engines or evaluation metrics.
• Modern tooling: Python 3.12+ codebase, strict static typing, dependency management via uv, MIT license.

Impact

ASR.lab addresses a critical industrial problem by transforming the choice of a speech recognition engine into a purely factual, quantified decision:

• Identification of resilient engines: testing under progressively challenging acoustic conditions demonstrates that while some models (such as Whisper) excel on clean signals, they degrade rapidly under heavy perturbations, whereas others (such as NVIDIA Parakeet models) prove highly resilient, maintaining near-constant accuracy regardless of the level of noise or reverberation applied.
• Fine-grained detection of non-acoustic biases: through the word-level coloring integrated into the interactive reports, users can directly visualize the nature of errors. This reveals that for the top-performing engines, an apparent drop in WER often stems not from poor acoustic recognition, but from minor normalization discrepancies (e.g., a number written in digits instead of words, or a word capitalized in the reference but lowercased in the transcription).
• Infrastructure cost savings: helps identify whether a lighter preprocessed model (e.g., DeepFilterNet + Whisper Base) outperforms a costly, unprocessed large model (such as Whisper Large-v3), thereby reducing GPU requirements and total cost of ownership (TCO).
• Scientific reproducibility: benchmarks are fully reproducible through YAML configuration storage, facilitating regression testing when deploying new model versions.