How End-to-End Phoneme Recognition Boosts English Pronunciation Detection

Opportunity

English is the global lingua franca, yet many Chinese learners suffer from "dumb English," Chinese‑style English, and inaccurate pronunciation, which hampers listening and speaking skills. Recent curriculum reforms have added oral exams to high‑school entrance tests, increasing demand for effective, scalable pronunciation feedback.

Computer‑assisted language learning can provide instant, targeted pronunciation guidance without the time‑space constraints of traditional classroom tutoring.

Industry Status

Most existing speech‑assessment apps only assign a score and rarely explain the reasons for errors, limiting their usefulness. Phoneme‑level error detection research has grown, enabling detection of insertions, deletions, and substitutions, and delivering expert‑level corrective suggestions.

Traditional forced‑alignment methods compute Goodness‑of‑Pronunciation (GOP) scores based on precise phoneme time boundaries, but they suffer from inaccurate boundaries, inability to handle insertions/deletions, sensitivity to timing, and complex training pipelines.

Our Practice

We evaluate our system using the L2‑ARCTIC corpus, which contains recordings from non‑native speakers of various L1 backgrounds with phoneme‑level annotations for insertions, deletions, and substitutions. Evaluation metrics include false‑alarm rate, recall, and diagnostic accuracy.

3.1 End‑to‑End Model Selection

Current end‑to‑end speech‑recognition models fall into three categories: CTC (Connectionist Temporal Classification), attention‑based encoder‑decoder (AED), and RNN‑Transducer (RNN‑T). Although AED and RNN‑T often outperform CTC on generic ASR tasks, we chose CTC for mispronunciation detection because it avoids language‑model bias that could mask error patterns.

3.2 Attention‑Based Text Fusion

Using CTC alone yields a high false‑alarm rate (~21%). Inspired by human evaluators who compare the spoken output with the target transcript, we feed the target phoneme sequence into the model as additional input, dramatically reducing false alarms.

3.3 Pronunciation Error Data Augmentation

Since manually annotated error data are scarce, we augment training data by randomly replacing phonemes to simulate errors. This reduces the false‑alarm rate to ~9% and raises diagnostic accuracy from 65% to 77%, though recall remains at 57%.

3.4 Defining Functional Boundaries

Analysis shows that frequent false alarms involve acoustically similar phoneme pairs (e.g., /ɪ/ vs /iː/). In teaching, correcting such subtle confusions is lower priority, so we lower their correction weight, further dropping false alarms to 7% and improving recall to 67%.

Conclusion and Outlook

By replacing forced alignment with end‑to‑end phoneme recognition, integrating target text via attention, and augmenting error data, we achieve significant improvements in mispronunciation detection. Future work includes collecting real‑world error annotations, applying multi‑task learning for phoneme attribute recognition, and exploring audio‑video multimodal fusion to further boost robustness, especially in noisy environments.