Voice Analysis for Financial Risk Control: Feature Extraction, Single-Channel Speech Separation, and Text Tagging

Introduction

In financial customer‑service scenarios, multimodal data such as audio, video, and images are generated. This presentation explores the use of voice analysis for risk control, focusing on voice feature extraction, single‑channel speech separation, and speech‑text labeling to identify and mitigate risk.

1. Voice‑Based Risk Feature Extraction

Large volumes of unlabelled call recordings are often underutilised. By converting each call into separate customer and agent time‑series signals, we extract time‑domain features such as amplitude, amplitude difference, duration, and average volume. These features enable labeling of calls into categories like AI‑answered, invalid audio, or invalid call, which can be directly applied to risk management.

We further enrich these labels with over 400 risk‑related features. Statistical analysis of 90‑day and 360‑day windows shows a positive correlation between the number of voice‑based tags and user risk, indicating that voice tags are predictive of real risk.

Experiments with traditional models (XGBoost, Logistic Regression) show that adding voice features increases the identification rate of high‑risk users from 3.24× to 4.16×. Voice tags can also be incorporated as additional features in credit‑risk models, further improving detection.

2. Single‑Channel Speech Separation Technology

While dual‑channel recordings allow straightforward separation of customer and agent speech, many real‑world recordings are mono. Single‑channel separation (the “cocktail‑party” problem) aims to isolate target voices from noisy mixtures.

Our approach uses a short‑time Fourier transform to obtain a spectrogram, a deep learning model (e.g., Grid LSTM) to predict a mask, and then applies the mask to the mixed spectrogram before inverse transforming to obtain separated audio.

The pipeline consists of two stages: training (computing loss between masked output and ground‑truth) and inference (applying the mask to new audio). Evaluation uses Signal‑to‑Distortion Ratio (SDR), achieving SDR≈16 in our scenario, supplemented by manual listening tests to verify separation quality.

3. Speech‑Text Tagging

Beyond raw audio, the transcribed text of calls provides additional risk signals. We propose a pipeline to identify user intent (e.g., willingness to make a payment) by first converting speech to text vectors, performing unsupervised clustering, manually labeling cluster centroids, and then refining labels through supervised training.

Unsupervised clustering is accelerated using a custom fast community‑detection algorithm that iteratively refines similarity matrices and merges clusters based on a Top‑K rule.

For supervised training, we select ~20 meaningful tags, annotate ~200 samples per tag, and fine‑tune BERT‑based models for multi‑class classification and sentence similarity tasks. Evaluation shows that models such as Chinese‑RoBERTa‑wwm‑ext + whitening + lstavg and Sentence‑Transformers‑paraphrase‑multilingual‑MiniLM‑L12‑v2 + lstavg perform best in unsupervised clustering, while regularized Sentence‑Transformers‑paraphrase‑multilingual‑MiniLM‑L12‑v2 + lstavg and fine‑tuned Chinese‑RoBERTa‑wwm‑ext excel in supervised tasks.

4. Summary and Outlook

The current work demonstrates the feasibility of using voice features, single‑channel separation, and speech‑text tagging for financial risk control. Future directions include deeper deep‑learning or reinforcement‑learning based feature mining, reducing reliance on manual labeling, and extending models to handle varying numbers of audio channels.

Voice feature extraction can be enhanced with advanced neural or reinforcement learning methods.

Manual labeling remains a bottleneck; unsupervised methods need better automatic evaluation.

Current models are limited to a fixed channel count; generalizing to mono, stereo, or multi‑channel inputs is required.

Thank you for listening.