Application and Implementation of Multimodal Relational Networks in Financial Risk Control

Introduction The presentation introduces the concept of multimodal relational networks and their application in financial risk control, outlining four main parts: background, key technologies, case studies, and summary.

1. Application Background Traditional relationship networks rely on structured data such as personal and device information, which have limited risk detection capability. In financial processes, users also generate unstructured data (ID photos, live photos, voice recordings, etc.) that are valuable for building richer relationship graphs.

2. Multimodal Relational Network By incorporating modalities such as live background, ID background, business premises, voiceprint, micro‑expressions, emotions, GPS, IP, device, phone number, and textual information, the network becomes more complex and informative, enabling better risk identification.

3. Key Technologies The solution includes data sources covering the entire loan lifecycle, a processing pipeline that extracts features from structured and unstructured data, and a system architecture that stores entities and relationships in a graph database while leveraging GPU‑accelerated deep‑learning models (e.g., DINOv2) and vector engines for fast similarity search. It also integrates rule engines, AI platforms, and modeling services.

4. Application Scenarios The multimodal network supports model development (enriching risk model features), strategy application (enhancing user profiles and detecting suspicious GPS clusters), data mining (vectorizing unstructured data), blacklist creation (voiceprint, facial, background blacklists), gang detection (identifying rapid repeated occurrences under the same GPS), relationship reasoning (inferring colleague or familial ties), and more.

5. System Architecture The architecture consists of business systems feeding data to the multimodal graph layer, which processes basic entities (devices, phones) and rich modalities (images, audio, video). A vector engine provides real‑time similarity search, while higher‑level services offer visualization, feature factories, facial and background search, and support downstream scenarios such as pre‑loan, in‑loan, post‑loan, and anti‑fraud investigations.

6. Technical Details Background similarity combines global deep‑learning features (DINOv2) with traditional image processing cues (texture, color, structure). Voiceprint extraction uses mel‑spectrograms transformed into images, followed by a triplet‑loss‑based training similar to facial recognition. Large‑model techniques are employed for image‑based fraud detection.

7. Risk Control Case Studies Examples include detecting short‑term GPS clusters indicative of black‑market loan applications, linking transactions to shared premises or backgrounds during fraud investigations, and identifying voiceprint clusters that reveal collusive intermediaries in post‑loan monitoring. The multimodal features also improve credit modeling, delivering roughly a 4% KS uplift.

8. Summary and Outlook The multimodal relational network enriches traditional graphs with image, voice, and text data, enhancing model precision, fraud detection, and risk assessment. Future work will explore additional modalities (micro‑expressions), performance optimization for super‑nodes, algorithmic improvements for background similarity, and tighter multimodal fusion.