SMedBERT: Knowledge‑Enhanced Pre‑trained Language Model for Medical Text Mining and Its Business Applications

Business Scenario Overview Dingxiangyuan started as a professional medical forum for doctors and later expanded to consumer‑facing apps, serving both B2B (doctors) and B2C (public) users with over 120 million C‑end accounts and 70 % of Chinese doctors registered.

Medical Knowledge‑Graph Construction A four‑layer taxonomy (entity → instance → concept → topic) is built by combining expert‑curated medical entities (diseases, symptoms, drugs, etc.) with algorithmic extraction (NER, relation extraction). The taxonomy enables fine‑grained user intent analysis and hierarchical representation of long‑ and short‑text queries.

SMedBERT Model The ACL paper A Knowledge‑Enhanced Pre‑trained Language Model with Structured Semantics for Medical Text Mining introduces SMedBERT, which enriches token embeddings with entity types, relations, and a “knowledge bridge” that incorporates one‑hop neighboring entities. Innovations include Mention‑neighbor Hybrid Attention and Mention‑neighbor Context Modeling, implemented via T‑Encoder, K‑Encoder, and specialized pre‑training tasks.

Experimental Results Trained on ~5 GB of Chinese medical text (~3 billion tokens), SMedBERT outperforms BERT, RoBERTa, and Knowledge‑BERT on downstream tasks such as CHIP and WebMedQA, especially when leveraging high‑frequency neighboring entities (D2) versus low‑frequency ones (D3).

Industrial Deployment and Reflections The knowledge graph is integrated into search pipelines (text correction, phrase extraction, NER, entity linking, semantic understanding) and query expansion via Bayesian and translation‑model approaches. Semantic matching progresses from Bi‑Encoder to Cross‑Encoder, Poly‑Encoder, and finally Poly‑Encoder enhanced with SMedBERT and contrastive learning (ConSERT, SimCSE). Future challenges include reducing annotation cost, improving graph reuse for new domains, and handling long‑tail low‑frequency user behavior.

Q&A The model is open‑source on GitHub; token embeddings are fused with graph embeddings via Trans‑based methods; data sources include medical books, drug manuals, and popular science articles.