CTR Prediction Optimization for App Store Recommendation: Integrating DeepWalk, BERT, and Attention Mechanisms

This article explores CTR (Click-Through Rate) prediction optimization for Tencent's App Store recommendation system. The authors address two key challenges: significant disparity in app exposure (head apps dominate while long-tail apps receive insufficient曝光) and extremely sparse user behavior data.

The research builds upon the Wide&Deep model as a baseline and proposes several optimization strategies:

1. Multi-behavior Fusion Training: The authors designed an appid embedding sharing mechanism across different user behavior types (clicks, downloads, installations). This approach reduced model parameters from 28 million to 20 million while improving the COPC (prediction bias) metric.

2. Long-period User Behavior via DeepWalk: Using graph-based DeepWalk embeddings trained on 30-day user download behavior to capture longer-term user preferences. They experimented with three training strategies: fixed initialization, initialization with fine-tuning, and feature distillation. Only the fine-tuning approach showed improvement.

3. App Description Text via BERT: Pre-trained BERT models were used to generate 768-dimensional text embeddings from app titles and descriptions. The authors explored PCA-based and end-to-end learned dimensionality reduction, finding the latter more effective.

4. Pre-trained Embedding Fusion: Both DeepWalk and BERT embeddings were projected into a shared vector space using learnable transformation parameters. Concatenation of projected embeddings outperformed addition.

5. Attention-based User Behavior Mining: Using the fused DeepWalk+BERT embeddings to initialize attention query and key vectors, significantly improving AUC metrics.

The final model demonstrates that combining behavioral embeddings (DeepWalk) with content-based embeddings (BERT) and attention mechanisms can effectively address sparse user behavior challenges in recommendation systems.