Insights from a Top Contestant on the Tencent Advertising Algorithm Competition: Transformer Modeling and Model Fusion

Jinzhen, the team leader from Xiamen University, introduced himself and his teammate, noting that they currently rank second in the 2020 Tencent Advertising Algorithm Competition and have previously achieved fifth place in the finals.

Their basic solution follows the common approach of using word2vec combined with a sequence model, feeding five concatenated inputs into the model and training it to predict 20 classes.

He discusses practical experience with transformers, recommending that the transformer learning rate be set to roughly one‑fifth of the LSTM learning rate before fine‑tuning.

Regarding sequence information in transformers, he suggests that position embedding may not be optimal and encourages exploring alternative methods.

When using a transformer followed by max‑pooling and a dense layer, he reminds readers to handle the outputs of masked positions before max‑pooling.

He describes their recent efforts to improve model fusion by increasing model diversity, not only through different architectures but also by varying input sequences and model parameters, which has yielded thousand‑level improvements.

The article concludes with encouragement for participants to maintain strong performance in the final week, wishes them success, and provides QR codes and links to join the official competition group.