How D2LLM and Codefuse‑CGE Are Redefining Search with Large Language Models

Introduction

With the explosion of information, search engines directly affect user experience, and the rapid development of Large Language Models (LLMs) creates new opportunities to improve search performance. The paper focuses on two innovations—D2LLM for semantic search and Codefuse‑CGE for code search—examining their design, training objectives, and experimental results.

D2LLM Model Design and Optimization

Model Overview

D2LLM combines an LLM with a bi‑encoder architecture.

A teacher‑student framework is used: the teacher model enhances the LLM’s understanding through contrastive learning, while the student model provides an efficient encoding structure.

Comparison with Existing Methods

BERT dual encoders.

LLM cross‑encoders.

Training Objectives and Loss Functions

Contrast imitation: contrastive learning of sentence relations from the teacher.

Ranking imitation: ordering of positive and negative samples.

Feature imitation: transfer of auxiliary feature information.

Experimental Results

Against state‑of‑the‑art (SOTA) baselines, D2LLM achieves higher accuracy and lower computational cost.

Runtime analysis shows superior speed on complex semantic tasks.

CGE Model in Code Embedding

Challenges in Code Embedding

Differences between code and natural‑language expressions.

Balancing precision with storage compression.

Limitations of Existing Methods

BERT‑based code embedding models.

LLM‑based code embedding methods that impose excessive storage pressure.

Architecture of CGE

Fine‑tuned from CodeQwen1.5‑7B‑Chat to improve code comprehension.

Uses a PMA (Pooling‑by‑Multi‑head‑Attention) module for sentence‑level semantic aggregation, ensuring embedding quality.

Modified PMA achieves multi‑dimensional storage compression.

Training Strategy and Objectives

Hard negative mining through multiple techniques.

Contrastive learning to strengthen model performance.

Embedding reconstruction to preserve code semantics efficiently.

Experimental Analysis

Shows high accuracy and efficiency across several datasets.

Demonstrates strong potential for open‑source release and industry adoption.

Conclusion and Future Work

The authors plan to further optimize D2LLM and CGE for semantic search and code retrieval tasks and to explore additional large‑scale language models for practical applications.