Multi-Engine Concurrent Search Architecture for Idlefish

Idlefish search serves many scenarios through a central search platform, involving multiple roles (engineers, algorithm engineers), businesses (e.g., second‑hand, rental, posts) and nodes (offline data aggregation, online recall, URF Rank). The existing single‑engine pipeline is highly coupled, leading to low iteration efficiency, high risk of changes, and weak cross‑business mixing capability.

The platform consists of offline dump, full/incremental indexing, and online services. Core components include the SP entry (query parsing, paging recall, data completion), HA3 full‑text engine, QRS dispatcher, and Searcher that performs recall, filtering, scoring and summarization.

Design goals focus on safe, flexible, and efficient multi‑business support: deep engine‑side isolation, concurrent multi‑engine recall, result merging for unified URF Rank, and real‑time intervention for business incubation.

Implementation highlights: function get_search_biz_type_list(query, param) ... end This Lua function parses the user query to extract multi‑engine identifiers and caches configuration. Flow control retrieves engine‑specific settings from Alibaba’s Diamond configuration center: function parse_diamond_config(query, param) ... end Query rewriting is encapsulated in functions such as: function rewrite_embedding_recall(query, switch, param) ... end Concurrent multi‑engine recall leverages SPL’s async coroutine capability: function multi_parse(res_vec) ... end

Results show significant improvements: dump build time reduced from ~14 h to ~5 h, integration time shortened from one week to two days, overall latency increase only ~15 ms, memory usage cut from 21.6 TB to ~210 MB for the second‑hand engine, and business metrics (exposure +50%, orders +33%) markedly improved.

Future work includes moving query‑rewrite and merging logic to the business side for better stability, providing visual flow‑control configuration, and migrating core SP code from Lua to Java to lower development cost.