Elastic's recent client‑library changes restrict connections to AWS‑hosted OpenSearch and older Elasticsearch versions, prompting AWS to fork multiple official clients and sparking a heated debate over open‑source licensing and developer freedom.
This article presents a comprehensive guide on optimizing Elasticsearch for large‑scale data platforms, covering Lucene fundamentals, index and shard architecture, doc‑values usage, routing strategies, practical performance‑tuning techniques, and real‑world testing results to achieve sub‑second query responses on billions of records.
This article outlines a backend messaging center architecture that targets 10,000 messages per second inbound via RocketMQ, 1,000 messages per second outbound to third‑party platforms, and ensures 100% high availability using Spring Cloud Gateway, Kubernetes, Elasticsearch, and related technologies.
This article introduces the most important and still mainstream components of the big data ecosystem—including Hadoop’s storage and compute framework, Hive data warehouse, HBase NoSQL database, Spark unified engine, Kafka messaging platform, and Elasticsearch search engine—explaining their core concepts, architectures, and typical use cases.
This article compares Elasticsearch and ClickHouse for log analytics, detailing their architectures, presenting a Docker‑Compose test setup that streams synthetic syslog data to both stacks, executing equivalent queries, measuring response times, and concluding that ClickHouse generally delivers faster performance, especially for aggregations.
This article explains the complete technical design of an e‑commerce coupon system, covering data models, product‑page exposure, coupon claiming rules, high‑concurrency caching, MQ‑driven expiration handling, order‑time coupon calculation, redemption flow, and search‑based coupon indexing.
This article explains the design and evolution of Baidu's word‑list service for content moderation, covering its background, multi‑layer architecture, management platform, strategy loading, matching workflow, performance optimizations for large texts, and future enhancements such as special‑character support and per‑business‑line deployment.
This article presents a practical, out‑of‑the‑box ELK‑based solution for private deployments, detailing design principles, rapid one‑click setup with Jenkins, component choices, log and metric collection using Beats, alerting with ElastAlert, and automated Kibana dashboard configuration.
This guide walks you through installing and configuring the ELK stack—Elasticsearch, Logstash, and Kibana—on Ubuntu, setting up Logstash shipper and indexer pipelines, integrating Spring Boot and Nginx logs, and managing the services with Supervisor for reliable, real‑time log analysis.
This article compares Elasticsearch and ClickHouse for log analytics, detailing their architectures, node roles, query capabilities, and performance through Docker‑Compose stacks and synthetic syslog data, concluding that ClickHouse generally outperforms Elasticsearch in speed and aggregation efficiency.
This article compares Elasticsearch and ClickHouse as log analytics solutions, detailing their architectures, deployment configurations, query capabilities, and performance benchmarks across various query types, and demonstrates that ClickHouse generally outperforms Elasticsearch in speed and aggregation efficiency.
This article compares ClickHouse and Elasticsearch for API log storage, detailing development activity, schema handling, query performance, statistical functions, MySQL integration, new features, and practical drawbacks, while providing concrete SQL examples and migration tips.
This article explores common causes of MySQL slow queries such as index misuse and lock contention, explains indexing strategies like index pushdown and covering indexes, and then compares Elasticsearch and HBase as complementary solutions for large‑scale search and write‑intensive workloads, offering practical tips for performance optimization.
This article outlines the technical, business, and product goals for a messaging center, presents a prototype and functional requirements, evaluates RocketMQ and Elasticsearch as core technologies, and details the architectural design, underlying frameworks, and DevOps strategies—including Spring Cloud Gateway, Kubernetes, and Docker—to achieve 10,000 msg/s upstream throughput, 1,000 msg/s downstream delivery, and 100 % high availability.
This article explains how Elasticsearch leverages inverted indexes, term dictionaries, and advanced compression techniques like Frame of Reference and Roaring Bitmaps to enable rapid full‑text search, covering the underlying concepts, data structures, and query optimizations essential for high‑performance search applications.
This article explains how Elasticsearch achieves fast search by using inverted indexes, detailing the structure of posting lists, term dictionaries, term indexes, and compression methods such as Frame of Reference and Roaring Bitmaps, as well as techniques for efficient union and intersection queries.
This article explains Loki's advantages over Elasticsearch, including low indexing overhead, concurrent query processing with caching, seamless integration with Prometheus and Grafana, detailed architecture components, installation steps, label handling, high‑cardinality challenges, and best practices for efficient log management.
This article examines common causes of slow queries in MySQL, presents practical indexing techniques, and extends the discussion to performance considerations in ElasticSearch and HBase, offering combined solutions such as sharding, read/write splitting, and using ES as a search layer for large‑scale systems.
Zhang Liang shares Didi's four-stage journey of constructing and governing large‑scale open‑source big‑data engine services—including engine selection, hardware sizing, PaaS platform building, proxy architecture, and governance—highlighting practical challenges, solutions, and ROI‑driven best practices for Kafka, Elasticsearch, Flink, and related technologies.
This article explains why large‑scale Elasticsearch clusters may miss log entries during keyword searches, dives into the fundamentals of inverted indexes and tokenization, and demonstrates practical index‑time and query‑time tokenizer optimizations—including custom analyzers for English and Chinese—to dramatically improve search recall and precision.
This tutorial walks through creating a Flask‑based search engine powered by Elasticsearch, covering configuration files, logging setup, blueprint routing, pagination, detail views, and deployment options using Flask‑Script and Gunicorn, with complete code examples and a GitHub reference.
This guide walks through installing the Python Elasticsearch client, building a reusable class with CRUD methods, importing data from MongoDB, writing a simple Baidu Baike crawler, and scaling the workflow with Celery and Flask for a complete search‑engine pipeline.
This tutorial walks you through why ELK is essential for large‑scale log collection, shows a simple architecture diagram, and provides detailed installation and configuration steps for Elasticsearch, Logstash, and Kibana on both macOS and Linux, including common pitfalls and Java integration.
Learn what Elasticsearch is, its core concepts, and how to integrate it into a Django project, including installation, configuration, defining indices and documents, populating data, and performing searches, all illustrated with code examples for seamless full‑text search capabilities.
This article explains Elasticsearch's write, read, and search mechanisms, the role of coordinating nodes, primary and replica shards, refresh and commit cycles, Lucene's inverted index, and how data becomes searchable in near‑real‑time.
This article explains why traditional databases like MySQL struggle with petabyte‑scale queries, introduces Elasticsearch’s architecture and inverted‑index advantages, and details a practical pipeline using Hive, Canal, Otter, and a custom DTS service to achieve near‑real‑time indexing for massive data volumes.
JD.com’s order center faced massive query loads, prompting a migration from MySQL to Elasticsearch and a multi-stage evolution of its ES cluster—from a chaotic initial setup to isolated nodes, replica tuning, master‑slave dual clusters, and real‑time backup—culminating in a robust, high‑throughput, fault‑tolerant search architecture.
This guide explains how to define and use Elasticsearch index templates, manage index aliases, configure the _source field, and understand the roles of master, data, and coordinating nodes in a 7.8 cluster, with practical API examples and visual illustrations.
This article explains why MySQL queries become slow, how indexes work and fail, the impact of MDL locks, large‑table challenges, sharding and read‑write splitting strategies, then introduces Elasticsearch’s search capabilities and HBase’s column‑family storage, offering practical guidance for each technology.
This article examines common performance pitfalls in Elasticsearch—including slow queries, cluster architecture bottlenecks, and business‑scenario challenges—and provides practical guidance such as caching key fields, data pre‑heating, hot‑cold separation, avoiding joins, and using tribe nodes to improve accuracy and response time.
The article examines the strengths and weaknesses of combining HBase and Elasticsearch for massive data storage and retrieval, outlines three integration patterns and their challenges, and presents Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost, strongly consistent solution that simplifies development and improves performance.
This article explains how Elasticsearch processes queries across shards, identifies two key performance bottlenecks—query computation time and segment file I/O—and offers practical optimization strategies such as simplifying query logic, maximizing OS file cache usage, increasing memory, reducing stored data, and applying hot‑cold data separation.
This comprehensive guide walks through Elasticsearch fundamentals, including its architecture, core concepts like indices, shards, and replicas, the write and update processes, search workflow, consistency mechanisms, master election, performance tuning, and strategies for deep pagination and scroll searches.
This article examines JD Daojia's order fulfillment system, detailing the challenges of high‑volume prompt‑sound queries, the division of responsibilities among Redis, MySQL, and Elasticsearch, the adoption of Canal for asynchronous data replication, deployment practices with Kafka and Zookeeper, and the key operational lessons learned.
This article analyzes common causes of slow MySQL queries such as index misuse, MDL locks, and large‑table bottlenecks, then presents practical solutions like proper indexing, sharding, read/write splitting, and evaluates when to complement MySQL with ElasticSearch or HBase for better performance.
This guide walks through installing Apache SkyWalking, setting up an ElasticSearch storage cluster, configuring agents, launching services, and integrating DingTalk notifications for alarm handling, providing complete commands, configuration snippets, and troubleshooting tips.
This article examines the complementary strengths of HBase and Elasticsearch, outlines three integration patterns and their associated challenges, and introduces Alibaba Cloud's Lindorm Searchindex as a SQL‑driven, low‑cost solution that simplifies storage and full‑text search for massive data workloads.
This guide walks you through downloading and installing ElasticSearch, explains core concepts like indices, types, documents, and fields, demonstrates CRUD operations via RESTful APIs, shows advanced query techniques, and provides complete Java integration examples using Maven and Docker.
Elasticsearch routes client write requests to the primary shard, buffers documents in memory, logs operations to a translog, periodically refreshes to create segment files, and uses flush and merge processes to ensure data durability, optimize I/O, and consolidate segments for efficient search.
This article presents a detailed case study of moving a 2‑petabyte smart‑product data workload from Elasticsearch to MongoDB, covering business background, source cluster architecture, MongoDB design and resource planning, step‑by‑step performance optimizations, latency and cost comparisons, and guidance on when each database is appropriate.
This guide walks through the complete architecture and step‑by‑step deployment of a billion‑scale ELK logging system, covering Filebeat agents, Kafka buffering, Logstash processing, Elasticsearch indexing, and Kibana visualization, including configuration files, version details, and best‑practice tips for scaling and security.
This guide walks through configuring Spring Cloud microservices with Nacos service discovery, adding Zipkin for distributed tracing, and persisting trace data to Elasticsearch, including all necessary Maven dependencies, YAML settings, controller code, and verification steps.
This talk explains JD.com’s order‑fulfillment workflow, the underlying storage stack of Redis, MySQL and Elasticsearch, the challenges of high‑traffic alert‑sound queries, the adoption of Canal for data heterogeneity, and a 4R‑based complexity‑oriented design methodology for building resilient backend systems.
Learn step‑by‑step how to design and deploy a billion‑scale log collection and analysis platform using the ELK stack—Filebeat, Kafka, Logstash, Elasticsearch, and Kibana—covering architecture, configuration, installation, and best practices for high‑availability and performance.
This article explains Elasticsearch’s three pagination methods—From + size, search_after, and Scroll—detailing their definitions, code examples, advantages, disadvantages, and suitable scenarios, while also discussing max_result_window limits, PIT views, and best practices for handling large result sets.
This article provides a detailed overview of the ELK stack—including Elasticsearch, Logstash, Kibana, and Beats—explaining its components, why to use it for centralized log management, various deployment architectures, system tuning, security setup, and step‑by‑step installation and configuration commands for a production‑grade environment.
This article details 58.com’s journey with Elasticsearch, covering the challenges of disparate deployments, common problems like disk exhaustion and write slowdown, the governance and automation platform they built, development standards, service architecture, real‑world application cases, and future plans for version upgrades and intelligent diagnostics.
This article provides a comprehensive introduction to the ELK stack—including component descriptions, architectural diagrams, reasons for adoption, and step‑by‑step installation and configuration of Filebeat, Logstash, Elasticsearch, and Kibana on Linux, with optional Kafka integration for advanced pipelines.
This article explains how to construct near real‑time Elasticsearch indexes for petabyte‑level datasets by comparing MySQL limitations, describing Elasticsearch fundamentals, and detailing a pipeline that uses Hive, wide tables, MySQL binlog, Canal, and Otter to achieve second‑level index updates.
This article describes a real-time log processing platform that integrates Kafka, Flink, Drools rule engine, Redis, and Elasticsearch to unify heterogeneous log formats, extract business metrics, and provide configurable, dynamic data processing for large‑scale logging scenarios.
This article explains how to collect, process, and visualize Flink job logs on Kubernetes using an EFK stack with Fluentd, covering logging architectures, deployment of Elasticsearch, Kibana, and Fluentd, and the backend logic for querying and displaying logs in a feature platform.
This case study details how JD’s "Browse" UGC project evolved from rapid agile delivery to a performance bottleneck as data grew, and how introducing Elasticsearch, redesigning the query flow, and refactoring storage components restored fast, flexible searches for both front‑end and operations users.
AWS introduced OpenSearch, an Apache‑licensed, fully open‑source fork of Elasticsearch and Kibana that removes commercial restrictions, offers advanced features like security and machine‑learning, and promises seamless migration for existing users while gaining broad industry support.
This tutorial demonstrates how to integrate Elasticsearch into a Spring Boot microservice, covering component selection, Maven configuration, client setup, index creation, data insertion, complex query execution, synchronization of question data, and front‑end search handling, all illustrated with complete Java code examples.
This tutorial explains how to set up a Spring Boot project with Elasticsearch 6.2.1, configure RestHighLevelClient and RestClient beans, define ES host settings, and perform index creation, deletion, pagination, various query types, sorting, and highlighting through the Java API.
This article explains why Elasticsearch handles complex conditional queries more efficiently than MySQL by using inverted indexes, term dictionaries, skip‑list and roaring bitmap structures, while also discussing the trade‑offs such as slower write performance.
This article explains why Elasticsearch's built‑in tokenizers struggle with Chinese text, introduces the IK analyzer plugin, provides step‑by‑step Docker and file‑based installation methods, shows how to configure custom dictionaries via Nginx, and demonstrates smart and max‑word tokenization queries.
The article describes how algorithm testing teams tackled data‑quality and interface‑semantic monitoring problems by building a unified business monitoring platform that checks table, storage and service consistency, validates response semantics, and, through dashboards, alerts and correction tools, quickly identified dozens of offline and online issues, guiding future reliability enhancements.
This article shares an interview experience that compares MongoDB and MySQL, explains the advantages and disadvantages of document‑oriented and relational databases, introduces other NoSQL solutions such as Elasticsearch, and offers guidance on choosing the right storage technology based on workload characteristics.
This article explains how to configure Elasticsearch nodes for hot and cold data, assign special node attributes, adjust index templates, and use API calls to migrate data, demonstrating significant query speed improvements through real‑world performance tests.
This article examines Elasticsearch’s distributed search architecture, explains the two‑phase query and fetch process, identifies performance and relevance scoring challenges, and presents optimization techniques such as Search After with point‑in‑time, Scroll API usage, and DFS query‑then‑fetch for accurate scoring.
This article explains how a growing e‑commerce platform replaced a heavyweight MySQL intermediate table with a binlog‑driven pipeline that streams changes to Elasticsearch via Kafka, detailing the architecture, modules, customizations, monitoring, and performance results.
This article explains Filebeat's role as a lightweight log shipper, its relationship to the Beats family, internal architecture, installation steps, configuration of inputs, outputs, keystore usage, module activation, and practical examples for sending logs to Logstash or Elasticsearch.
This article presents practical best‑practice guidelines for configuring Elasticsearch in production, covering heap memory sizing, CPU considerations, shard and replica planning, hot‑warm node architecture, node role settings, common pitfalls, monitoring APIs, and troubleshooting tips.
This article presents multiple real‑world Elasticsearch deployments across Chinese tech giants, detailing how they handle massive query loads, scale clusters, integrate with Kafka and Kibana, and apply the Elastic Stack for search, analytics, and security monitoring.
This article explains the architecture and core concepts of Elasticsearch and Lucene, outlines the requirements for cross‑month and high‑speed queries on massive datasets, and provides detailed index and search performance tuning techniques—including bulk writes, shard routing, doc‑values management, and pagination strategies—to achieve sub‑second response times on billions of records.
Zhongtong’s ESPaaS platform automates deployment, unified monitoring, real‑time alerting, and diagnostic analysis for over 40 Elasticsearch clusters, leveraging custom exporters, Prometheus, Grafana, and DingTalk integrations to track resource, cluster, and node metrics, reduce noise, and prevent production incidents.
The article describes Uber’s Gairos platform—a unified real‑time data ingestion, storage, and query system built on Kafka, Elasticsearch and RT‑Gairos—detailing its architecture, Uber use cases such as dynamic pricing, the scalability and reliability challenges faced, and the suite of optimization techniques (sharding, query routing, caching, index merging, template tuning, and data‑pruning) applied to achieve low latency and high throughput.
This guide explains what Filebeat is, how it fits into the Beats ecosystem, its architecture and processing flow, installation steps, key configuration options, keystore usage, and practical examples of sending logs to Logstash or Elasticsearch, helping you set up reliable log collection on Linux.
This article explains how Elasticsearch handles data ingestion, retrieval, and full‑text search by describing the roles of coordinating, primary, and replica nodes, the refresh‑commit‑flush cycle, segment files, translog, and the Lucene‑based inverted index that powers its near‑real‑time capabilities.
This article analyzes common causes of slow MySQL queries—especially index misuse—offers practical indexing techniques, explains MDL locks and large‑table bottlenecks, and then compares ElasticSearch and HBase as complementary solutions for high‑performance search and storage.
This article explains how to optimize Elasticsearch for cross‑month and multi‑year queries on billions of records, covering Lucene fundamentals, index and search performance tweaks, configuration settings, and practical testing results to achieve sub‑second response times.
This article provides a comprehensive introduction to the ELK stack—Elasticsearch, Logstash, Kibana, and Filebeat—including its components, why it’s used for centralized log management, detailed architecture diagrams, step‑by‑step installation commands, configuration examples, and a practical Kafka‑based data pipeline demonstration.
This article recounts how Shay Banon’s early cooking‑app project led to the creation of Compass, the evolution into Elasticsearch, and the pivotal role of Apache Lucene, distributed testing, and visionary design in building today’s powerful search platform.
This article explores why MySQL queries become slow, delves into index pitfalls and optimization techniques, then compares ElasticSearch and HBase architectures, offering practical guidance on when to use each technology and how to combine them for high‑performance data retrieval.
This article explains how to design and implement a flexible, accurate data reporting platform using Elasticsearch aggregation, covering architecture, data synchronization, metric definition, SDK integration, and future scaling considerations for enterprise analytics.
This guide provides a comprehensive walkthrough of Kibana, covering installation, connection to Elasticsearch, data discovery, query syntax, visualizations, dashboards, and monitoring, with step‑by‑step instructions and screenshots to help users effectively explore and visualize their Elasticsearch data.
This article recounts how Shay Banon's early cooking‑app project led to the creation of Compass, the evolution of Apache Lucene, and ultimately the development of Elasticsearch—a powerful, distributed search platform built with extensive testing infrastructure and inspired by futuristic data‑interaction concepts.
This article explains Elasticsearch's internal mechanisms for writing, reading, and searching data, covering the roles of coordinating nodes, primary and replica shards, buffers, translog, segment files, refresh cycles, commit and flush operations, as well as Lucene's inverted index and how deletions and updates are handled.
The article details Elastic's shift of Elasticsearch and Kibana from Apache 2.0 to SSPL/Elastic License, the ensuing dispute with AWS over alleged coercion and trademark issues, AWS's plan to fork the projects under Apache 2.0, and the broader community backlash against the licensing change.
The article compares three approaches—synchronous dual write, asynchronous dual write with a message queue, and CDC via Canal—to keep Elasticsearch and a relational database consistent for cross‑database aggregated full‑text search, outlining their steps, advantages, and drawbacks.
The article explains Elastic's switch to SSPL/Elastic License for Elasticsearch and Kibana, Logz.io's criticism, AWS's fork under Apache 2.0, and the broader open‑source licensing battle between cloud providers and the original vendor.
The article examines how cloud service providers exploit open‑source software, why Elastic changed Elasticsearch and Kibana’s license to SSPL/Elastic License to curb this practice, and how similar moves by MongoDB, Redis and others reflect a broader shift in open‑source business models.
This article explains Ctrip's design and implementation of a flexible Elasticsearch data‑synchronization framework that handles full, incremental, ID‑based, and time‑based syncing from multiple sources, addressing the shortcomings of existing tools and simplifying complex data assembly for business search.
This article describes a high‑throughput solution for calculating hourly feature coverage, positive‑sample ratio and negative‑sample ratio on billions of records by streaming data with Spark, indexing per experiment‑hour in Elasticsearch, and executing parallel aggregation tasks with Java code.
This article summarizes a year‑long performance tuning of an ELK logging system, analyzing bottlenecks such as write thread pool saturation, JVM heap and GC settings, refresh intervals, translog durability, merge threads, shard and replica counts, and provides concrete configuration changes that reduced latency, eliminated data loss, and stabilized node resource usage.
This guide consolidates everyday Elasticsearch tuning techniques—covering configuration file tweaks, system‑level settings, and usage‑level optimizations such as memory locking, discovery settings, fault detection, queue sizing, translog handling, bulk indexing, shard management, and disk I/O—to help you build a stable, high‑throughput search cluster.
This article explains how to design, tune, and optimize an Elasticsearch‑based data platform handling hundreds of billions of records, covering Lucene fundamentals, shard routing, index and query performance tricks, and practical benchmark results for large‑scale deployments.
This article walks through the concepts, architecture, download, installation, agent setup, service startup, and alert configuration of Apache SkyWalking, an open‑source APM platform for cloud‑native microservices, including Elasticsearch integration and DingTalk notifications.
This article introduces Elasticsearch’s core concepts—including clusters, node roles, documents, mappings, and shards—and walks through the complete indexing workflow from client request to replica synchronization, highlighting key settings, routing calculations, and the role of refresh and flush operations.
This guide explains what Cerebro is, how to install it (including binary and Docker options), how to run it on Linux, macOS, and Windows, and how to use its UI to connect to an Elasticsearch node, view cluster overviews, manage shards, and execute DSL queries.
This article details how Vivo transformed its monolithic e‑commerce order service into a scalable, service‑oriented system by applying data archiving, sharding‑JDBC based sharding, MySQL‑to‑Elasticsearch sync, zero‑downtime migration, and distributed‑transaction techniques, while sharing practical pitfalls and performance results.
ebatis is an Elasticsearch ORM framework that simplifies query building by providing a MyBatis-like interface, eliminating the need for manual DSL construction and enabling seamless version upgrades.
The guide explains how to configure Tencent Cloud Elasticsearch clusters for multi‑AZ high availability by using zone‑aware node attributes, deploying data nodes in multiples of AZs, assigning three dedicated masters across zones, setting replica shards and force‑awareness rules, and safely upgrading single‑AZ clusters without service interruption.
This article explains the different index mechanisms used by MySQL (MyISAM and InnoDB) and Lucene, compares them with Elasticsearch's inverted index, and discusses how these structures affect storage, memory usage, and query performance.
This step‑by‑step guide shows how to deploy a production‑grade log collection pipeline by installing Filebeat on log sources, forwarding logs to a hardened Kafka cluster, processing them with Logstash, storing them in an Elasticsearch 7.2.0 cluster secured with X‑Pack, and visualizing the data in Kibana, all on CentOS 7.6 with detailed security hardening, configuration scripts, and verification commands.
This article explains how Alibaba Cloud’s intelligent syntax service leverages LSIF, Elasticsearch, and distributed scheduling to provide fast, cloud‑backed code navigation—eliminating the need for local clones and dramatically improving code review speed, accuracy, and resource efficiency.
This article explains Elasticsearch's data ingestion, retrieval, and search workflows, details the underlying indexing mechanisms, and provides comprehensive system‑level, shard‑level, and query‑level tuning recommendations—including configuration snippets and best‑practice strategies for high‑throughput and low‑latency deployments.
The article explains how to improve query performance by flattening multi‑table MySQL data and synchronizing it to Elasticsearch—using DataX for one‑time bulk loading and Canal (with Canal‑Adapter) for real‑time binlog‑driven incremental updates—while detailing configuration steps, job examples, and common pitfalls.
This tutorial walks through installing and configuring Apache SkyWalking, an open‑source APM system for micro‑services and cloud‑native environments, covering its architecture, Elasticsearch and InfluxDB storage setup, agent deployment, service startup, alarm integration, and essential documentation links.
