log collection | BestHub

FunTester

May 28, 2025 · Cloud Native

Extending Automated Thread Dumps: Log Collection, Resource Monitoring, Chaos Engineering, Performance Analysis, and Environment Cleanup

The article explores how automated thread dumps can be expanded into multiple testing scenarios—including log collection, resource monitoring, fault injection, performance result analysis, and environment cleanup—by leveraging Kubernetes APIs, Prometheus, Chaos Mesh, and scripting tools to improve efficiency, observability, and system resilience.

AutomationChaos EngineeringKubernetes

0 likes · 9 min read

Extending Automated Thread Dumps: Log Collection, Resource Monitoring, Chaos Engineering, Performance Analysis, and Environment Cleanup

Didi Tech

Mar 27, 2025 · Operations

Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection

Didi replaced its legacy agent with Alibaba’s open‑source iLogTail, re‑architected it to use a shared thread‑pool and SIMD‑accelerated parsing, rewrote critical plugins in C++ and added robust Kafka retry logic, achieving over twice the throughput while cutting CPU usage by more than half and maintaining near‑zero latency at massive scale.

C++KafkaPerformance Optimization

0 likes · 10 min read

Performance Optimization and Architecture of iLogTail for High‑Scale Log Collection

Sanyou's Java Diary

Mar 6, 2025 · Backend Development

8 Real-World MQ Use Cases Every Backend Engineer Should Know

This article explores eight practical scenarios for using message queues—such as asynchronous processing, service decoupling, traffic shaping, delayed tasks, log collection, distributed transactions, remote calls, and broadcast notifications—providing code examples with RabbitMQ, RocketMQ, and Kafka to illustrate each pattern.

Broadcast NotificationsMessage QueueService Decoupling

0 likes · 15 min read

8 Real-World MQ Use Cases Every Backend Engineer Should Know

Linux Ops Smart Journey

Feb 18, 2025 · Cloud Native

Deploy Filebeat with Helm on Kubernetes: Automated Log Collection to Kafka

This step‑by‑step guide shows how to use a Helm chart to deploy Filebeat in a Kubernetes cluster, automatically collect container logs, and forward them to a Kafka cluster for reliable, scalable observability.

Cloud NativeFilebeatHelm

0 likes · 7 min read

Deploy Filebeat with Helm on Kubernetes: Automated Log Collection to Kafka

Top Architect

Nov 4, 2024 · Operations

Centralized Log Collection and Monitoring with Filebeat and Graylog

This article explains how to use Filebeat as a lightweight log shipper together with Graylog for centralized log aggregation, storage, and analysis, covering installation, configuration files, Docker deployment, Spring Boot integration, and query techniques for efficient operations monitoring.

DockerElasticsearchFilebeat

0 likes · 18 min read

Centralized Log Collection and Monitoring with Filebeat and Graylog

Architecture Digest

Jun 19, 2024 · Operations

Log Collection Solution: Filebeat + Graylog – Configuration and Deployment Guide

This article provides a comprehensive guide on building a unified log collection system using Filebeat and Graylog, covering the tools' concepts, configuration files, component functions, Docker deployment, and practical code examples for efficient log monitoring across multiple environments.

DockerELKFilebeat

0 likes · 14 min read

Log Collection Solution: Filebeat + Graylog – Configuration and Deployment Guide

Architect

Feb 28, 2024 · Cloud Native

Lightweight Kubernetes Log Collection with Loki: Deployment and Configuration Guide

This article provides a comprehensive, lightweight solution for collecting Kubernetes logs using Grafana Loki, covering its advantages, component comparison, deployment modes (All‑In‑One, microservices, bare‑metal), required configuration files, ConfigMap and PersistentVolume setup, Promtail installation, Helm deployment, and common troubleshooting steps.

HelmKubernetesLoki

0 likes · 20 min read

Lightweight Kubernetes Log Collection with Loki: Deployment and Configuration Guide

Code Ape Tech Column

Dec 12, 2023 · Operations

Centralized Log Collection with Filebeat and Graylog

This article explains how to use Filebeat together with Graylog to collect, ship, store, and analyze logs from multiple environments, covering tool introductions, configuration files, Docker deployment, Spring Boot integration, and practical search syntax for effective log monitoring.

DockerElasticsearchFilebeat

0 likes · 20 min read

Centralized Log Collection with Filebeat and Graylog

DataFunTalk

Jul 9, 2023 · Operations

Building High‑Performance Observability Data Pipelines with Vector and Honghu

This article explains the concepts and importance of observability, introduces the Vector data‑pipeline tool and its architecture, demonstrates how to configure sources, transforms and sinks, and shows how to integrate Vector with the Honghu platform to build a complete, real‑time monitoring solution for modern distributed systems.

Big DataHonghuMonitoring

0 likes · 33 min read

Building High‑Performance Observability Data Pipelines with Vector and Honghu

Zhuanzhuan Tech

Jun 21, 2023 · Operations

Rapid Issue Localization and Alerting for B2C Backend Using Custom Log Agent and Prometheus

This article describes how the ZhiZhuan B2C backend team built a standardized logging, custom Apollo‑based log agent, Prometheus‑driven alerting service, and first‑responsible‑person mechanism to quickly locate and resolve service timeouts, exceptions, and other production issues, even when working off‑site.

AlertingJava agentMonitoring

0 likes · 10 min read

Rapid Issue Localization and Alerting for B2C Backend Using Custom Log Agent and Prometheus

Ops Development Stories

Apr 25, 2023 · Operations

Simplify Monitoring with Categraf: All‑in‑One Agent for Metrics, Logs, and Traces

Categraf is an all‑in‑one, Go‑based monitoring agent that consolidates metric, log, and trace collection, offering remote_write support, lightweight deployment, and extensive plugin configurations to replace multiple exporters in Prometheus‑based observability stacks.

AgentCategrafConfiguration

0 likes · 14 min read

Simplify Monitoring with Categraf: All‑in‑One Agent for Metrics, Logs, and Traces

Efficient Ops

Apr 17, 2023 · Operations

Mastering Container Log Collection in Kubernetes: Strategies and Best Practices

This article explains how container log collection in Kubernetes differs from traditional host logging, outlines common deployment methods such as DaemonSet and Sidecar, compares log storage options, and offers practical guidance on handling stdout and file‑based logs for reliable operations.

Container LoggingDaemonSetKubernetes

0 likes · 12 min read

Mastering Container Log Collection in Kubernetes: Strategies and Best Practices

Zhuanzhuan Tech

Feb 1, 2023 · Cloud Native

Evolution of Log Collection at ZuanZuan: From Bare Metal to Cloud‑Native Era

This article traces ZuanZuan's log‑collection journey from the bare‑metal era through containerization to a cloud‑native solution, detailing the challenges, custom developments like log‑pilot, ByteCompass, and fb‑advisor, and the performance gains achieved with each transition.

Cloud NativeContainerizationFilebeat

0 likes · 12 min read

Evolution of Log Collection at ZuanZuan: From Bare Metal to Cloud‑Native Era

Efficient Ops

Jan 17, 2023 · Operations

Mastering Observability Data Collection in Kubernetes with iLogtail

This article explains the types and value of observability data, the characteristics and data‑collection requirements of Kubernetes deployments, the challenges of large‑scale log, metric and trace ingestion, and practical solutions such as DaemonSet vs Sidecar, ConfigServer, CRD automation, serverless support, and eBPF integration.

Cloud NativeKubernetesObservability

0 likes · 20 min read

Mastering Observability Data Collection in Kubernetes with iLogtail

NetEase Yanxuan Technology Product Team

Jan 9, 2023 · Operations

Loggie: A High-Performance Log Collection Agent System Design and Implementation

Loggie is a cloud-native, Go-based log-collection agent that replaces Filebeat and Flume by using a micro-kernel producer-consumer architecture with hot-swappable pipelines, achieving 2 GB/s read speeds, 1.6‑2.6× higher throughput while using only a quarter of the CPU, and providing built-in observability, reliability, and latency monitoring for large-scale enterprise deployments.

GoMonitoringObservability

0 likes · 16 min read

Loggie: A High-Performance Log Collection Agent System Design and Implementation

Architecture Digest

Dec 2, 2022 · Big Data

Design and Implementation of Vivo's Bees Log Collection Agent

This article presents the design principles, core techniques, and practical solutions of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, checkpointing, resource control, platform management, and a comparison with open‑source alternatives.

Agent DesignBig DataJava

0 likes · 25 min read

Design and Implementation of Vivo's Bees Log Collection Agent

High Availability Architecture

Nov 30, 2022 · Big Data

Design and Implementation of Vivo's Bees Log Collection Agent

This article presents the design principles, core features, and implementation details of Vivo's self‑developed Bees log collection agent, covering file discovery, unique identification, real‑time and offline ingestion, resource control, platform management, and comparisons with open‑source solutions.

Big DataHDFSJava

0 likes · 22 min read

vivo Internet Technology

Nov 23, 2022 · Big Data

Design and Implementation of Vivo's Bees Log Collection Agent

Vivo’s Bees‑agent is a custom, lightweight log‑collection service that discovers rotating files via inotify, uniquely identifies them with inode and hash signatures, supports real‑time and offline ingestion to Kafka and HDFS, offers checkpoint‑resume, resource isolation, rich metrics, and a centralized management platform, outperforming open‑source collectors in latency, memory usage, and scalability.

Agent DesignBig DataHDFS

0 likes · 24 min read

NetEase Yanxuan Technology Product Team

Oct 17, 2022 · Big Data

Design and Evolution of a Unified Log Platform

NetEase Yanxuan built a unified log platform that consolidates collection, delivery, processing, storage, analysis, and alerting for near‑real‑time and offline logs, using Flume, a Golang‑based Loggie agent, Kafka, Flink, HBase and Elasticsearch to ensure high performance, data‑quality, container‑native isolation, full‑link traceability, and automated scaling, with Loggie now open‑sourced.

Big DataCloud NativeLog Management

0 likes · 38 min read

Design and Evolution of a Unified Log Platform

Wukong Talks Architecture

Aug 24, 2022 · Cloud Native

Overview of ZuanZuan Cloud Platform: Architecture, Image Management, Release Upgrade, Container Monitoring, and Log Collection

The article introduces the ZuanZuan cloud platform, detailing its overall architecture, image management workflow that abstracts Dockerfiles, release‑upgrade strategies with custom controllers, container monitoring evolution to Prometheus, and log‑collection mechanisms that handle large Java‑based log volumes.

Cloud NativeContainerDeployment

0 likes · 8 min read

Overview of ZuanZuan Cloud Platform: Architecture, Image Management, Release Upgrade, Container Monitoring, and Log Collection