BestHub
Sign in
Tagged articles
251 articles
Page 3 of 3
DataFunTalk
DataFunTalk
Mar 28, 2020 · Big Data

Applying Flink State Management for Real-Time Recommendation Scenarios

This article explains how Apache Flink's flexible state management can be leveraged to solve data correlation challenges in real‑time recommendation platforms, compares Flink with Spark and Storm, describes the underlying broadcast and managed state mechanisms, and provides a step‑by‑step implementation using Kafka, Druid, and custom broadcast functions.

Big DataFlinkStreaming
0 likes · 14 min read
Applying Flink State Management for Real-Time Recommendation Scenarios
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 19, 2020 · Big Data

Can Flink Unify Real‑Time and Offline Data Warehouses? A Deep Dive

This article examines the challenges of maintaining separate offline and real‑time data warehouses, explains the three‑layer ODS‑DW‑ADS model, evaluates the traditional Lambda architecture, and explores how a unified Flink stack with Kafka, HiveCatalog and streaming sinks can simplify metadata, SQL development, data import/export, and stateful processing for both batch and streaming workloads.

Data WarehouseFlinkLambda architecture
0 likes · 12 min read
Can Flink Unify Real‑Time and Offline Data Warehouses? A Deep Dive
Huajiao Technology
Huajiao Technology
Dec 24, 2019 · Frontend Development

How Huajiao Enables Server‑Driven Dynamic Voice‑Chat Layouts with Zero Client Code

The article explains Huajiao's server‑and‑H5 driven dynamic layout architecture for multi‑user voice rooms, detailing its design principles, benefits, three core concepts, synchronization mechanisms, interaction flow, and how it drastically reduces client development effort while supporting flexible, real‑time UI changes.

FrontendH5dynamic layout
0 likes · 9 min read
How Huajiao Enables Server‑Driven Dynamic Voice‑Chat Layouts with Zero Client Code
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 30, 2019 · Big Data

Building a Real‑Time Data Processing Pipeline with Apache Kafka, Spark Streaming, and Cassandra

This tutorial explains how to create a highly scalable, fault‑tolerant real‑time data processing platform by configuring a Kafka topic, a Cassandra keyspace, adding Spark and connector dependencies, developing a Java‑based Spark Streaming pipeline, enabling checkpoints, and deploying the application with spark‑submit.

Big DataJavaKafka
0 likes · 8 min read
Building a Real‑Time Data Processing Pipeline with Apache Kafka, Spark Streaming, and Cassandra
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 21, 2019 · Cloud Computing

How Alibaba’s Cloud‑Powered Wristband Turned 60,000 Attendees into a Live LED Screen

Alibaba’s 20‑year anniversary event showcased a cloud‑enabled IoT wristband equipped with 102 full‑color LEDs, real‑time sensors and a high‑density wireless network that synchronized over 60,000 devices, delivering dynamic lighting, personalized guidance, interactive games, and large‑scale live video, illustrating the challenges and solutions of massive real‑time IoT deployments.

IoTLEDcloud
0 likes · 13 min read
How Alibaba’s Cloud‑Powered Wristband Turned 60,000 Attendees into a Live LED Screen
dbaplus Community
dbaplus Community
Oct 15, 2019 · Big Data

How to Build Real‑Time Data Pipelines for E‑Commerce Promotions

This article examines the surge in real‑time data demands for e‑commerce promotions, outlines how to collect, compute, and deliver streaming data, compares batch and stream processing, lists typical use cases, and discusses the challenges of building scalable, low‑latency pipelines.

Data StreamingMonitoringreal-time
0 likes · 11 min read
How to Build Real‑Time Data Pipelines for E‑Commerce Promotions
DataFunTalk
DataFunTalk
Aug 8, 2019 · Artificial Intelligence

Practical Experience of JD E‑commerce Recommendation System: Architecture, Ranking, Real‑time Updates, and Experiment Platform

This article shares JD's e‑commerce recommendation system practice, covering the overall online/offline architecture, recall and ranking modules, real‑time feature and model updates, multi‑objective and diversity strategies, first‑stage index‑based ranking, KNN recall, and a layered experiment platform for rapid iteration.

Learning-to-RankMachine Learninge-commerce
0 likes · 14 min read
Practical Experience of JD E‑commerce Recommendation System: Architecture, Ranking, Real‑time Updates, and Experiment Platform
Big Data Technology Architecture
Big Data Technology Architecture
Aug 7, 2019 · Big Data

Why Choose Apache Flink for Real‑Time Stream Processing: Features and Lessons Learned

This article explains why the author chose Apache Flink for real‑time stream processing, highlighting its unique combination of high throughput, low latency, event‑time support, stateful computation, flexible windows, and fault tolerance, while also reflecting on the challenges of adopting a less‑documented technology.

Event TimeFlinkStateful Computation
0 likes · 7 min read
Why Choose Apache Flink for Real‑Time Stream Processing: Features and Lessons Learned
Xianyu Technology
Xianyu Technology
Jun 14, 2019 · Big Data

Xianyu IFTTT: Scalable Real-Time User Relationship Platform

Xianyu IFTTT is a scalable real-time user-relationship platform that enriches metadata, enables bidirectional buyer-seller interactions, integrates quickly via SLS logs, uses a chain-of-responsibility for customizable lists, processes push actions with fatigue filtering, and stores TB-scale data in Lindorm, delivering billions of daily records and more than double the click-through rate of offline pushes.

Big DataIFTTTUser Relationship
0 likes · 9 min read
Xianyu IFTTT: Scalable Real-Time User Relationship Platform
ITPUB
ITPUB
Apr 4, 2019 · Big Data

Achieving Sub‑Second Real‑Time Product Selection with Xianyu’s Mach and Blink

Xianyu’s Mach system tackles the e‑commerce challenge of instantly selecting high‑quality items from billions of products by leveraging Blink’s low‑latency stream computing, detailing its architecture—including state, windows, custom UDX functions, data merging, rule execution, and SQL‑to‑MVEL conversion—to achieve sub‑second processing at massive scale.

FlinkRule Engineblink
0 likes · 18 min read
Achieving Sub‑Second Real‑Time Product Selection with Xianyu’s Mach and Blink
Meituan Technology Team
Meituan Technology Team
Nov 1, 2018 · Operations

An Overview of Meituan-Dianping's Open‑Source Distributed Monitoring System CAT

Meituan‑Dianping’s open‑source CAT is a high‑availability, real‑time distributed monitoring platform that now supports Java, C/C++, Python, Node.js and Go, processes billions of messages daily, offers scalable architecture, binary protocol, sampling aggregation, and is expanding its UI, stack and community.

Distributed MonitoringJavaMulti-language
0 likes · 10 min read
An Overview of Meituan-Dianping's Open‑Source Distributed Monitoring System CAT
21CTO
21CTO
Sep 2, 2018 · Operations

How We Built a Real‑Time Log Collection System with Go, Kafka, and Etcd

This article examines the shortcomings of a legacy log‑collection setup, introduces a streamlined real‑time architecture that eliminates per‑machine Hadoop clients, centralizes configuration via Etcd with a web UI, and details the Go implementation—including configuration design, Etcd watching, Kafka integration, tail management, and per‑second rate limiting—complete with code snippets and performance graphs.

System Architectureetcdlog collection
0 likes · 16 min read
How We Built a Real‑Time Log Collection System with Go, Kafka, and Etcd
ITPUB
ITPUB
Aug 3, 2018 · Operations

How to Monitor Log Files in Real-Time with Python: 3 Simple Methods

When high service reliability demands immediate detection of slow requests, this guide shows three Python techniques—using tail via subprocess, file.tell/seek loops, and a generator with yield—to continuously watch log files and trigger alerts as soon as specified patterns appear.

Log MonitoringOperationsPython
0 likes · 4 min read
How to Monitor Log Files in Real-Time with Python: 3 Simple Methods
DataFunTalk
DataFunTalk
Jun 13, 2018 · Artificial Intelligence

Evolution of E‑commerce Platform Recommendation System Architecture

This article reviews the evolution of recommendation system architecture for C2C e‑commerce platforms, tracing stages from simple offline‑online pipelines through granular feed‑flow improvements, real‑time processing, and machine‑learning‑driven models, while highlighting user‑profile construction, challenges, and best‑practice guidelines.

Machine Learningarchitecturee-commerce
0 likes · 10 min read
Evolution of E‑commerce Platform Recommendation System Architecture
JD Tech
JD Tech
Apr 17, 2018 · Operations

Overwatch: A Distributed Real‑Time RPC Monitoring Platform for System Observability

The article describes Overwatch, a distributed monitoring system developed by Dada‑JD Daojia that collects, aggregates, and visualizes RPC traffic in real time using consumer‑side agents, Kafka, Storm, and a Node.js CQRS architecture, enabling engineers to quickly locate and resolve service failures.

CQRSDistributed MonitoringRPC
0 likes · 8 min read
Overwatch: A Distributed Real‑Time RPC Monitoring Platform for System Observability
Efficient Ops
Efficient Ops
Feb 5, 2018 · Operations

How WeChat Scales Massive Real-Time Monitoring: Design & Practices

This article details the architecture and practical techniques behind WeChat's large‑scale monitoring system, covering lightweight data collection, classification of real‑time, non‑real‑time and user‑specific metrics, anomaly detection algorithms, automated configuration, and high‑performance storage solutions for billions of events per minute.

MonitoringOperationsdata collection
0 likes · 14 min read
How WeChat Scales Massive Real-Time Monitoring: Design & Practices
Dada Group Technology
Dada Group Technology
Sep 29, 2017 · Operations

Overwatch: A Distributed System Monitoring Platform for Real‑Time RPC Visibility

Overwatch is an open‑source distributed monitoring platform built by Dada‑Jingdong Home that collects, aggregates, and visualizes RPC traffic across thousands of micro‑services in real time, enabling engineers to quickly pinpoint the root cause of system failures using directed‑graph visualizations and CQRS‑based data queries.

CQRSKafkaMonitoring
0 likes · 10 min read
Overwatch: A Distributed System Monitoring Platform for Real‑Time RPC Visibility
21CTO
21CTO
Sep 15, 2017 · Artificial Intelligence

Mastering Recommendation Systems: Goals, Algorithms, and Real-World Practices

This article explains the objectives of recommendation systems, outlines four recommendation approaches, dives into personalized recommendation architecture and core algorithms, and discusses practical challenges such as real‑time processing, cold‑start, diversity, content quality, and exploration‑exploitation trade‑offs.

Recommendation Systemscold startcollaborative filtering
0 likes · 16 min read
Mastering Recommendation Systems: Goals, Algorithms, and Real-World Practices
Architecture Digest
Architecture Digest
Sep 15, 2017 · Artificial Intelligence

Overview of Recommendation Systems: Goals, Methods, Architecture, and Practical Considerations

This article explains the objectives of recommendation systems, compares popular recommendation approaches, details the components and algorithms of personalized recommendation pipelines, and discusses practical challenges such as real‑time processing, freshness, cold‑start, diversity, content quality, and surprise handling.

Machine Learningcold startdata pipeline
0 likes · 15 min read
Overview of Recommendation Systems: Goals, Methods, Architecture, and Practical Considerations
21CTO
21CTO
Jul 8, 2017 · Big Data

Ctrip’s Scalable Real‑Time User Behavior System with Kafka, Storm, Redis

This article details Ctrip’s redesign of its real‑time user behavior service, covering the new architecture, data flow, use of Java, Kafka, Storm, Redis, and MySQL, and how it achieves high real‑time performance, availability, scalability, and fault‑tolerance to support massive travel‑industry traffic.

KafkaRedisStorm
0 likes · 12 min read
Ctrip’s Scalable Real‑Time User Behavior System with Kafka, Storm, Redis
System Architect Go
System Architect Go
Jun 17, 2017 · Frontend Development

Server-Sent Events (SSE): Implementing Lightweight Real-Time Server Push in Web Applications

Server-Sent Events (SSE) provide a lightweight, HTTP‑based mechanism for real‑time server push, and this article explains its principles, compatibility, and step‑by‑step implementation—including a simple server that streams timestamps every second and a front‑end page that displays the incoming data.

HTTPSSEServer-Sent Events
0 likes · 3 min read
Server-Sent Events (SSE): Implementing Lightweight Real-Time Server Push in Web Applications
Architecture Digest
Architecture Digest
May 18, 2017 · Backend Development

Design and Architecture of Ctrip's Real‑Time User Behavior Service

The article describes how Ctrip rebuilt its real‑time user behavior platform using a Java‑based stack (Kafka, Storm, Redis, MySQL) to achieve millisecond‑level latency, high availability, scalable performance, and robust handling of traffic spikes, failures, and data back‑pressure.

Backend ArchitectureKafkaRedis
0 likes · 12 min read
Design and Architecture of Ctrip's Real‑Time User Behavior Service
Tongcheng Travel Technology Center
Tongcheng Travel Technology Center
Apr 10, 2017 · Operations

Sentinel Monitoring System: Real‑Time Business Log Monitoring and Incident Detection for an Airline Ticket Platform

The Sentinel system was built to provide real‑time, zero‑modification monitoring of airline ticket business services by consuming Tianwang logs through a Storm cluster, offering flexible rule configuration, addressing performance pitfalls, and planning future enhancements such as custom monitoring scripts and visual dashboards.

KafkaLog ProcessingMonitoring
0 likes · 6 min read
Sentinel Monitoring System: Real‑Time Business Log Monitoring and Incident Detection for an Airline Ticket Platform
21CTO
21CTO
Jan 18, 2017 · Big Data

Build a Lightweight, High‑Availability Real‑Time Stream Processing System

Learn how to construct a simple, high‑availability real‑time stream processing platform using lightweight components such as Kafka, Zookeeper, Thrift/Avro, and optional storage like MongoDB or Elasticsearch, offering a practical alternative to heavyweight frameworks like Storm and Spark Streaming for small‑to‑medium enterprises.

Big DataKafkalightweight architecture
0 likes · 5 min read
Build a Lightweight, High‑Availability Real‑Time Stream Processing System
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 26, 2016 · Operations

How Alibaba’s SunFire Powers Real‑Time Monitoring for Billion‑Scale Transactions

Alibaba’s SunFire platform delivers massive‑scale, real‑time log collection, processing, and visualization for e‑commerce spikes like Double 11, using low‑overhead agents, asynchronous Map/Reduce pipelines, fault‑tolerant task scheduling, and shared inputs to ensure accurate, low‑latency monitoring across billions of transactions.

AlibabaMonitoringOperations
0 likes · 18 min read
How Alibaba’s SunFire Powers Real‑Time Monitoring for Billion‑Scale Transactions
Qunar Tech Salon
Qunar Tech Salon
Nov 1, 2016 · Frontend Development

QReport vs Beacon: A Frontend‑Centric Real‑Time Data Reporting Solution

The article introduces QReport, a front‑end‑focused real‑time data reporting system developed to overcome the limitations of the Beacon log‑analysis platform, detailing its architecture, page‑identification method, flow analysis, real‑time capabilities, richer metrics, and overall advantages for web analytics.

Data AnalyticsFrontendQReport
0 likes · 8 min read
QReport vs Beacon: A Frontend‑Centric Real‑Time Data Reporting Solution
Architecture Digest
Architecture Digest
Oct 30, 2016 · Frontend Development

Challenges and Solutions in H5 Live Streaming Development

The article outlines the end‑to‑end workflow of H5 live streaming, discusses state control, WebSocket‑based comment handling, heartbeat reconnection, video compatibility issues, and practical code techniques for detecting and recovering from stream stalls.

H5live streamingreal-time
0 likes · 9 min read
Challenges and Solutions in H5 Live Streaming Development
Meituan Technology Team
Meituan Technology Team
Oct 28, 2016 · Big Data

Design and Architecture of the CAT Real-Time Monitoring System

The CAT real‑time monitoring system, open‑sourced in 2014 for Java applications, combines a lightweight ThreadLocal‑based client SDK, Netty‑driven asynchronous transport, and a highly scalable backend that processes ~100 TB of logs daily across 70 machines, using custom binary serialization, in‑memory modeling, segmented storage with 48‑bit indexing, and hourly aggregation to provide near‑full‑volume fault detection, localization, and performance analysis.

Distributed SystemsJavaMonitoring
0 likes · 18 min read
Design and Architecture of the CAT Real-Time Monitoring System
Architecture Digest
Architecture Digest
Sep 12, 2016 · Artificial Intelligence

Design and Implementation of a Real‑Time, Highly Available General Recommendation Platform at YHD

The article describes how YHD's precision recommendation team built a real‑time, highly available, traceable general recommendation platform, detailing its background, overall architecture, visual configuration and traceability subsystems, and reporting significant improvements in development speed, reuse and user satisfaction.

AIHBaseKafka
0 likes · 8 min read
Design and Implementation of a Real‑Time, Highly Available General Recommendation Platform at YHD
Big Data and Microservices
Big Data and Microservices
Apr 19, 2016 · Industry Insights

Designing a Scalable Real‑Time Stock Prediction Architecture with Open‑Source Tools

This article outlines a reference architecture for a low‑latency, horizontally scalable real‑time stock prediction system built with open‑source components such as Spring Cloud Data Flow, Apache Geode, Spark MLlib, and Hadoop, and discusses data flow steps, simplified deployment, and algorithm choices for market forecasting.

Big DataMachine Learningopen-source architecture
0 likes · 7 min read
Designing a Scalable Real‑Time Stock Prediction Architecture with Open‑Source Tools
21CTO
21CTO
Mar 13, 2016 · Backend Development

How 58’s Messaging Platform Solved High‑Load Real‑Time Delivery Challenges

This article details the design and implementation of 58’s real‑time messaging platform, covering its business background, overall architecture, protocol design for extensibility and debugging, and solutions for fast TCP reconnection to ensure stable, high‑throughput delivery for intra‑city freight services.

MessagingScalabilityTCP
0 likes · 9 min read
How 58’s Messaging Platform Solved High‑Load Real‑Time Delivery Challenges
21CTO
21CTO
Jan 9, 2016 · Big Data

How We Scaled Real‑Time Log Analysis to 2 TB Daily with ELK

This article shares the author's practical experience building a real‑time log analysis platform at Sina, covering service scope, ELK architecture, performance optimizations, usability improvements, new features, common pitfalls, and a concise Q&A for engineers handling massive log streams.

ELKElasticsearchKafka
0 likes · 12 min read
How We Scaled Real‑Time Log Analysis to 2 TB Daily with ELK
Architect
Architect
Dec 26, 2015 · Artificial Intelligence

Open-Source Reference Architecture for Real-Time Stock Prediction

The article presents an open‑source, highly scalable reference architecture that combines real‑time data ingestion, machine‑learning model training, and low‑latency prediction using components such as Spring Cloud Data Flow, Apache Geode, Spark MLlib, and Hadoop to enable continuous stock price forecasting.

Apache GeodeSpring Cloud Data Flowreal-time
0 likes · 7 min read
Open-Source Reference Architecture for Real-Time Stock Prediction
ITPUB
ITPUB
Dec 11, 2015 · Backend Development

Inside Meizu’s Real‑Time Push System: Architecture, Challenges & Solutions

This article presents a detailed walkthrough of Meizu’s real‑time push platform, covering its four‑layer architecture, high‑concurrency design, micro‑service RPC framework, power‑saving strategies, duplicate‑message handling, DNS reliability, load‑balancing tactics, monitoring setup, and gray‑release deployment.

Gray ReleaseMicroservicesMonitoring
0 likes · 11 min read
Inside Meizu’s Real‑Time Push System: Architecture, Challenges & Solutions
Java High-Performance Architecture
Java High-Performance Architecture
Dec 3, 2015 · Backend Development

Unlock Real-Time Messaging: How Redis Pub/Sub Works and When to Use It

The article explains the publish‑subscribe (pub/sub) messaging model, its time, space, and synchronization decoupling features, typical real‑time scenarios such as chat and log processing, and details how Redis implements pub/sub through channels and pattern subscriptions, including command syntax and internal data structures.

AsynchronousMessagingRedis
0 likes · 4 min read
Unlock Real-Time Messaging: How Redis Pub/Sub Works and When to Use It
Architect
Architect
Nov 12, 2015 · Backend Development

Web Real‑Time Message Push: Techniques and Choosing Comet with Tornado

This article reviews various web real‑time message push techniques—including polling, long‑polling, iframe streaming, socket plugins, and WebSocket—evaluates their pros and cons, and explains why the author selected the Comet approach implemented with the Python Tornado framework for a web push system.

backendreal-timetornado
0 likes · 4 min read
Web Real‑Time Message Push: Techniques and Choosing Comet with Tornado