This article provides a comprehensive, step‑by‑step analysis of Linux zero‑copy mechanisms—sendfile, mmap, and splice—detailing their internal workflows, performance trade‑offs, code examples, and practical selection guidelines for high‑throughput backend development.
Linux’s Virtual Memory Area (VMA) is the kernel’s core structure for managing a process’s address space, and this article explains its definition, key fields, linked‑list and red‑black‑tree organization, allocation, protection, page‑fault handling, copy‑on‑write, and practical usage examples.
This article walks through Milvus 2.3‑2.6.x storage optimizations—Mmap, tiered storage, and clustering compaction—explaining their principles, configuration hierarchy, benchmark results, and concrete deployment templates that together can boost query performance up to 25‑fold while halving memory consumption.
The article shows how to diagnose and dramatically cut Python’s memory usage by using built‑in tools such as sys.getsizeof, psutil, __slots__, generator expressions, memory‑mapped files (mmap) and string interning, providing concrete code examples, benchmarks and practical tips to avoid common pitfalls.
This article explains the fundamentals of Linux memory mapping (mmap), covering its API, parameters, shared vs. private modes, kernel internals, page‑fault handling, performance benefits, zero‑copy I/O techniques, practical code examples, and common pitfalls for developers.
mmap, the classic Linux memory‑mapping technique, often surpasses the modern async io_uring in various I/O scenarios by eliminating redundant data copies, reducing system calls, and enabling zero‑copy access, while the article explains its fundamentals, workflow, performance comparisons, practical usage, pitfalls, and code examples.
The article explains how Direct Memory Access (DMA) eliminates CPU‑bound data copies, compares traditional I/O with zero‑copy techniques such as mmap and sendfile, and shows how reducing system calls and context switches can double file‑transfer throughput while highlighting the limits of kernel cache for large files.
This article explains the low‑level mechanisms of brk and mmap in Linux, compares their characteristics, shows why malloc selects one over the other based on allocation size, and provides practical code examples, performance tips, and common pitfalls for developers.
mmap maps a file directly into a process’s address space, allowing programmers to read and write disk data as if it were memory, eliminating extra system calls and copies, while introducing complexities like page faults, address‑space limits, and performance trade‑offs that must be evaluated per workload.
mmap maps files directly into a process’s virtual memory, eliminating the double‑copy between kernel and user space and reducing costly read/write system calls, which boosts I/O performance, simplifies code, but requires careful handling of address space limits, page faults, and concurrency.
This article explains the fundamentals of the mmap system call, its internal working mechanism, zero‑copy I/O model, advantages, typical use cases, and detailed usage guidelines—including function prototypes, parameter meanings, mapping steps, and sample code—while also exploring how mmap‑mapped files behave during program crashes.
This article explains how using mmap for shared memory in C/C++ can break process isolation, introduce permission‑related code‑injection vulnerabilities, and cause multithreading synchronization issues, making it a riskier choice than the traditional malloc allocation.
mmap shared memory lets multiple processes access the same physical memory, which can break process isolation, expose permission misconfigurations like PROT_EXEC, and cause cross‑process crashes or code‑injection attacks, making it far riskier than heap allocations with malloc that remain confined to a single process.
This comprehensive guide explores Linux's memory management ecosystem, detailing user‑space allocators like malloc, kernel‑space mechanisms such as kmalloc, vmalloc, and mmap, and core kernel algorithms including the Buddy system, CMA, slab allocator, and memory pools, while providing code examples and practical usage tips.
This article explains how mmap maps files into a process's virtual memory to eliminate double data copies and reduce system‑call overhead, offering performance gains, a simpler programming model, and discusses its limitations such as address‑space constraints and page‑fault latency.
This article explains why traditional I/O interfaces rely on data copying, demonstrates the hidden overhead of read/write in a network server, and introduces zero‑copy methods such as mmap, sendfile, DMA Gather Copy, and splice to dramatically reduce copies and context switches for faster I/O.
This article explains how the C functions malloc and free allocate and release memory, covering stack vs heap, the brk and mmap system calls, fragmentation, header metadata, and why both allocation strategies are needed for efficient memory management.
This article breaks down zero‑copy data transfer techniques—write+read, mmap+write, sendfile, sendfile + SG‑DMA, and splice—showing how they reduce CPU copies and context switches to boost I/O performance in modern operating systems.
This article explains the concept of zero‑copy in Linux, compares the four main system calls—sendfile, mmap, splice and tee—describes their APIs, internal mechanisms, performance characteristics, typical use‑cases and provides practical code examples for high‑performance network programming.
This article explains how mmap and zero‑copy mechanisms, combined with DMA and shared‑memory APIs, can dramatically reduce CPU involvement, context switches, and data copies during file and network I/O, thereby improving system performance for high‑throughput applications.
Zero‑copy eliminates CPU‑mediated data copies between user and kernel spaces by using DMA and memory‑mapping, dramatically improving I/O performance, reducing context switches, and enabling high‑throughput applications such as file servers, Kafka brokers, and Java networking frameworks.
This article explains how Linux zero‑copy techniques—DMA, sendfile, splice, mmap + write, and tee—reduce CPU involvement in large file and network transfers by moving data directly within kernel space, detailing their workflows, code examples, performance trade‑offs, and suitable use cases.
This article explains how the Linux kernel implements heap memory management, covering the heap data structure, the mm_struct fields start_brk and brk, the brk/sbrk and malloc/free allocation methods, the brk and mmap system calls, and the internal glibc structures heap_info and malloc_state that track heap state.
The article explains how Kafka’s use of sendfile zero‑copy gives it higher throughput than RocketMQ, which relies on mmap, and discusses the trade‑offs between performance and feature richness when choosing between the two message‑queue systems.
The article compares Kafka and RocketMQ, explaining how Kafka’s use of sendfile zero‑copy reduces system calls and data copies, while RocketMQ relies on mmap, leading to lower throughput but richer features, and offers guidance on choosing between them based on scenario.
This article explains the concept of zero‑copy, compares traditional I/O copying with mmap, sendfile, DMA scatter/gather and splice techniques, and shows how Java NIO leverages mmap and sendfile to achieve high‑performance data transfer while minimizing CPU involvement.
This article explains how Kafka attains extremely high throughput by using batch sending, message compression, sequential disk I/O, PageCache, zero‑copy transfers, and memory‑mapped index files, detailing the underlying code paths and trade‑offs that enable millions of messages per second.
Zero‑copy techniques such as mmap + write, sendfile, and sendfile + DMA scatter/gather reduce CPU‑memory copies and context switches compared with traditional read/write IO, improving performance for high‑throughput systems like RocketMQ and Kafka, while explaining user‑kernel mode, DMA, and their trade‑offs.
The article explains how RocketMQ’s reliance on mmap‑based zero‑copy leads to more data copies and system‑call overhead compared to Kafka’s sendfile approach, resulting in lower throughput, and discusses when to choose each message‑queue system based on functional needs and performance requirements.
This article examines mmap's historical origins, its performance benefits and drawbacks, and analyzes how Prometheus' time‑series database employs memory‑mapped files, revealing why mmap does not degrade Prometheus performance despite known kernel‑level issues such as TLB misses and lock contention.
This article explains the key techniques behind Kafka's high‑throughput performance, including batch sending, message compression, sequential disk reads/writes, efficient use of PageCache, zero‑copy transfers, and memory‑mapped index files, with code examples illustrating each mechanism.
This article explains why a Java page took five seconds to load, identifies three bottlenecks involving large BLOB fields and file I/O, and walks through four optimization steps—including lazy loading, buffered streams, memory‑mapped files, and sendFile zero‑copy—that reduce the load time to under one second.
This article explains Linux's mmap mechanism, its relationship with shared memory, the advantages and disadvantages of mmap versus traditional shared memory, and provides practical code examples and detailed steps for using mmap in interprocess communication and large‑file handling.
This article explains the principles of Direct Memory Access (DMA), compares it with traditional I/O, details the DMA data‑transfer workflow, and explores zero‑copy techniques such as mmap + write and sendfile, highlighting how they reduce context switches, data copies, and improve overall Linux I/O efficiency.
Zarr provides a modern, chunked and compressed storage format that lets you treat massive NumPy arrays like in‑memory objects, offering on‑demand loading, flexible back‑ends (disk, S3, zip), automatic caching, resizing, parallel reads/writes, and superior performance compared to traditional mmap‑based memmap files.
This article explains Direct Memory Access (DMA), compares it with traditional I/O, details the complete DMA‑based I/O workflow, and explores zero‑copy techniques such as mmap, sendfile, and asynchronous I/O, highlighting their impact on context switches, data copies, and overall transfer performance.
This article explains how Linux processes can share large data efficiently by creating a memory file with memfd_create, mapping it with mmap, and transferring the file descriptor over a Unix Domain Socket, while detailing the kernel mechanisms that enable true cross‑process memory sharing.
This article explains the fundamentals of memory‑mapped files in Linux, covering the mmap/munmap interfaces, mapping types, protection flags, error handling, virtual‑memory structures, kernel implementation details, driver examples, and practical use‑cases such as shared memory and high‑performance I/O.
This article explains Linux's flexible memory allocation system, covering user‑space malloc, kernel‑space kmalloc, vmalloc for large buffers, mmap for file and anonymous mappings, page allocation functions, the slab allocator, and memory pools, with detailed code examples and operational insights.
This article explains Linux memory mapping (mmap), covering its purpose, API parameters, different mapping types, internal kernel implementation, page‑fault handling, copy‑on‑write semantics, practical use cases, and includes a complete Objective‑C example demonstrating file mapping and manipulation.
This article explains the Linux kernel's mmap implementation, covering how virtual memory is allocated, the role of get_unmapped_area, mmap_region, overcommit policies, VMA merging, and the underlying data structures such as mm_struct and vm_area_struct.
This article provides an in‑depth exploration of the Linux mmap system call, covering its role in virtual memory management, page table structures, various mapping types (anonymous, file‑backed, shared, private), flag options, and advanced concepts such as huge pages and transparent huge pages, with kernel‑level diagrams and code examples.
The article analyzes IO‑intensive bottlenecks in mobile apps, explains how MMKV’s mmap‑based storage, rewrite, and expansion mechanisms cause main‑thread stalls, and presents concrete optimizations such as value comparison before write, pre‑expansion, compression, expiration handling, and proper instance management to dramatically reduce latency.
This article explains how mmap maps file segments into a process's virtual address space, how virtual memory and physical memory interact, and the mechanisms of address translation, paging, and segmentation that enable efficient memory sharing and file I/O in Linux.
This article explains Linux's virtual address space layout, the six user‑space memory segments, how malloc decides between brk and mmap based on allocation size, the behavior of free, and why both allocation methods are needed for performance and memory management.
This article explains how Linux’s free command reports memory usage, clarifies the roles of buffer and page caches, demonstrates which caches can be reclaimed, and shows why tmpfs, shared memory, and MAP_SHARED mmap allocations remain in cache until explicitly released.
This article explains how Linux’s free command reports memory usage, clarifies the roles of buffer and page caches, and demonstrates through practical experiments why certain caches—such as those used by tmpfs, shared memory, and MAP_SHARED mmap regions—cannot always be reclaimed as free memory.
This article explains the concepts of virtual memory, kernel and user modes, DMA, and why data must move between kernel and user spaces, then compares standard I/O with zero‑copy techniques, MMAP, and off‑heap buffers, showing how they reduce copy operations and improve Java server performance.
This article explains the zero‑copy technique in Java, covering I/O fundamentals, kernel read/write flow, mmap+write and Sendfile methods, virtual memory mapping, MappedByteBuffer and DirectByteBuffer usage, channel‑to‑channel transfer, Netty composite buffers, and how frameworks like RocketMQ and Kafka leverage zero‑copy for performance.
Facing a challenge of managing roughly one million server-side images and 180 client images, the TOOSIMPLE team built a high-performance backend using fingerprinting, parallel processing, mmap-SSE2 acceleration, and sparsemap indexing, achieving sub-second response times while ensuring correct ordered display.
This article explains how Kafka attains million‑level transactions per second by leveraging sequential disk writes, memory‑mapped files, zero‑copy data transfer, and batch processing, detailing each technique's mechanics and performance impact.
This article explains how zero‑copy techniques such as DMA, sendfile, mmap and Direct I/O reduce data copies and context switches in Linux, compares their mechanisms, advantages and drawbacks, and shows typical use cases like Kafka for high‑performance I/O.
The article explains how traditional I/O incurs four data copies and context switches, and how DMA, zero‑copy techniques such as sendfile, mmap, splice, and Direct I/O reduce copying and system calls, improving performance for disk‑to‑network data transfers in Linux.
The article uses a cultural analogy about Emperor Yongzheng to introduce the concept of mapping, then explains how Linux's mmap can map a physical GPIO register into user space, provides a complete code example for toggling an LED, and discusses mmap's broader uses and performance trade‑offs.
This article provides a comprehensive walkthrough of Linux kernel memory management, covering process address space layout, memory allocation mechanisms, OOM selection criteria, where allocated memory resides, and both manual and automatic memory reclamation techniques, complete with code examples and diagrams.
The article explains how to redesign a high‑traffic user‑registration flow by introducing an asynchronous queue layer, detailing the broker architecture, commitlog, consumeQueue, page‑cache, mmap, topic/tag routing, high‑availability strategies, and the role of a nameserver in a RocketMQ‑style system.
This article explains Linux's process memory layout, allocation mechanisms, out‑of‑memory handling, where different types of memory reside, and both manual and automatic memory reclamation techniques, providing practical examples and kernel‑level insights for developers and system engineers.
This article explains the concept of zero‑copy I/O, how DMA offloads data movement between memory, disks and network cards, and compares practical implementations such as sendfile, mmap and Direct I/O, highlighting their benefits, limitations and typical use cases in Linux systems.
This article explains how Linux zero‑copy mechanisms such as DMA, sendfile, mmap and Direct I/O reduce data copies and context switches, detailing their operation, advantages, limitations, and real‑world usage in systems like Kafka and MySQL.
This article explains how DMA offloads memory‑to‑device data transfers, reduces the four data copies and context switches of traditional I/O, and introduces zero‑copy methods such as sendfile, mmap and Direct I/O, illustrating their mechanisms, advantages, limitations, and real‑world use cases like Kafka and MySQL.
This article explains how DMA offloads memory‑to‑device data transfers, reduces the four data copies and context switches of traditional I/O, and introduces zero‑copy mechanisms such as sendfile, mmap and Direct I/O, with practical examples like Kafka and MySQL.
This article explains the principles of disk I/O, covering read/write workflows, DMA acceleration, page cache, memory‑mapped files, buffering techniques, Linux kernel parameters and practical Java code examples to illustrate how to reduce CPU involvement and improve overall system performance.
The article explains the fundamentals of disk I/O, covering read/write processes, IO interrupts, DMA, page cache, mmap, buffered versus unbuffered file operations, ByteBuffer usage, Linux dirty‑page parameters, and how these mechanisms affect application performance and reliability.
This article provides an in‑depth analysis of Android's Binder driver, explaining why the driver’s core concepts are simple yet involve complex kernel interactions, and detailing the mmap stage, physical page allocation, and per‑page data copying using kmap/kunmap.
This article explains how Linux manages process memory, covering allocation via mmap and brk, OOM killer selection, the role of page cache in different mapping types, manual and automatic memory reclamation, and related kernel parameters.
This article explains how Linux allocates memory for a process, the role of OOM when memory runs out, where different types of memory reside, the behavior of shared and private mappings, tmpfs usage, and both manual and automatic memory reclamation mechanisms.
This article explains Linux kernel memory management by covering process address space layout, allocation mechanisms, OOM killer behavior, overcommit settings, various types of file and anonymous mappings, tmpfs usage, and both manual and automatic memory reclamation techniques.
This article explains the costly four‑copy, four‑context‑switch data path in traditional Linux I/O, introduces DMA as a co‑processor that offloads memory transfers, describes zero‑copy techniques such as sendfile, mmap and Direct I/O, and shows how Kafka and MySQL leverage these methods to reduce CPU overhead and improve throughput.
This tutorial demonstrates how to reproduce a MySQL memory‑spike bug, monitor the process with Linux perf to capture mmap system calls, and analyze the resulting perf.out to identify which SQL statements trigger large memory allocations, while also discussing the method’s limitations.
This article explains why zero‑copy techniques such as mmap, sendfile and DMA dramatically reduce CPU copies and context switches in traditional read/write I/O, improving the performance of high‑throughput systems like RocketMQ and Kafka.
This article describes the design, mmap‑based implementation, architecture, and performance evaluation of a lightweight, low‑overhead mobile logging SDK for Android/iOS, highlighting its compression, encryption, policy‑driven upload, and significant CPU and GC improvements over traditional Java log libraries.
Zero‑copy I/O minimizes data copying by using DMA and kernel mechanisms such as mmap + write or the sendfile() system call, with sendfile achieving true zero‑copy through fewer system calls, context switches, and memory copies compared to traditional read/write paths.
This article explains why MySQL 8.0 appears to lose about 1 GB of disk space when temporary tables are used, detailing the experiment, the role of TempTable, mmap allocation, and how performance_schema and procfs can be used to observe the process.
The article walks through Linux’s mmap file‑read workflow—from the kernel entry point and VMA creation, through page‑fault handling that allocates pages and invokes synchronous or asynchronous readahead, to munmap’s unmapping steps and the deferred file‑cache reclamation mechanisms.
This article explains how zero‑copy techniques such as mmap, sendfile, and splice reduce CPU involvement in data transfer by avoiding memory copies between kernel and user spaces, and shows how Java NIO and Netty implement these mechanisms for high‑performance backend I/O.
Zero-copy techniques such as mmap + write, sendfile, and sendfile + DMA scatter/gather reduce CPU copies and context switches by leveraging kernel‑space data transfers and DMA, dramatically improving I/O performance for high‑throughput systems like RocketMQ and Kafka, as explained with detailed process flows and comparisons.
The article explains Linux zero‑copy I/O techniques—mmap, sendfile, splice, and tee—detailing their design principles, execution flows, context‑switch and data‑copy reductions, advantages, limitations, and appropriate use cases, helping developers choose the optimal method for high‑performance file and network transfers.
This article explains the concept of zero-copy I/O, covering buffer types, virtual memory, mmap+write and sendfile techniques in Java, and demonstrates how Java NIO, Netty, and other frameworks like RocketMQ and Kafka leverage zero-copy to reduce data copying and improve performance.
This article explains how Kafka attains million‑level transactions per second by using sequential disk reads/writes, memory‑mapped files, DMA‑based zero‑copy transfers, and batch data transmission, detailing each technique and its impact on throughput and latency.
Kafka can sustain millions of transactions per second by writing data sequentially to disk, leveraging memory‑mapped files, employing zero‑copy DMA transfers, and batching messages, each technique reducing I/O overhead and CPU involvement, which together enable its high‑throughput performance in big‑data pipelines.
The article explains Linux shared‑memory fundamentals, why it outperforms file‑based IPC, demonstrates the mmap() system call, and walks through a complete Go implementation that creates, synchronizes, reads, and protobuf‑serializes advertising‑tracking metrics in the VCS monitoring platform.
This article explains Linux's zero‑copy techniques—mmap, sendfile, and splice—detailing how they reduce data copies between user and kernel space, improve I/O efficiency, handle edge cases like SIGBUS, and leverage DMA to minimize CPU overhead in server file transfers.
This article explains the zero‑copy technique in Java, covering basic I/O concepts, buffer management, mmap + write and Sendfile methods, and how frameworks like NIO, Netty, Kafka, and RocketMQ use these mechanisms to eliminate data copies and improve performance.
This article explains why traditional read/write file serving incurs multiple data copies and context switches, then introduces mmap, sendfile, and splice as zero‑copy methods that reduce CPU load, discuss their usage, limitations, and practical pitfalls for high‑performance Linux servers.
This article explains the principles of Linux I/O, the need for zero‑copy, the role of page and buffer caches, various data‑transfer mechanisms such as DMA, Direct I/O and mmap, and compares their performance and usage scenarios.
The article explains Linux kernel virtual memory management on 64‑bit ARM64 Android systems, detailing user‑ and kernel‑space address layout, physical vs. linear addresses, allocation mechanisms such as brk and mmap, common allocators, key structures like mm_struct and vm_area_struct, and the functions that control mmap layout and unmapped‑area selection.
This article explains how zero‑copy techniques such as DMA, mmap, and sendfile work, compares them with traditional file I/O, shows their Java NIO implementations, and presents performance test results demonstrating the advantages of mmap for small files and sendfile for large files.
This article explains the hierarchical storage pyramid, Linux kernel I/O stack, page‑cache synchronization policies, the atomicity of write operations, the role of mmap and Direct I/O, and how disk characteristics influence multithreaded read/write performance and design decisions.
This article explains the zero‑copy principle, describes core I/O concepts such as buffers and virtual memory, and details Java implementations including mmap + write, Sendfile, MappedByteBuffer, DirectByteBuffer, channel‑to‑channel transfer, and Netty’s composite buffers for high‑performance data transfer.
This article explains Linux’s virtual address space, the mmap system call, its prototype and typical applications—including shared file mapping, inter‑process communication, and file I/O optimization—while illustrating concepts with diagrams and code examples, and discusses alignment, copy‑on‑write, and memory swapping considerations.
This article explains the Linux mmap system call, covering virtual address space fundamentals, how mmap interacts with the page cache and inode structures, typical use‑cases such as shared memory and anonymous mappings, code examples, and special considerations like alignment, copy‑on‑write and memory swapping.
This article explores Linux kernel memory management, detailing process address space allocation, the behavior of the OOM killer, various memory mapping types (shared, private, anonymous), cache usage, manual and automatic memory reclamation, and related kernel parameters, providing practical insights and examples.
This article explores Linux kernel memory management in depth, covering process address space layout, allocation mechanisms, OOM killer behavior, the whereabouts of allocated memory, and both manual and automatic memory reclamation techniques.
This article presents the background, requirements, and technical solution of a mobile log retrieval component that ensures secure, complete, and efficient log collection, storage, and on‑demand retrieval for iOS and Android applications, including design details such as mmap‑based writing, stream compression, multi‑process handling, and a management platform.
This article explores Linux kernel memory management, detailing process address space layout, allocation mechanisms, OOM handling, various mmap mappings, cache behavior, tmpfs usage, and the kernel's automatic memory reclamation strategies.
This article explains the evolution of data transfer in computers—from early CPU‑bound copying to DMA, channel architectures, and I/O processors—then demonstrates how zero‑copy techniques like sendFile and memory‑mapped I/O dramatically reduce CPU usage and improve throughput.
This article explains Linux kernel memory management, covering process address space allocation, OOM selection criteria, memory mapping types, cache behavior, tmpfs, and both manual and automatic memory reclamation mechanisms.
MMKV is a high‑performance, mmap‑based key‑value component that uses protobuf for serialization, offering fast writes, incremental updates, space management, and data integrity checks, making it a superior alternative to NSUserDefaults for iOS applications.
This article explains Linux kernel memory management, covering process address space layout, memory allocation methods, OOM killer behavior, where different types of memory reside, and both manual and automatic memory reclamation techniques, illustrated with diagrams and command examples.
This article explains the Linux mmap system call, its usage, flags, error handling, related calls like munmap and msync, two shared‑memory techniques, and the underlying virtual‑address and vm_area_struct mechanisms that enable memory mapping.
