In-Memory Databases: Concepts, Evolution, Applications, and Selection Guide

In‑memory database management systems store data primarily in RAM, offering a solution for high‑concurrency, low‑latency data management. Recent DRAM capacity growth and price reductions have made large‑scale in‑memory storage feasible, leading to mature products such as Redis and Memcached.

In the coming years, the commercialization of non‑volatile memory (NVM) will further expand opportunities for in‑memory databases.

This whitepaper defines in‑memory databases, reviews their development history and key characteristics, analyzes typical scenarios in e‑commerce, live streaming, and telecom, introduces and compares major products, and offers product‑ and hardware‑selection guidance from both technical and management perspectives, concluding with future trends.

Download Link: In‑Memory Database Whitepaper

An in‑memory database (also called main‑memory database) relies mainly on RAM for data storage.

Traditional databases use memory buffers to reduce disk I/O, whereas in‑memory databases place the entire database in RAM, delivering orders‑of‑magnitude performance improvements suitable for performance‑critical workloads.

1. Maturity of Memory Technology

After 1982, 30‑pin 256 KB SIMM modules appeared, marking the start of modern memory.

Late 1980s saw 72‑pin SIMMs with 512 KB‑2 MB capacities; early 1990s introduced EDO‑DRAM (4‑16 MB).

1995 brought 64‑bit SDRAM, reaching 64 MB and delivering major performance gains.

Following Moore’s law, DDR3 capacities reached 16 GB by 2019.

Memory price per megabyte has dropped by nearly nine orders of magnitude since the 1970s, making large‑scale in‑memory data processing affordable.

2. Bottlenecks and Breakthroughs

Traditional memory hierarchies place hot data close to the CPU. Current in‑memory databases store all data in DRAM, which, despite price drops, remains costly for massive datasets and is volatile, requiring persistence solutions.

Persistent Memory (PM) or Storage‑Class Memory (SCM) sits between DRAM and SSD, offering load/store access with data durability, bridging the latency gap between volatile and non‑volatile storage.

3. Development Stages

In‑memory databases have progressed through prototype, theoretical maturity, market growth, and rapid expansion phases.

4. Advantages and Challenges

Advantages: microsecond‑level read/write latency and QPS exceeding 100 k, scaling to several hundred thousand QPS with user‑space networking and large pages.

Challenges: DRAM volatility leads to data loss on power failure; persistence mechanisms reduce performance, and current key‑value stores offer limited durability.

Two main persistence approaches exist: (1) persisting every operation, which hurts performance, and (2) policy‑based persistence, which balances speed and risk.

5. Classification

Major in‑memory databases fall into three categories:

Key‑Value stores (e.g., Redis, Memcached, Aerospike) – simple data model, high performance.

Relational in‑memory databases (e.g., Oracle TimesTen, SAP HANA, MemSQL, SQLite) – SQL support, suitable for complex queries.

Other types (e.g., graph in‑memory databases like RedisGraph).

6. Product Landscape

According to DB‑Engines Ranking, the most active ten in‑memory databases include open‑source Redis and Memcached (key‑value) and SQLite (relational). Commercial products like SAP HANA lead in popularity among relational offerings.

Oracle TimesTen, released in 1995, remains active; Apache Ignite (2014) supports both key‑value and relational models and is gaining traction. Most relational in‑memory databases claim ACID support but often compromise performance.

7. Selection Recommendations

Database selection should start from business requirements: data volume, concurrency, read/write patterns, consistency, latency, complexity, and continuity dictate technical needs such as consistency, fault tolerance, scalability, and security.

Technical Factors

Performance requirements – high‑concurrency, low‑latency scenarios (e.g., real‑time gaming leaderboards, live‑stream fan counts) favor in‑memory databases.

Strong consistency needs – if ACID transactions are essential, traditional relational databases may be preferable, or hybrid solutions with careful architecture.

SQL compatibility – complex relational queries and fixed schemas benefit from relational in‑memory databases; flexible, simple workloads suit key‑value stores.

Additional considerations include data size, cost, scalability, and maintainability.

Non‑Technical Factors

Ecosystem maturity – tooling, community, commercial support.

Architecture fit – compatibility with existing application stack and programming languages.

Team expertise – familiarity, learning curve, and operational tooling.

Source: Full‑Stack Cloud Architecture

Download: In‑Memory Database Whitepaper

Additional References

Financial‑Grade Database Disaster Recovery Report (2021)

Database Development Research Report (2021)

Distributed Database Principles and Architecture Design

Disclaimer: Please credit the author and source when reposting. Contact us for copyright issues.

Recommended Reading: See the “Architect’s Full‑Store Technical Materials Pack” for more architecture knowledge.