Designing and Optimizing Log Storage and Query in HBase

We first summarize the business characteristics of log data: logs have a fixed format per component but contain many custom tags for later troubleshooting, making query fields highly flexible.

HBase is selected for log storage because its qualifier is flexible for semi‑structured tag data and it belongs to the Hadoop ecosystem, facilitating offline analysis and data mining.

However, HBase lacks secondary indexes, so tag‑based queries often require full table scans, effectively treating HBase as a simple key‑value store.

Drawbacks of self‑built indexes

The implementation stores logs in a log table, index metadata in a meta table, and creates a dynamic index table for each tag combination. This design leads to low index creation efficiency due to nested loops and full scans, and query performance heavily depends on the completeness of the index.

HBase query fundamentals

HBase supports three access methods: exact rowKey match, rowKey range with filters, and full table scan. From a programming perspective, only get (single row) and scan (range or full scan) are available.

To improve query efficiency, careful rowKey design is essential. RowKey should embed query factors such as time intervals and common log attributes, and be of fixed length with numeric or alphabetic segments to enable precise range scans.

Two rowKey patterns are presented: one for structured business logs and another for unstructured component logs, using collection time as the timestamp when parsing is impossible.

Filters can further narrow results after the rowKey range is determined, limiting time span and enabling pagination.

Copressor usage

Instead of external Storm jobs for index building, HBase coprocessors (Observer and EndPoint) can intercept data writes and construct indexes in real time, similar to database triggers.

Third‑party indexing

For large‑scale log data, integrating full‑text search engines like Solr or Elasticsearch provides more efficient indexing, with HBase handling raw storage while the search engine handles query acceleration.

The article also includes architecture diagrams illustrating the overall design.