Optimizing Marketing System Blacklist Filtering with Bitmaps

In marketing systems, blacklist filtering can become a performance bottleneck when processing large audience groups.

Traditional blacklist handling may take half an hour for 300k users; the article explores using bitmap data structures to drastically reduce processing time.

Performance optimization

1. Multithreading splits the account set into many tasks, reducing processing of 250k/500k accounts from ~30‑60 minutes to about 1‑2 minutes.

2. Bitmap optimization leverages CDP‑provided bitmap files for both target audience and blacklist. By performing an AND‑NOT operation on the two bitmaps, blacklist accounts are removed in milliseconds.

DataLoader dataLoader = new DataLoader(token, bitMapBaseUrl);
ABitmap customerBitmap = dataLoader.loadGroup(customerGroupCode);
ABitmap blacklistBitmap = dataLoader.loadGroup(blacklistGroupCode);
customerBitmap.andNot(blacklistBitmap);

The bitmap for 500k users occupies only ~2 MB and the AND‑NOT operation finishes in ~80 ms, a dramatic improvement over the previous half‑hour runtime.

Bitmap principles

Bitmaps store boolean values as individual bits, achieving high space efficiency; a 40‑billion‑element set can be represented in ~0.47 GB instead of ~30 GB using 64‑bit longs.

Java’s BitSet implements this concept, using an array of longs where each bit corresponds to an element’s presence. Operations such as add, remove, and contains are performed with bitwise shifts and masks.

Compressed bitmaps

Standard BitSet can consume excessive memory for sparse data; compressed formats like RoaringBitmap, WAH, EWAH, and Concise provide much smaller footprints and faster operations. RoaringBitmap splits integers into high‑ and low‑order parts, using ArrayContainer or BitmapContainer depending on density.

Examples show RoaringBitmap storing a single integer in only 144 B and performing set operations orders of magnitude faster than uncompressed bitmaps.

Applications

Beyond blacklist filtering, bitmaps are useful in Redis for tracking online status, daily sign‑ins, and AB‑testing; in Java collections for efficient bulk removals; and as the basis for Bloom filters, which map strings to bits via multiple hash functions.

Conclusion

By introducing multithreading and bitmap‑based filtering, the blacklist processing time drops from tens of minutes to milliseconds, demonstrating how space‑efficient data structures and parallelism can dramatically improve large‑scale marketing system performance.