Design and Implementation of a FIFO Export Queue for Large Data Exports in Java

The business requirement is to manage growing database export volumes, where full‑data exports can heavily impact server performance due to intensive MySQL I/O and file stream operations, prompting the need for a queued export mechanism.

The proposed solution uses a FIFO (first‑in‑first‑out) queue with a fixed maximum size; users are enqueued for export, and after the export completes they are dequeued, allowing the next user to proceed. An asynchronous file‑export table can also record export status and file locations.

package com.example.system.config; import com.example.system.api.domain.ExportUser; import lombok.extern.slf4j.Slf4j; import org.springframework.stereotype.Component; import java.util.LinkedList; @Slf4j @Component public class ExportQueue { private final int MAX_CAPACITY = 10; // queue max capacity private LinkedList queue; // user queue public ExportQueue(LinkedList queue) { this.queue = new LinkedList<>(); } /** * Add a user to the queue; wait if the queue is full. */ public synchronized LinkedList add(ExportUser sysUser) { while (queue.size() >= MAX_CAPACITY) { try { log.info("Current queue is full, please wait"); wait(); } catch (InterruptedException e) { e.getMessage(); } } queue.add(sysUser); log.info("Current export queue size: " + queue.size()); notifyAll(); return queue; } /** * Retrieve and remove the next user from the queue; wait if empty. */ public synchronized ExportUser getNextSysUser() { while (queue.isEmpty()) { try { wait(); } catch (InterruptedException e) { e.printStackTrace(); } } ExportUser sysUser = queue.remove(); notifyAll(); // wake up waiting threads return sysUser; } }

The abstract class AbstractExport defines generic export operations using EasyExcel, handling pagination, writer creation, and providing abstract methods for counting records, fetching export details, and template‑based export.

package com.example.system.config; import cn.hutool.core.bean.BeanUtil; import cn.hutool.core.util.PageUtil; import com.alibaba.excel.EasyExcel; import com.alibaba.excel.ExcelWriter; import com.alibaba.excel.write.metadata.WriteSheet; import com.example.system.api.domain.ExportUser; import lombok.extern.slf4j.Slf4j; import javax.servlet.http.HttpServletResponse; import java.io.IOException; import java.net.URLEncoder; import java.util.List; @Slf4j public abstract class AbstractExport { public abstract void export(ExportUser sysUser) throws InterruptedException; public void export(HttpServletResponse response, int pageSize, T t, Class k, String fileName) throws Exception { ExcelWriter writer = null; try { writer = getExcelWriter(response, fileName); int total = this.countExport(t); int loopCount = PageUtil.totalPage(total, pageSize); BeanUtil.setProperty(t, "pageSize", pageSize); for (int i = 0; i < loopCount; i++) { BeanUtil.setProperty(t, "pageNum", PageUtil.getStart(i + 1, pageSize)); List kList = this.getExportDetail(t); WriteSheet writeSheet = EasyExcel.writerSheet(fileName).head(k).build(); writer.write(kList, writeSheet); } } catch (Exception e) { String msg = "Export " + fileName + " exception"; log.error(msg, e); throw new Exception(msg + e); } finally { if (writer != null) { writer.finish(); } } } public ExcelWriter getExcelWriter(HttpServletResponse response, String fileName) throws IOException { response.setContentType("application/vnd.ms-excel"); response.setCharacterEncoding("utf-8"); String fileNameUtf = URLEncoder.encode(fileName, "UTF-8").replaceAll("\\+", "%20"); response.setHeader("Content-disposition", "attachment;filename*=utf-8''" + fileNameUtf + ".xlsx"); return EasyExcel.write(response.getOutputStream()).build(); } public abstract void complexFillWithTable(T t, String fileName, HttpServletResponse response); public abstract int countExport(T t); public abstract List getExportDetail(T t); }

The concrete implementation ExportImpl extends AbstractExport , injects ExportQueue , simulates a long‑running export with Thread.sleep(20000) , and removes the processed user from the queue.

package com.example.system.service.impl; import com.alibaba.excel.ExcelWriter; import com.example.system.api.domain.ExportUser; import com.example.system.config.AbstractExport; import com.example.system.config.ExportQueue; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import javax.servlet.http.HttpServletResponse; import java.io.IOException; import java.util.LinkedList; import java.util.List; @Service @Slf4j public class ExportImpl extends AbstractExport { @Autowired private ExportQueue exportQueue; @Override public void export(ExportUser sysUser) throws InterruptedException { log.info("Export method started"); LinkedList queue = exportQueue.add(sysUser); log.info("Export queue: " + queue); Thread.sleep(20000); // simulate export processing time ExportUser nextSysUser = exportQueue.getNextSysUser(); log.info("Next user after removal: " + nextSysUser.getUserName()); } @Override public void export(HttpServletResponse response, int pageSize, Object o, Class k, String fileName) throws Exception { super.export(response, pageSize, o, k, fileName); } @Override public ExcelWriter getExcelWriter(HttpServletResponse response, String fileName) throws IOException { return super.getExcelWriter(response, fileName); } @Override public void complexFillWithTable(Object o, String fileName, HttpServletResponse response) { // not implemented } @Override public int countExport(Object o) { return 0; } @Override public List getExportDetail(Object o) { return null; } }

A test controller creates a new thread for each export request, constructs an ExportUser with the thread name, and invokes export.export(sysUser) . The logs show that when the queue reaches its limit of 10, further requests wait until a slot frees up.

Test results confirm that the first and second requests, spaced 10 seconds apart, behave as expected: after the first export finishes, the user is removed from the queue and the next waiting user becomes the head of the queue. The article also notes unfinished parts such as file‑table design, OSS upload, download handling, and considerations for high‑concurrency scenarios, suggesting alternatives like a Redis‑based queue.