Improving Automated Test Execution Efficiency with Parallel Runners (hjRunner)

Background – Traditional automated test execution runs in a single process; for large test suites (e.g., 594 cases) the runtime exceeds ten minutes, which is acceptable for regular regression but too slow for rapid release pipelines.

Evolution of the approach – The original framework used Python + nosetests with a single‑process runner, taking 11 minutes for 594 cases. By distributing the suite to eight parallel Runner instances, the total runtime dropped to 2 minutes 25 seconds.

NOTE: A single‑process Python program occupies only one CPU core; the concurrency level can be set to roughly twice the number of cores, but actual speedup depends on test case duration distribution.

Feature Overview – Master

Assign test tasks and start Runners

Collect results from Runners

Merge results and generate a consolidated report

Upload result files and screenshots to a log server

Send result emails

Trigger WeChat alerts

Create Jira bugs for failures

Rerun failed test cases

Feature Overview – Runner

Execute a single test file as a unit

Synchronize each test's result back to the Master

After execution, send the request logs to the Master

Current usage – The tool runs daily monitoring tasks (25,850 executions) and regression/inspection runs (649 executions) at Hujiang, and is being integrated into the DevOps automated‑testing pipeline.

Master‑Runner communication – Implemented with pyzmq request‑reply pattern. The Master starts a server to receive messages; each Runner uses a client to send messages. A five‑attempt retry with a five‑second timeout prevents message loss and server blocking.

Command and parameter handling – Test execution is based on nosetests . A custom nose plugin named hjRunner adds support for additional parameters:

hj-nose-parameters : execution, upload, and notification options (passed as a dictionary for extensibility)

run-parameters : environment variables initialized before execution (also a dictionary)

tests : directories or files to run, comma‑separated

local-runners : number of concurrent runners

{
'uploadResult': True,
'runEnv': 1,
'reportCategory': 0,
'checkCasesDescription': False,
'mailSend': {
'sendMail': False,
'showUrlInMail': True
},
'wxAlarm': False,
'submitBug': True
}

Result collection – The nose plugin overrides success, failure, and error callbacks to gather:

Test case name

Relative file path

Result status

Detailed step logs

Exception messages

These details are sent to the Master via the client’s sendData method.

After all tests finish, the Master’s report method aggregates logs and final status, then dispatches them to the appropriate channels.

Master concurrent execution

1. Generating nosetests command

The Master injects hj-nose-parameters and run-parameters into the nosetests command, sets the desired concurrency, and launches multiple processes, each handling a distinct test file.

2. Task allocation among runners

When a Runner finishes its assigned task, it reports results to the Master, which then dispatches the next pending test file to the now‑idle Runner, ensuring continuous utilization of all workers.

Result archiving, email reporting, and WeChat alerts

Test logs are stored in a database and can be viewed on the QAClub platform; HTML reports are uploaded to a log server and attached to notification emails.

Failed cases trigger a WeChat alert ("Hujiang‑Ops‑Monitoring") to the responsible personnel, configurable via QAClub.

Failed cases automatically create Jira bugs and assign them to the configured QAClub users.

Conclusion

hjRunner is a nose‑based plugin that adds multi‑process parallel execution, significantly improving test throughput while remaining extensible for future features, making automated testing more efficient and convenient for development teams.