Implementing Automated Interface Testing with Jenkins, JMeter, and Groovy to Identify and Count False Positives

In this article, a false positive is defined as a failure that is not caused by the test case itself but by external factors such as network or environment problems.

The author first sets up an automated testing framework by integrating Jenkins with Ant and JMeter, then containers the JMeter execution. Ant runs the test scripts in batch, and Jenkins produces a test report.

The report contains information about successful and failed test cases. By filtering the failures that are likely false positives (e.g., network errors), the pipeline displays these cases in Jenkins for the user to confirm. Confirmed false positives are counted as successful cases, and the success, failure and false‑positive rates are recalculated.

Preparation steps include generating a test report with JMeter and parsing the report with Jenkins.

To generate the report, a JMeter test case is created (intentionally with an incorrect address) and executed via Ant. The command‑line output and XSLT processing produce HTML summary and detail reports.

ZeyangdeMacBook-Pro:jmetertest zeyang$ ls
README.md        jmeter.results.shanhe.me.xsl
build.xml        result
jmeter-results-detail-report_21.xsl  scripts
jmeter.log
ZeyangdeMacBook-Pro:jmetertest zeyang$ ant -buildfile ./build.xml
Buildfile: /Users/zeyang/Desktop/jmetertest/build.xml

run:

test:
   [jmeter] Executing test plan: /Users/zeyang/Desktop/jmetertest/scripts/blog.jmx => /Users/zeyang/Desktop/jmetertest/result/jtlfile/TestReport_201911241112.jtl
   [jmeter] 2019-11-24 11:12:31,449 main ERROR FileManager (jmeter.log) java.io.FileNotFoundException: jmeter.log (Permission denied)
   [jmeter] Creating summariser
[jmeter] Created the tree successfully using /Users/zeyang/Desktop/jmetertest/scripts/blog.jmx
   [jmeter] Starting the test @ Sun Nov 24 11:12:31 CST 2019 (1574565151991)
   [jmeter] Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
   [jmeter] summary +      1 in 00:00:00 =    4.3/s Avg:    34 Min:    34 Max:    34 Err:     1 (100.00%) Active: 3 Started: 3 Finished: 0
   [jmeter] summary +      3 in 00:00:01 =    3.9/s Avg:   338 Min:     0 Max:   807 Err:     1 (33.33%) Active: 0 Started: 3 Finished: 3
   [jmeter] summary =      4 in 00:00:01 =    4.0/s Avg:   262 Min:     0 Max:   807 Err:     2 (50.00%)
   [jmeter] Tidying up ...    @ Sun Nov 24 11:12:33 CST 2019 (1574565153497)
   [jmeter] ... end of run

report:
   [xslt] Processing /Users/zeyang/Desktop/jmetertest/result/jtlfile/TestReport_201911241112.jtl to /Users/zeyang/Desktop/jmetertest/result/htmlfile/201911241112_DetailReport.html
   [xslt] Loading stylesheet /usr/local/apache-jmeter-5.1.1/extras/jmeter.results.shanhe.me.xsl
   [xslt] Processing /Users/zeyang/Desktop/jmetertest/result/jtlfile/TestReport_201911241112.jtl to /Users/zeyang/Desktop/jmetertest/result/htmlfile/201911241112_SummaryReport.html
   [xslt] Loading stylesheet /usr/local/apache-jmeter-5.1.1/extras/jmeter-results-detail-report_21.xsl

BUILD SUCCESSFUL
Total time: 4 seconds

After the report is generated, it can be uploaded to GitHub (optional) and displayed in Jenkins.

For parsing the HTML report, a Groovy regular expression is used to extract common patterns. The following Jenkinsfile demonstrates the parsing logic and user interaction:

#!groovy

def falsePositive = []

node {
    stage("GetCode"){
        //checkout
        checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '24982560-17fc-4589-819b-bc5bea89da77', url: 'https://github.com/zeyangli/jmetertest.git']]])
        sh "curl https://raw.githubusercontent.com/zeyangli/jmetertest/master/SummaryReport.html -o SummaryReport.html"
        sh "cat SummaryReport.html"
        publishHTML([allowMissing: false, alwaysLinkToLastBuild: false, keepAll: false, reportDir: '', reportFiles: 'SummaryReport.html', reportName: '接口测试报告', reportTitles: ''])
    }

    stage("fenxi"){
        def fileContext = readFile "./SummaryReport.html"
        println(fileContext)
        def pattern = "
.*?
" - "
" - "

The pipeline shows the parsing result, lets the user select false‑positive cases via an extendedChoice parameter, and prints the selected cases.

Finally, the number of false‑positive test cases is calculated and can be fed into a metrics system for further analysis.