Handling Android Permission Dialogs Using Template Matching and SIFT Feature Matching

Background: During automated test execution, Android apps often present permission dialogs that need to be handled, but the test scripts rely on uiautomator and cannot start the dialog assistant service, so the scripts must handle dialogs themselves.

Solution: The system currently uses two image‑recognition algorithms—template matching and SIFT feature‑point matching—implemented with the Python OpenCV library. Template matching slides a template over the target image, computes normalized cross‑correlation (TM_CCOEFF_NORMED), finds the highest score with minMaxLoc , and derives the center coordinate of the matched region.

Template matching works only when scale, illumination, and resolution match, which limits its generalization across different devices. To overcome this, SIFT feature matching extracts scale‑invariant keypoints and 128‑dimensional descriptors from both template and target images, matches them with knnMatch , applies homography, and computes the center point of the matched template.

The overall workflow launches a screenshot‑and‑click service on the phone, captures screenshots, sends them together with the template to the image‑recognition service, receives the center coordinates, and invokes the click service to press the appropriate button.

Future work: To handle diverse and unknown dialogs, the authors propose combining image localization with OCR to extract button text, then using a BERT‑based model and cloud rule configuration to decide which button to click.