iOS Multi‑Device Remote Control: Technical Analysis and Implementation

Introduction: Mobile multi‑device remote control technology is well‑established for Android (e.g., STF) but remains limited for iOS, where existing solutions are either single‑device or suffer from poor video quality.

Key technology analysis – Event‑driven technology: iOS automation frameworks fall into non‑intrusive and intrusive categories. After evaluating WedDriverAgent, UIAutomation, Appium, and others, the non‑intrusive WedDriverAgent (using private APIs and XCTest) was selected as the driver solution.

Screen‑capture technology: Three non‑jailbreak methods are reviewed – AirPlay Mirror (private protocol, limited commercial implementations), AVFoundation/CoreMediaIO (e.g., ios‑minicap, fast but limited to one mac‑to‑one device), and idevicescreenshot (simple but low frame rate, ~3‑5 fps). Limitations of each method are discussed.

Solution approach and practice: The synchronization logic in WebDriverAgent (wda) was removed to speed up event handling. Private API XCEventGenerator was used for click, swipe, long‑press, and home‑key events. Screenshots are compressed using UIImageJPEGRepresentation(img,0.1) , yielding ~50 KB images with 30‑50 ms capture‑to‑send latency. Two socket sub‑threads were injected into wda to handle real‑time event commands and continuous screenshot streaming, with screenshots forced onto the main UI thread and timers set to NSRunLoopCommonModes to avoid interference.

Other issues and solutions: Coordinate conversion for various iPhone models is implemented to map logical to screen coordinates. iOS 10+ network‑permission dialogs, which may block wda, are bypassed either by invoking private APIs to force the dialog or by proxying network traffic through macOS using iproxy and usbmuxd . HierarchyView tree synchronization was optimized by filtering to only name and rect attributes, reducing retrieval time to ~800 ms. Multi‑simulator support is added via FBSimulatorControl .

Conclusion: The proposed iOS multi‑device remote control framework achieves millisecond‑level event response and 20‑30 fps video streaming, enabling cluster testing and efficient utilization of iOS devices for remote automation.