Automating Video Learning with Tauri: A Frontend Developer’s Guide to Building a Desktop Automation Tool

Background

My father’s company requires employees to watch online training videos every year, which consumes a lot of time. I was asked if there was a way to automate the learning process.

Previously I created a browser extension for my wife that could automatically complete all courses and pass exams on a medical training site, reducing weeks of work to minutes.

Encouraged by that success, I took on my father’s request, despite the challenges of a desktop‑only banking learning platform that embeds a browser without a console or plugin support.

Technology Selection

Because the platform did not expose any APIs, I first considered using a macro‑recording tool to click automatically, but the learning curve for colleagues would be too high. I decided to develop a small custom tool.

As a frontend developer, I initially thought of Electron, but the resulting binary size was too large for a simple utility.

Therefore I chose Tauri for the project.

Requirement Analysis

The core idea is to:

Let the user select a screen region and record its color information and coordinates.

Continuously scan the region and match colors.

When a match is found, click the region.

This enables automatic progression when a "next lesson" button appears, similar to a "whack‑a‑mole" game.

Getting Started with Rust

Tauri provides many frontend‑callable APIs, but some functionality required for this project needed a bit of Rust knowledge.

Below are the essential commands to install Rust and verify the installation:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

rustc --version
# >>> rustc 1.73.0 (cc66ad468 2023-10-03)

cargo --version

Recommended editor: CLion or VSCode with rust-analyzer .

Variable and Constant Declarations

Both JavaScript and Rust use let and const , but Rust variables are immutable by default. Use mut to make them mutable.

let x = 1;
x = 2; // ❌

let mut x = 1;
x = 2; // ✅

Constants are written in uppercase with underscores.

Data Types

Rust distinguishes many numeric types (i8, u32, f64, etc.). For beginners, i32 is a safe default.

Arrays have a fixed length, e.g.:

let a: [i32; 5] = [0; 5];

For dynamic collections, use Vec :

let mut v: Vec
= Vec::new();
v.push(4);

Importing Packages

Rust imports modules with use :

use autopilot::{geometry::Point, screen, mouse};

This is analogous to JavaScript’s import { Point, screen, mouse } from 'autopilot'; .

Other Syntax

Loops and conditionals omit parentheses:

for i in 0..colors.len() {}

if colors[i] != screen_colors[i] {}

Software Development

Tauri combines a webview (frontend) with Rust (backend). The project structure separates src (frontend) and src-tauri (backend).

Creating the Project

Initialize a Vite + Vue + TypeScript project and add Tauri support (screenshots omitted for brevity).

Frontend

Use Tauri’s JavaScript API to invoke Rust functions and communicate between windows.

import { invoke } from "@tauri-apps/api";
invoke('event_name', payload);

Inter‑window communication example:

import { listen } from "@tauri-apps/api/event";
listen<{ index: number }>("location", async (event) => {
  const index = event.payload.index;
  // ...
});

Creating a transparent overlay window for region selection:

import { WebviewWindow } from "@tauri-apps/api/window";
const screenshot = new WebviewWindow("screenshot", {
  title: "screenshot",
  decorations: false,
  url: `/#/screenshot?index=${props.index}`,
  alwaysOnTop: true,
  transparent: true,
  hiddenTitle: true,
  maximized: true,
  visible: false,
  resizable: false,
  skipTaskbar: false,
});

Backend

The entry point is src-tauri/src/main.rs . Expose Rust functions to the frontend via invoke_handler :

tauri::Builder::default().invoke_handler(tauri::generate_handler![scan_once, ...])

Screen‑color acquisition (using the autopilot crate):

use autopilot::{geometry::Point, screen};

pub fn scan_colors(start_x: f64, end_x: f64, y: f64) -> Vec<[u8; 3]> {
    let mut points: Vec
= Vec::new();
    let mut x = start_x;
    while x < end_x {
        points.push(Point::new(x, y));
        x += 1.0;
    }
    let mut colors: Vec<[u8; 3]> = Vec::new();
    for point in points {
        let pixel = screen::get_color(point).unwrap();
        colors.push([pixel[0], pixel[1], pixel[2]]);
    }
    colors
}

Mouse automation:

use autopilot::{geometry::Point, mouse};
mouse::move_to(Point::new(x, y));
mouse::click(mouse::Button::Left, None);

Configure permissions in src-tauri/tauri.conf.json (example snippet):

"tauri": {
  "macOSPrivateApi": true,
  "allowlist": {
    "all": true
  }
}

Build

Because the target Windows binary must be built on Windows, the author uses GitHub Actions to produce cross‑platform releases.

Key steps in .github/workflows/release.yml include checking out the repo, setting up Node.js and Rust, building the Vite + Tauri app, and publishing the release via the tauri-action . (Full YAML omitted for brevity.)

Conclusion

Choosing between Tauri, Electron, Flutter, or Qt should be based on project pain points rather than brand bias.

Rust has a steep learning curve, but diving in and building real tools accelerates mastery.

Tauri, backed by Rust, is a promising option, especially with upcoming mobile support.

Practical projects—like automating video learning—provide the best motivation to learn new languages and frameworks.