AI‑Driven Security Operations (AISECOPS): Architecture, Practices, and Evaluation

The rapid growth of large models raises the question of what operations teams should do; in the era of LLMs, Ops essentially integrates algorithms into various security scenarios to improve fault detection, incident response, and overall efficiency.

The presentation focuses on five key topics: SECOPS industry pain points, AISECOPS practice, AISECOPS+ extensions, AISECOPS cost assessment, and a demo of results.

SECOPS Pain Points include the paradox of simpler front‑end features demanding increasingly complex back‑end, architecture, and security infrastructure, leading to massive log volumes (terabytes per day) that cannot be handled manually.

AISECOPS Practice outlines four practical scenarios: DNS reverse‑lookup domain detection, web‑request anomaly detection, host command‑execution monitoring, and HIDS data analysis. Data are embedded using large‑model embeddings and classified with algorithms such as XGBoost, SVM, and MLP, achieving >99% accuracy.

Model Selection compares OpenAI embeddings (Ada, Babbage, Curie, Davinci) with open‑source ST5 variants; ST5‑Large (768‑dimensional) offers a good balance of performance, size (569 MB), and cost, running on modest CPU/GPU resources.

Performance & Cost tests show ST5‑Large alone reaches ~60 QPS on a G4dn.xlarge instance, while ST5‑Large + SVM reaches ~20 QPS, meeting the required throughput. Training costs for DNS and web detection amount to roughly ¥1,300, with inference costing $0.000005 per request.

AISECOPS+ Extensions discuss integrating agents (e.g., Aily) for automated rule generation, security matrix enforcement, and DBA assistance, emphasizing API‑based authorization to prevent unsafe actions such as accidental data deletion.

The article also outlines a security evaluation framework (Helpful, Truthful, Harmless) and highlights risks like prompt leakage, supply‑chain attacks, over‑reliance on models, and the need for privacy protection, robustness, explainability, and performance guarantees.

Finally, a Q&A clarifies that the evaluated models were GPT‑3 embeddings (Ada, etc.) and ST5 series, with ST5 chosen for its cost‑effectiveness and suitability for Ops scenarios.