Tired of Alert Overload? How Amazon DevOps Agent Automates Root‑Cause Investigation

As AWS workloads grow—spanning EC2, RDS, ECS/EKS, Lambda, networking and load balancing—operations teams confront four major pain points: a flood of alerts from CloudWatch and third‑party monitors, difficulty correlating data across services, long response times (30 minutes to hours), and fragmented knowledge that hinders scaling.

Amazon DevOps Agent is introduced as an AI‑driven autonomous operations service that receives alerts from any AWS service, automatically gathers metrics, logs, CloudTrail events, and configuration data, performs deep root‑cause analysis, and produces a structured Markdown journal without human intervention.

Core capabilities include:

Full‑stack service analysis: the agent builds an application topology by calling AWS APIs to discover resources and dependencies across accounts and regions.

AI‑autonomous investigation: after an alarm arrives, the agent decides the investigation order, collects CloudWatch metrics, CloudTrail records, code repository data, CI/CD history, etc., without a predefined runbook.

Deep root‑cause analysis: it not only reports the symptom (e.g., CPU spike) but also traces why it happened and suggests remediation (e.g., upgrade instance type).

Structured output: investigation results are saved as Markdown journal records containing symptom, finding, observation, gap, and summary fields.

Event‑driven integration: investigation lifecycle events are published to EventBridge, enabling seamless downstream processing.

Rich third‑party integrations: native support for Datadog, Dynatrace, New Relic, Splunk, Grafana, PagerDuty, ServiceNow, GitHub, GitLab, Azure DevOps, Slack, and custom MCP integrations.

Security & compliance: the agent runs under an IAM service principal aidevops.amazonaws.com, all actions are logged in CloudTrail, and records are persisted for audit.

Demo scenario uses an EC2 CPU alarm to illustrate an eight‑step event‑driven workflow:

Generate high CPU load with stress --cpu 4.

CloudWatch alarm triggers when CPU > 80 % for two evaluation periods.

EventBridge Rule‑1 captures the alarm event.

Lambda‑A invokes create_backlog_task(taskType='INVESTIGATION') to start an investigation.

DevOps Agent autonomously analyzes CloudWatch metrics, CloudTrail events, and EC2 configuration (5‑15 min).

Agent publishes an Investigation Completed event.

Lambda‑B calls list_journal_records() to retrieve the Markdown summary.

The summary is sent to a Feishu chat group.

The article provides the exact AWS CLI commands for creating the required IAM role, attaching the AWSLambdaBasicExecutionRole and custom aidevops policies, publishing a Lambda layer with the latest boto3, and deploying the two Lambda functions ( devops-agent-trigger-investigation and devops-agent-notify-feishu).

aws iam create-role \
  --role-name DevOpsAgentDemoLambdaRole \
  --assume-role-policy-document file://iam/lambda-role-trust.json

aws iam attach-role-policy \
  --role-name DevOpsAgentDemoLambdaRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

aws iam put-role-policy \
  --role-name DevOpsAgentDemoLambdaRole \
  --policy-name DevOpsAgentAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": ["aidevops:CreateBacklogTask","aidevops:ListJournalRecords"],
      "Resource": "*"
    }]
  }'

Deployment of the Lambda functions uses commands such as:

aws lambda create-function \
  --function-name devops-agent-trigger-investigation \
  --runtime python3.12 \
  --handler lambda_a.lambda_handler \
  --role "arn:aws:iam::${AWS_ACCOUNT_ID}:role/DevOpsAgentDemoLambdaRole" \
  --zip-file fileb://lambda/lambda_a.zip \
  --timeout 30 --memory-size 128 \
  --layers "${LAYER_ARN}" \
  --environment "Variables={DEVOPS_AGENT_SPACE_ID=${DEVOPS_AGENT_SPACE_ID}}"

EventBridge rules bind the alarm to Lambda‑A and the investigation‑completed event to Lambda‑B:

aws events put-rule \
  --name "DevOps-Agent-Demo-Alarm-To-Lambda" \
  --event-pattern '{"source":["aws.cloudwatch"],"detail-type":["CloudWatch Alarm State Change"],"detail":{"alarmName":["DevOps-Agent-Demo-CPU-High"]}}'

aws events put-targets \
  --rule "DevOps-Agent-Demo-Alarm-To-Lambda" \
  --targets "Id=trigger-investigation,Arn=arn:aws:lambda:${AWS_REGION}:${AWS_ACCOUNT_ID}:function:devops-agent-trigger-investigation"

Chat API enables real‑time conversational interaction. After creating a chat session with client.create_chat, messages are sent via client.send_message, and the streaming EventStream returns events such as responseCreated, contentBlockDelta, and responseCompleted. Using the same executionId preserves context across multiple turns.

The article concludes that Amazon DevOps Agent marks a shift from reactive, rule‑based ops to proactive, AI‑driven remediation, with future directions including automated fix execution, predictive failure detection, and multi‑cloud support (Azure and on‑premises via MCP).