CloudWatch AI Operations: What Investigations Actually Does

Why I looked at this

I spent a morning in an AWS workshop on CloudWatch AI Operations. Real account, real broken services, hands on. I went in with one question: is this good enough to use on a customer estate?

Yes, with one condition. The report is only as good as the facts you give it. CloudWatch runs the investigation, finds the root cause from your telemetry, and drafts the incident report. You supply the things telemetry cannot know: impact duration, impact percentage, customer counts, resolution actions, recovery verification. Fill those in and you get a sophisticated RCA document. Skip them and you get a skeleton.

That is the whole model, and once you see it that way the feature makes sense.

Start in dev and QA so your team learns what the hypotheses look like and which facts you always need to collect. Then move to production.

How CloudWatch got here

This diagram is the clearest picture I have seen of what CloudWatch has become.

cloudwatch-observability-stack

Data at the bottom: cross-account observability, logs centralization, Live Tail, audit and event logs, data protection, and the collectors (ADOT, CloudWatch agent).

Analytics in the middle, and it is wide now: Application Signals and span analytics, Application Map, Synthetics, RUM, Dashboards, Alarms, Logs Insights, Metrics Insights, Container / Lambda / Database / Contributor Insights, Anomaly Detection, Network and Internet Monitor, CloudTrail Insights.

AI at the top, deliberately short: CloudWatch investigations, GenAI observability, and MCP servers for CloudWatch, Application Signals and CloudTrail.

The top layer reads what the bottom two collected. If a customer has no traces, no Application Signals and unstructured logs, investigations will still run and still produce confident output based on very little. Fix the Data layer first.

What is in AI Operations

CloudWatch's left nav has an AI Operations group with Overview, Investigations and Configuration.

It also appears where you already work. On a Lambda function's Monitor tab there is an Operational troubleshooting panel with Explore related and Investigate tabs, so you start an investigation from the metric in front of you.

lambda / monitor / investigate

An investigation contains four things: an observations feed (metric anomalies, log anomalies, CloudTrail change events), an agent queue showing the analysis tasks in flight, hypotheses you accept or rule out, and an incident report.

Configure it before you need it

Run an investigation without setting up an investigation group and it still works, using your console session's read-only permissions. But that investigation is visible only to you and is deleted after 24 hours with no recovery. Fine for a demo, not for an incident where three people are on the bridge.

Creating an investigation group is a one-time task per region. Per the setup guide:

• Retention: 7 to 90 days, default 90. An investigation you do not close manually moves to CLOSED after 7 days, and retention runs from there. Reports inherit their investigation's retention, so copy anything you need long term out of the console.
• Encryption uses an AWS owned key by default. Choose Customize encryption settings for a customer managed KMS key.
• An IAM role for the group. This is your control point.

The group role decides what the agent can read

Permissions in the group role determine which resources investigations can touch. The recommended option auto-creates a role with the AWS managed policies for AI Operations. In my account: AIOpsRole-DefaultInvestigationGroup with AIOpsAssistantPolicy. Bring your own role instead if you need it scoped tighter. It is a read-first model.

To create the group you need AIOpsConsoleAdminPolicy or AdministratorAccess, plus iam:CreateRole, iam:AttachRolePolicy and iam:PutRolePolicy for auto-create.

For human access there are three managed policies: AIOpsConsoleAdminPolicy for admins, AIOpsOperatorAccess for engineers who run investigations, AIOpsReadOnlyAccess for viewers. That maps cleanly onto a managed services team.

Two more: CloudTrail change events are on by default and carry a lot of the value, and if X-Ray and Application Signals are deployed the group role will use them for topology and health. One note worth reading before you promise data residency: investigations use cross-Region inference.

The run

I deployed a Lambda, dynamo-capacity-test, that lists DynamoDB tables for capacity checks. Its execution role was missing dynamodb:ListTables. Every invocation failed with AccessDeniedException at lambda_function.py line 43, inside a paginator.paginate() call. 100% failure, successRate at 0.0, from 04:48:00 UTC.

From the Monitor tab I opened the error metric and clicked Investigate. Impact start time was picked up automatically.

The agent queue is my favourite part. It shows the tasks in flight, each expanding into a written summary, so you can see where it is going while it works.

agent queue

It returned one hypothesis, with a root cause summary, impact, sequence of events, evidence list and dependency graph.

hypothesis details

The supporting data included three sample log lines behind a pattern @message | anomaly Log Insights query, plus two IAM change events it picked up from CloudTrail on its own.

Accept or rule out. You cannot generate a report until you accept one, which keeps a human in the loop on the root cause before anything gets written down.

accept a hypothesis to continue

Fill the facts, get the RCA

The report structure is complete: Background, Summary, Metrics, Customer Impact with blast radius, What Went Well, Incident Response Analysis (Detection, Diagnosis, Mitigation, Recovery Confirmation), Timeline, 5 Whys, Lessons Learned, and numbered action items with priorities and target dates.

incident report

Alongside it sits the Facts panel. Mine listed 20 facts needing input.

update facts

These are the things telemetry cannot tell you: impact duration, impact end time, impact percentage, affected and total customer counts, incident severity, initial responders, escalation path, mitigation steps, time to mitigate, time to resolve, resolution actions, recovery verification method, prevention measures, monitoring enhancements, process improvements, key learnings, business impact, communication activity.

You can enter them as free text or field by field. Each fact is versioned. Then hit Regenerate and the report rewrites itself around them, with the missing-fact markers replaced by your numbers. That is the loop: CloudWatch brings the telemetry evidence, you bring the operational context, and the output is a full RCA rather than a metrics dump.

Practical advice: keep a short checklist of those fields in your incident channel. If your on-call captures impact start, impact end, severity, responders and mitigation steps as the incident runs, filling the facts afterwards takes two minutes and the report comes out clean the first time.

Try it yourself with EC2 and an alarm

You do not need a workshop account. A single EC2 instance is enough to see the whole flow.

1. Launch an EC2 instance with an instance profile that includes AmazonSSMManagedInstanceCore, so Systems Manager can reach it. Amazon Linux 2023 ships the SSM Agent already.
2. Connect with Session Manager instead of SSH, and turn on Session Manager logging to CloudWatch Logs in Systems Manager preferences. Now every command you run is in a log group.
3. Install and configure the CloudWatch agent to ship application and system logs plus memory and disk metrics. This is the Data layer from the diagram. Without it there is nothing for the agent to read.
4. Break something on purpose in the Linux config. Stop the web service, point it at a bad config path, fill the disk, or misconfigure the app so it starts throwing 5xx. Anything that produces a real error signal in logs and metrics.
5. Create a CloudWatch alarm on that signal: error count, 5xx rate, disk utilisation, whatever you broke.
6. Wait for the alarm to go into ALARM state, open it, and click Investigate. The investigation picks up the alarm's time window automatically and starts pulling logs, metrics and change events.

You can also attach the investigation as an alarm action so it starts the moment the alarm fires, which is the version you actually want on call. Then accept the hypothesis, fill the facts, and generate the report. Whole loop, one instance.

Cost, DevOps Agent, and my take

What it costs

CloudWatch investigations carries no additional charge. Per the cost documentation, it does use AWS service capacity for telemetry and resource queries, and a few categories are billable exceptions: certain CloudWatch APIs (ListMetrics, GetDashboard, ListDashboards, GetInsightRuleReport), X-Ray APIs (GetServiceGraph, GetTraceSummaries, BatchGetTraces), Cloud Control APIs that can pull in Kinesis Data Streams and Lambda usage, and SNS if you integrate with a chat application. AWS describes these as minimal for normal usage.

Two things to say to a customer. Running an Automation runbook from a suggested action bills against Systems Manager Automation pricing, so acting on the investigation is not always free. And the real spend is the telemetry underneath: if this pushes a team to instrument properly, the bill moves in the ingestion line, not in anything labelled AI. Check the current service quotas on concurrent and monthly investigations per region before planning a rollout.

Investigations vs AWS DevOps Agent

Investigations makes troubleshooting faster inside CloudWatch. DevOps Agent is a different shape, built to run the on-call loop and close the prevention gap afterwards. If you are already on CloudWatch and want RCA to stop eating 40 minutes, start here. I broke an EC2 instance to test DevOps Agent back in March, and that write-up is here.

My take

Configure the group before you need it. Scope the group role deliberately, because that role is the whole security story. Set retention with the 7-day auto-close in mind and copy reports out. If customer logs carry PII, put a CloudWatch Logs data protection policy on the log group first so masking happens before the agent ever reads it.

Use it as a first responder. It is very good at ruling out the wrong layer fast, which is where most incident time goes. Verify the hypothesis before you accept it, fill the facts, and you get an RCA worth sending.

Run it in dev and QA first. Then production. The terminal route through Kiro CLI and the CloudWatch MCP servers is covered in a companion post.