Automated remediation on AWS

Automated remediation is a narrow tool most AWS teams reach for too early. An organization trail and a service control policy remove most of the CloudTrail-disruption problem before any code runs. What remains needs a loop that is idempotent, bounded, reversible and switchable-off — because an automation holding a privileged role is an availability risk an attacker can aim.

Scope and lineage

This article descends from "Automated Remediation for CloudTrail Disruption" by Setu Parimi and Steve George, published on cloudsecops.com. That piece described a Lambda function, triggered by a CloudWatch Events rule watching eight CloudTrail control-plane operations, that re-enabled logging and sent an SNS email, deployed by CloudFormation. Its closing claim was that the setup "will act as a first level defense when someone tries to stop or disrupt the logs in CloudTrail on the account."

The original page does not expose a publication date, and it does not contain the Lambda source or the rule definition — only references to downloadable templates. Nothing here is presented as preserved text, and no original date is claimed. This is newly written and verified against AWS documentation on 2026-08-06, with a second verification pass on 2026-08-07. The subject and the defensive thesis carry forward. Several technical recommendations have changed materially, and the article says where.

In scope. AWS only. Event-driven and evaluation-driven remediation. The mechanisms compared are EventBridge with Lambda or Step Functions, AWS Config remediation via SSM Automation runbooks, Security Hub automation rules and custom actions, GuardDuty findings as triggers, and service control policies and resource control policies as prevention. The cross-cutting concerns are idempotency, concurrency, retries, dead-letter handling, loop protection, rollback, blast radius, authorization of the remediation identity, testing, exceptions and audit. One pattern is worked end to end: CloudTrail disruption.

Out of scope. Other clouds. Kubernetes admission control. SOAR product selection. Forensic acquisition, which appears only as a reason to contain rather than remediate. Detection engineering itself, beyond deciding whether a signal is good enough to act on without a human.

Assumptions. The reader operates AWS Organizations with all features enabled, or can. A log archive account exists or is achievable. Infrastructure reaches production through Terraform, CDK or CloudFormation.

A note on labelling. No code here is production-ready. Every block is captioned illustrative or pseudocode: illustrative blocks are structurally complete enough to adapt, pseudocode blocks convey structure only. Where a block's behaviour was exercised outside AWS during review, the caption says what was run. Anything not stated as exercised was not.

The decision that comes before the code

The interesting question is not how to write a remediation function. It is which findings deserve one. Teams that skip that question end up with a fleet of small privileged automations whose combined behaviour nobody models, and whose failure modes surface for the first time during an incident.

CloudSecOps uses four axes to classify a finding, and four outcomes. The axes and the thresholds below are our judgment, not AWS guidance.

• Reversibility. Does the action have a stated inverse that restores the prior state, and can it run within minutes? "Delete the public bucket policy" is reversible only if the prior policy document was captured first.
• Blast radius. How many resources, accounts and dependent services can one triggering event touch? An action safe against one resource is not automatically safe against four hundred fired in the same minute.
• Confidence. What is the false-positive rate, and is the signal resistant to attacker influence? A finding an attacker can generate at will is a finding an attacker can use to drive your automation.
• Authorization. Has a named owner agreed, in advance and in writing, that this action may be taken without asking?

Figure 1 — Decision flowchart routing a security finding to auto-remediation, auto-remediation with notification, human approval, or manual handling, based on reversibility, blast radius, resistance to attacker influence, and pre-agreed authorization.

Applied to real finding classes, the matrix is less permissive than most teams expect.

The two rows worth arguing about are DeleteTrail and the crypto-mining instance. Auto-recreating a deleted trail looks obviously correct and is not: CloudTrail permits five trails per Region and the quota cannot be increased, so recreating on every deletion converts a cheap API call into permanent consumption of a scarce resource. Stopping a compromised instance looks obviously correct and destroys memory an investigation may need. In both, the reflex and the engineering diverge.

Remediation is not containment

These two words are used interchangeably in most internal documentation, and the conflation is how teams destroy evidence.

The AWS Security Incident Response Guide defines containment as "the process or implementation of a strategy during the handling of a security event that acts to minimize the scope of the security event and contain the effects of unauthorized usage within the environment," in three categories: source containment, which filters or routes to prevent access from a source; technique and access containment, which removes access to affected resources; and destination containment, which filters or routes to prevent access to a target.

None of those restore a desired configuration. That is the point. Containment reduces reachability while preserving state. Remediation restores a desired state and, in doing so, overwrites the attacker's.

The rule CloudSecOps applies: during a suspected active intrusion, prefer containment. Remediate configuration drift, not intrusions. A public S3 bucket found by a posture scan is drift and can be remediated. A public S3 bucket created ninety seconds after an unfamiliar role assumed a privileged identity is an intrusion artifact, and removing the policy destroys the timeline while leaving the attacker's access intact.

One citation worth checking in your own playbooks: NIST SP 800-61 reached Revision 3 in April 2025, retitled "Incident Response Recommendations and Considerations for Cybersecurity Risk Management: A CSF 2.0 Community Profile." The AWS Well-Architected security pillar still enumerates detection, analysis, containment, eradication and recovery. Both are current. Playbooks citing "the NIST incident response lifecycle" as current NIST guidance should be checked against Revision 3 rather than Revision 2 — the phase vocabulary remains workable, the citation may not be.

Prevention first: what an SCP removes from the problem

For the CloudTrail-disruption case specifically, most of the remediation problem is deletable.

Organization trails. A trail created in the management account, or by a delegated administrator, that applies to every account in the organization. The documentation is direct: users in member accounts do not have sufficient permissions to delete organization trails, turn logging on or off, change what types of events are logged, or otherwise change an organization trail in any way.

That sentence removes the original threat model for every member account. If your only trail is an organization trail owned by the management account, a compromised principal in a workload account cannot stop it, and there is nothing for a remediation loop to remediate.

The residual. The prevention story has named limits.

• A CloudTrail delegated administrator can delete organization trails — the documented capability table lists creating, updating and deleting them. An organization may register up to three. Compromise of a delegated admin account is compromise of the trail.
• The management account owns the resource and is not constrained by SCPs at all. SCPs do not affect users or roles in the management account, and do not affect any service-linked role.
• An SCP grants nothing. Effective permissions are the intersection of what every policy on every parent in the path from the root allows.
• organizations:LeaveOrganization, called successfully, removes an account from the organization trail's scope — a trail-disruption technique the original watch list cannot see, because it is not a CloudTrail API call.
• None of this exists outside Organizations. In a standalone account, remediation is not the second-best control; it is the only automated one available, and the guardrails below matter more, not less.

Service control policies are the right prevention primitive here. Resource control policies are not. RCPs restrict access to resources across an enumerated list of resource-holding services: S3, KMS, STS, SQS, Secrets Manager, DynamoDB, CloudWatch Logs, ECR and similar. CloudTrail is not on that list, and the list is revised as services are added, so read it rather than trusting any count published in an article. An RCP cannot stop a member-account principal calling cloudtrail:StopLogging. What it is genuinely good for here is the log archive bucket and the KMS key that encrypts it, the resources an attacker attacks when the trail is out of reach. Two documented gaps apply even there: RCPs do not apply to AWS managed KMS keys, and they do not restrict kms:RetireGrant.

AWS's rollout advice for RCPs is the discipline this article demands for remediation code: do not attach at the organization root without testing the impact, and review CloudTrail logs for AccessDenied errors before widening scope.

Figure 2 — Layered control diagram: an SCP denying the CloudTrail stop-logging call in a member account, an organization trail owned by the management account that member accounts cannot modify, and a remediation loop handling only what passes both gates.

Illustrative — a service control policy denying CloudTrail disruption and log-archive tampering in member accounts, with a single named break-glass exemption. Not executed; not attached to a live organization during writing. Account IDs and key IDs are redacted. Attach to one non-production OU first and review AccessDenied events before widening scope.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyCloudTrailDisruption",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "cloudtrail:PutEventSelectors",
        "cloudtrail:PutInsightSelectors"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotEquals": {
          "aws:PrincipalArn": [
            "arn:aws:iam::123456789012:role/BreakGlassTrailAdmin"
          ]
        }
      }
    },
    {
      "Sid": "DenyMintingTheExemptRole",
      "Effect": "Deny",
      "Action": [
        "iam:CreateRole",
        "iam:UpdateAssumeRolePolicy",
        "iam:PutRolePolicy",
        "iam:AttachRolePolicy"
      ],
      "Resource": "arn:aws:iam::*:role/BreakGlassTrailAdmin"
    },
    {
      "Sid": "ProtectLogArchiveBucketAndKey",
      "Effect": "Deny",
      "Action": [
        "kms:ScheduleKeyDeletion",
        "kms:DisableKey",
        "kms:PutKeyPolicy",
        "s3:DeleteBucket",
        "s3:PutBucketPolicy",
        "s3:PutBucketVersioning",
        "s3:PutLifecycleConfiguration"
      ],
      "Resource": [
        "arn:aws:kms:*:123456789012:key/EXAMPLE-KEY-ID",
        "arn:aws:s3:::example-org-cloudtrail-archive",
        "arn:aws:s3:::example-org-cloudtrail-archive/*"
      ],
      "Condition": {
        "ArnNotEquals": {
          "aws:PrincipalArn": [
            "arn:aws:iam::123456789012:role/BreakGlassTrailAdmin"
          ]
        }
      }
    }
  ]
}

Four things about that policy, three of which are why it is not the version most teams write.

The exemption is pinned to one account, not to a name. aws:PrincipalArn and aws:PrincipalTag are the workable condition keys for carving a role out of an SCP deny. The tempting form is arn:aws:iam::*:role/BreakGlassTrailAdmin, matching a naming convention across every account. That form is a privilege-escalation path: any principal in any member account holding iam:CreateRole can create a role with exactly that name and step outside the deny. DenyMintingTheExemptRole closes the local-minting route; pinning the account ARN closes the rest. If the break-glass identity genuinely must exist in many accounts, key the exemption on a tag whose setting is itself denied by the same SCP, and accept that tag-setting permissions are now security-critical.

ProtectLogArchiveBucketAndKey overlaps with the RCP on purpose. The SCP stops principals in member accounts; the RCP, attached where the resources live, stops principals the SCP does not reach. Two ceilings on the same resources, not substitutes.

Denying cloudtrail:UpdateTrail will break pipelines. Any Terraform, CDK or CloudFormation run managing an account-local trail calls UpdateTrail on drift. Expect the deny to surface first as a failed deploy, not as a blocked attack. That is the change you are actually making, and it is why staged rollout is not optional.

Every use of the break-glass role should page someone. In normal operation the count is zero, which makes it one of the few security signals with no tuning problem.

AWS publishes SCP examples in the aws-samples/service-control-policy-examples repository, including a Deny-changes-to-security-services directory. The policy above is CloudSecOps-authored from documented primitives rather than reproduced from it, and is labelled illustrative accordingly.

What the original design got right, and what has changed

The original design was correct in its instinct and is incomplete against AWS as it exists now.

All eight original operations — StopLogging, StartLogging, UpdateTrail, DeleteTrail, CreateTrail, RemoveTags, AddTags, PutEventSelectors — remain real CloudTrail management-event API names, still logged by default. Two do not belong on a remediation path: AddTags and RemoveTags do not disrupt logging. StartLogging is the operation the remediator itself calls, which is the self-trigger hazard.

The larger problem is what the list does not cover. CloudSecOps recommends this three-tier replacement. The tiering is our analysis, not AWS guidance.

Disabling the KMS key that encrypts the log bucket is functionally equivalent to disabling the trail, and none of the original eight operations see it. That is the single largest gap in the inherited design.

Two capabilities exist now that the original could not have covered. Log file integrity validation produces an hourly digest file referencing the previous hour's log files and containing a hash of each, using SHA-256 for hashing and SHA-256 with RSA for signing; each digest also contains the digital signature of the previous digest, if one exists. That chain makes silent deletion detectable after the fact, and it is the argument for "detect and prove" over "auto-restore" wherever a log's evidentiary value matters more than its continuity. Network activity events, carrying an eventCategory of NetworkActivity, record API calls made through VPC endpoints. They are off by default, chargeable, configured through advanced event selectors, and supported for a source list that includes cloudtrail.amazonaws.com itself.

The event path, and why it fails exactly when you need it

This is the correction that matters most to the inherited design.

Events reaching the EventBridge default bus from CloudTrail carry the detail-type AWS API Call via CloudTrail. Three documented properties constrain everything built on that path:

1. API actions starting with List, Get or Describe are not matched by rules in the default ENABLED state, with the named exceptions of the STS actions GetFederationToken and GetSessionToken. Read-only management events are opt-in per rule: setting the rule state to ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS includes them. Reconnaissance is therefore off this path by default, not unreachable — if you want it, you turn it on per rule and pay for the volume.
2. A CloudTrail trail with logging enabled is required for those events to be recorded.
3. Delivery is best effort.

Separately, the CloudTrail quotas page states that CloudTrail does not send events over 256 KB to CloudWatch Logs or EventBridge.

Point two breaks the naive design. A loop whose trigger is AWS API Call via CloudTrail depends on the very trail the attacker is attacking. The first StopLogging is normally observable, because the trail was still logging when the call was made. Subsequent disruption in an account whose only trail is now stopped is not guaranteed to reach the bus. The pattern is a one-shot tripwire, not a durable control. Point three compounds it: best-effort delivery is acceptable for a control that runs continuously and converges, and not acceptable as the sole detection of a one-time, high-consequence event.

That gives the design rule this article is built around, and it generalises well beyond CloudTrail:

The detection path must not depend on the resource being defended.

An organization trail delivering to a bucket in a separate log archive account, with the remediation loop consuming events in a third security-tooling account, satisfies the rule. A single account-local trail feeding a local rule invoking a local Lambda does not: one API call removes detection, remediation and evidence at once.

One dependency in that shape is undocumented, and you should not assume it. The EventBridge documentation requires "a trail with logging enabled" but does not say whether an organization trail satisfies that requirement for events raised in a member account. If it does, the organization trail keeps the member account's EventBridge path alive even when a local trail is stopped — a second reason to prefer it. If it does not, that loop loses its trigger exactly as before. A test case should verify it directly: with only the organization trail logging, stop an account-local trail in a test member account and confirm the event reaches the bus. Treat the answer as environment-specific until you have run it.

Where full independence is not achievable, add a second, differently-shaped signal — an AWS Config periodic evaluation of cloudtrail-enabled, or a scheduled external check asserting recent object arrivals in the archive bucket. A control that only fires when the thing it protects is healthy is not a control.

Comparing the mechanisms honestly

Five AWS mechanisms are used for this, and their summaries obscure real differences in latency, reach and capability.

Four facts in that table routinely surprise people.

Security Hub automation rules cannot remediate anything. The documentation opens by saying you use them to automatically update findings. The action set is Confidence, Criticality, Note, RelatedFindings, Severity, Types, UserDefinedFields, VerificationState and Workflow. Nothing there touches a resource. Remediation driven from Security Hub goes out through EventBridge, using Security Hub Findings - Imported for the automatic path and Security Hub Findings - Custom Action for the human-initiated one. A custom action covers up to 20 findings at a time, each sent as a separate event; an account may create up to 50, and where findings are managed from the aggregation Region the documentation tells you to create them there.

AWS Config auto-remediation can fire against compliant resources. From the documentation: any noncompliant resource updated between snapshot schedules continues to be remediated based on the last known compliance data snapshot, which means auto remediation can be initiated even for compliant resources, because the bootstrap processor uses a database that can hold stale evaluation results. Retries occur only on failure and within a specified window, governed by the MaximumAutomaticAttempts and RetryAttemptSeconds fields of the remediation configuration. Design for an action that may be applied to a resource already in the desired state — which is another way of saying it must be idempotent.

Config is the wrong trigger for CloudTrail disruption specifically. The managed rule cloudtrail-enabled, identifier CLOUD_TRAIL_ENABLED, is a periodic rule, not a configuration-change rule. Using it as a tripwire concedes evaluation-interval latency by design. It remains useful as the independent second signal above, precisely because its path does not depend on EventBridge.

GuardDuty's CloudTrail finding is Low severity, and repeats are delayed. Stealth:IAMUser/CloudTrailLoggingDisabled carries a default severity of Low, and is triggered by a successful deletion or update of a trail, or by deletion of an S3 bucket storing logs for a GuardDuty-associated trail. New findings reach EventBridge in near real time, but GuardDuty aggregates subsequent occurrences of a finding type into six-hour intervals by default, configurable by the administrator account to 15 minutes or one hour. So any remediation gated on a severity floor, commonly detail.severity at 7 or above, ignores this finding forever, and a GuardDuty-triggered remediator is a second line rather than the primary tripwire.

Building versus adopting. For findings that map to a published standard, adopting usually wins. Automated Security Response on AWS, the AWS Solutions implementation, stood at v3.1.8, dated 2026-07-28 in its repository change log. Its architecture is the one this article would otherwise argue you toward: Security Hub findings aggregated in a delegated administrator account start a Step Functions workflow, which invokes a remediation SSM Automation document in the member account holding the resource. It ships playbooks for AWS Foundational Security Best Practices v1.0.0, CIS AWS Foundations Benchmark v1.2.0, v1.4.0 and v3.0.0, PCI DSS v3.2.1, NIST SP 800-53 Revision 5, and a consolidated Security Control playbook. Adopt it for standards-mapped findings and read its runbooks. A bespoke loop earns its keep only for the residual — and CloudTrail disruption where prevention cannot reach is exactly that.

That change log is also the best available evidence for what this automation costs to own. Between 2026-03-03 and 2026-07-28 the solution shipped six releases, most of them dependency-vulnerability upgrades, and the 3.1.8 entry records granting ssm:StartAutomationExecution on the document/ resource form so remediations would keep running after an SSM API change replaced the automation-definition/ form. An AWS-maintained remediation solution needed an IAM policy change to survive an AWS API change. Your bespoke loop will need the same class of maintenance, and nobody publishes a change log for it.

Guardrails that make a remediation loop safe to run

An automation holding a privileged role, with no ceiling on how many times it can act, is an availability risk the organisation built itself. These eleven are the operational core.

Four of these deserve more than a table row.

Idempotency, and the way it disappears silently. The Powertools for AWS Lambda idempotency utility computes a key from the Lambda function name, the fully-qualified name of the decorated function, and a hash of the payload or of the parts you select, then persists it to DynamoDB with partition key id and TTL attribute expiration. Records carry status INPROGRESS, COMPLETE or EXPIRED, the default expiry is 3,600 seconds, and concurrent duplicates raise IdempotencyAlreadyInProgressError. Call register_lambda_context so a timed-out invocation does not hold the key for the full expiry window.

Choose the keyed subset deliberately, then check that it resolves. Two behaviours exercised during review against Powertools 3.31.1, outside AWS, with a stub persistence layer, decide whether this control works at all:

• Keying on a payload carrying the CloudTrail event id produced two persisted keys and two executions for what was semantically one action. Keying on the tuple of account, Region, trail and operation produced one key and one execution. That is the whole argument for keying on the action rather than the event, reproducible in about thirty lines.
• With the default raise_on_no_idempotency_key=False, a payload from which the configured JMESPath resolved nothing ran on every call with no persistence and no error — only a UserWarning saying the persistence layer was skipped. The control was absent while appearing configured. Set raise_on_no_idempotency_key=True.

A further wrinkle: with a multi-field key expression, a payload missing one field still produces a key and still does not raise, because something resolved. Validate the fields before handing them to the decorator rather than relying on the library to notice.

Concurrency and the kill switch are the same control. Reserved concurrency bounds normal operation and, set to zero, stops the function: the documented behaviour is that to intentionally throttle a function you set its reserved concurrency to 0, which stops it processing any events until you remove the limit. You can reserve up to the unreserved account concurrency minus 100. Set a real ceiling — a remediator that legitimately needs more than single-digit concurrency is remediating something a policy should have prevented. On the SSM side, Automation rate control provides concurrency targets and error thresholds, against a limit of 25 concurrent rate-control automations per account with up to 1,000 queued.

The DLQ must be alarmed, not merely configured. EventBridge retries a target for up to 24 hours and up to 185 times with exponential backoff and jitter. Some events reach the DLQ with no retry at all, because no retry helps until someone fixes the cause: the documented examples are missing permissions to a target, a target that no longer exists, and a target that cannot be found through an invalid address or DNS failure. DLQ messages carry RULE_ARN, TARGET_ARN, ERROR_CODE, ERROR_MESSAGE, EXHAUSTED_RETRY_CONDITION and RETRY_ATTEMPTS, enough to triage without reproducing. Only standard SQS queues are supported. A DLQ with no alarm converts a loud failure into a silent one.

"No inverse, no ship." Before an action is permitted in the auto or auto-with-notify tiers, its inverse must exist as code and be recorded in the ledger entry before the forward action runs. If the inverse requires state the forward action destroys, such as the prior bucket policy, trust document or event selectors, capturing that state is part of the action, not an enhancement to it. An action whose inverse cannot be written is a manual action wearing a costume.

Rollback then exists at three layers, and teams routinely build only the third. Layer one is the action's inverse, executable from the ledger record alone. Layer two is the kill switch: reserved concurrency at zero and the rule disabled, reversible in seconds by on-call, requiring no pipeline. Layer three is the deployment revert — the slowest, and the wrong tool at 03:00.

The remediation role is a target

The remediation identity holds standing permission to change security-relevant configuration across accounts, and it runs unattended. An attacker who reaches it gets a privileged, pre-authorised, cross-account change mechanism that is expected to make changes and therefore generates no anomaly by acting.

• Scope to the action set, not the service. The CloudTrail remediator gets cloudtrail:StartLogging, cloudtrail:PutEventSelectors and cloudtrail:GetTrailStatus on named trail ARNs. Not cloudtrail:*, and not the managed AWSCloudTrail_FullAccess policy, which AWS's own CloudTrail security best practices advise limiting.
• Separate roles per action class. One role that can re-enable trails, remove public bucket policies, revoke security-group rules and delete access keys is one compromise away from being a general-purpose destruction tool.
• The role must not be assumable by the workloads it remediates. Otherwise the blast radius of any application compromise includes the remediation surface. Trust policies should name the automation's execution principal and nothing else, with an aws:SourceAccount or equivalent condition where the integration supports it.
• Constrain it with the same SCPs it enforces, then exempt it explicitly. An implicit exemption, such as placing the automation account in an OU where the SCP is not attached, produces an account with no policy ceiling at all.
• Alert on the role's own behaviour. Every action it takes is a management event in CloudTrail. Baseline its call profile. Calls outside that profile, calls from an unexpected source, and any use of the break-glass role are the signals that the automation itself has been turned.
• Credentials must be short-lived and non-exportable. Lambda execution roles and SSM service roles satisfy this. Long-lived IAM user access keys running remediation scripts on a schedule from outside AWS do not, and are the version of this pattern most often found in an assessment.

When a human must approve

An approval gate that approves everything is worse than no gate: it adds latency and produces a false record of deliberation.

What the approver must be shown, in the notification itself rather than behind a console link: the exact diff the action will apply; the resource count in scope; the stated inverse; the confidence basis for the finding; and the deadline with its default outcome. An approver who has to open three consoles will approve on the basis of who sent the request.

The mechanics, with their real constraints. The SSM Automation aws:approve action supports a maximum of 10 approvers; MinRequiredApprovals defaults to 1 and cannot exceed the approver count; the SNS topic named in NotificationArn must have a title prefixed with "Automation"; Message is capped at 4,096 characters; the timeout defaults to 7 days with a maximum of 30 days; outputs are ApprovalStatus, holding Approved, Rejected or Waiting, and ApproverDecisions. The constraint that decides most architectures: aws:approve does not support multi-account and Region automations. For organization-wide response, that rules it out.

Step Functions is the alternative. The .waitForTaskToken pattern pauses a state until the token is returned, up to the one-year execution quota, and works across accounts because the callback is an API call rather than a console interaction. AWS documents a callback sample using SQS with SNS and Lambda, and recommends a HeartbeatSeconds interval to avoid executions that wait indefinitely when the callback never arrives.

Anti-rubber-stamp mechanics. CloudSecOps practice rather than documentation, applied as a set:

• Default-deny on timeout. The Catch on the approval state routes a timeout to the deny path, never to proceed. A gate that proceeds on silence is a delay, not a gate.
• A short deadline. Seven days is far too long for a security action. If the action matters, the deadline is tens of minutes and the fallback is containment. If seven days is acceptable, the finding did not need automation.
• Approver rotation. A fixed single approver becomes a routing rule. Rotate across a named group and record who decided.
• Approval-rate monitoring. Track the proportion approved, per approver and per action type. A rate at or near 100% over a meaningful sample says the gate has degraded, and it is measurable without asking anyone.
• Sampled post-hoc audit. Pull a small random sample each month and reconstruct whether the approver had enough information to decide. This finds notification-content problems that rate monitoring cannot.

Worked pattern: the CloudTrail disruption loop, rebuilt

This applies to the residual after prevention: account-local trails an organization trail does not cover, accounts outside Organizations, and the management account, where SCPs do not apply. If your entire estate sits inside an organization trail with the SCP above attached, you do not need this loop, and building it anyway adds a privileged identity for no coverage gain.

Figure 3 — Event-driven remediation architecture: CloudTrail management events cross a cross-account EventBridge bus into a Step Functions workflow with a decision gate, an optional approval step, an SSM Automation in the target account, dead-letter queues on the rule and the state machine, and an action ledger in the log archive account.

The rule, with loop protection

Loop protection belongs in the event pattern, not only in the handler. A handler-side check works, but it still costs an invocation, a log line and a unit of concurrency — under a flood, those costs are the failure. Implement both.

The obvious pattern is also subtly wrong. Excluding the remediator by putting anything-but on userIdentity.sessionContext.sessionIssuer.arn narrows the rule to events that have that field, because EventBridge matches on the presence of the fields a pattern names. Calls made with long-lived IAM user access keys, and calls made by the account root, carry no sessionIssuer. A rule written the obvious way therefore ignores exactly the principal type most likely to be holding a leaked static credential. The $or below restores that coverage.

Illustrative — an EventBridge event pattern matching disruptive CloudTrail operations while excluding the remediation role's own calls and still matching principals that have no session issuer. Not deployed during writing. Validate with aws events test-event-pattern against a captured event from your own trail before use, and validate the role ARN prefix against your own naming.

{
  "source": ["aws.cloudtrail"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["cloudtrail.amazonaws.com"],
    "eventName": [
      "StopLogging",
      "DeleteTrail",
      "UpdateTrail",
      "PutEventSelectors",
      "PutInsightSelectors"
    ],
    "$or": [
      {
        "userIdentity": {
          "sessionContext": {
            "sessionIssuer": {
              "arn": [
                {
                  "anything-but": {
                    "prefix": "arn:aws:iam::123456789012:role/cloudsecops-remediator"
                  }
                }
              ]
            }
          }
        }
      },
      {
        "userIdentity": {
          "sessionContext": {
            "sessionIssuer": {
              "arn": [{ "exists": false }]
            }
          }
        }
      }
    ]
  }
}

EventBridge supports prefix, suffix, anything-but, equals-ignore-case, wildcard, numeric, exists, cidr and $or, and anything-but composes with prefix. $or throws InvalidEventPatternException past 1,000 rule combinations, which this is nowhere near. exists works only on leaf nodes, which is why the second branch tests arn rather than sessionIssuer.

Deliberately absent: StartLogging, CreateTrail, AddTags and RemoveTags. Those go to a second rule on the notify path. Mixing tiers in one rule is how a notify-only signal ends up invoking a privileged action.

Figure 4 — Sequence diagram contrasting a remediation loop that excludes its own principal from its trigger pattern with one that does not, and re-invokes itself indefinitely.

The handler

Illustrative — a remediation handler showing idempotency, a dry-run gate, capture of the prior state, an explicit inverse, and a structured ledger record. The idempotency behaviour was exercised during review against Powertools for AWS Lambda 3.31.1 outside AWS, with a stub persistence layer standing in for DynamoDB; the AWS API calls were not executed and the error paths are not exhaustive. Pin your Powertools version and re-check the decorator surface before adapting.

import os
import boto3
from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent_function,
)

logger = Logger(service="cloudtrail-remediator")
persistence = DynamoDBPersistenceLayer(table_name=os.environ["IDEMPOTENCY_TABLE"])
ssm = boto3.client("ssm")

# The remediator's own role ARN, supplied by the deployment. Compared exactly
# below -- never as a substring, since role names are attacker-choosable.
REMEDIATOR_ROLE_ARN = os.environ["REMEDIATOR_ROLE_ARN"]

# Key on the action tuple, not the raw event: the event id makes every
# redelivery unique. raise_on_no_idempotency_key is not optional -- at the
# default of False, an expression that resolves to nothing skips the
# persistence layer entirely and the action runs on every delivery.
IDEMPOTENCY = IdempotencyConfig(
    # `mode` belongs in the key. Without it, a dry run and a live run of the
    # same action collapse to one idempotency record: whichever executed first
    # wins for the whole TTL, so flipping the flag from dry-run to enforce is
    # silently ignored for an hour.
    event_key_jmespath="[account, region, trail, operation, mode]",
    raise_on_no_idempotency_key=True,
    expires_after_seconds=3600,
)

# CloudTrail does not use one request-parameter name for the trail. The API
# input shapes differ -- StopLogging and UpdateTrail take Name, while
# PutEventSelectors takes TrailName -- and requestParameters follows them.
# Reading "name" for every operation yields None for half the watch list and
# a silent no-op that reports success.
TRAIL_PARAM = {
    "StopLogging": "name",
    "UpdateTrail": "name",
    "PutEventSelectors": "trailName",
}

# Forward operation -> inverse. An operation absent from this map may not ship
# in the auto tier. "executable" is False where an inverse exists on paper but
# must never run unattended: nothing about an incident makes re-stopping a
# trail the right automated move. The rule earns its keep on actions that
# destroy state, which is why prior state is captured either way.
INVERSE = {
    "StartLogging": {
        "operation": "StopLogging",
        "executable": False,
        "reason": "reversing a logging restore is never a correct unattended action",
    },
}

MAX_TARGETS_PER_EXECUTION = 5

def dry_run_enabled() -> bool:
    param = ssm.get_parameter(Name="/cloudsecops/remediation/dry_run")
    return param["Parameter"]["Value"].lower() == "true"

def target_client(account_id: str, region: str):
    sts = boto3.client("sts")
    creds = sts.assume_role(
        RoleArn=f"arn:aws:iam::{account_id}:role/CloudSecOpsRemediationTarget",
        RoleSessionName="cloudtrail-remediation",
    )["Credentials"]
    return boto3.client(
        "cloudtrail",
        region_name=region,
        aws_access_key_id=creds["AccessKeyId"],
        aws_secret_access_key=creds["SecretAccessKey"],
        aws_session_token=creds["SessionToken"],
    )

@idempotent_function(
    data_keyword_argument="action",
    config=IDEMPOTENCY,
    persistence_store=persistence,
)
def remediate(action: dict, ctx: dict) -> dict:
    # Only "action" forms the idempotency key. "ctx" carries the correlation id
    # and the actor, which must not change the key -- two redeliveries of one
    # API call are one action. A consequence worth knowing: on a duplicate,
    # the cached record returned is the FIRST invocation's, correlation id
    # included, so reconstruct from the ledger rather than from the response.
    client = target_client(action["account"], action["region"])

    # Trail names resolve only in the trail's home Region. GetTrailStatus with
    # a bare name outside that Region will fail; pass an ARN where you have one.
    pre_state = client.get_trail_status(Name=action["trail"])
    record = {
        "correlation_id": ctx["correlation_id"],
        "actor": ctx["actor_arn"],
        "operation": action["operation"],
        "target": action["trail"],
        "pre_state": {"IsLogging": pre_state["IsLogging"]},
        "decision_path": ctx["tier"],
        "dry_run": ctx["dry_run"],
    }

    # Route by the operation that fired. StartLogging only undoes StopLogging.
    # Applying it to an UpdateTrail or PutEventSelectors event restores nothing
    # -- the trail is still logging, so the post-state check passes and the
    # ledger records "applied" while the tampered configuration stands. A
    # remediation that reports success without restoring anything is worse than
    # none, because it closes the alert.
    if action["operation"] != "StopLogging":
        record["outcome"] = "escalated_no_safe_inverse"
        record["reason"] = (
            "restoring this operation requires the trail's known-good "
            "configuration; StartLogging is not its inverse"
        )
        logger.warning("remediation_escalated", extra=record)
        return record

    record["forward_action"] = "StartLogging"
    record["inverse"] = INVERSE["StartLogging"]
    record["restore_from"] = {"IsLogging": pre_state["IsLogging"]}

    if ctx["dry_run"]:
        record["outcome"] = "dry_run_no_change"
        logger.info("remediation_ledger", extra=record)
        return record

    client.start_logging(Name=action["trail"])
    post_state = client.get_trail_status(Name=action["trail"])
    record["post_state"] = {"IsLogging": post_state["IsLogging"]}
    record["outcome"] = "applied" if post_state["IsLogging"] else "failed"
    logger.info("remediation_ledger", extra=record)
    return record

@logger.inject_lambda_context
def handler(event, context):
    IDEMPOTENCY.register_lambda_context(context)

    detail = event["detail"]
    operation = detail["eventName"]
    identity = detail.get("userIdentity", {})
    actor_arn = (
        identity.get("sessionContext", {}).get("sessionIssuer", {}).get("arn")
        or identity.get("arn")
        or "unknown"
    )

    # Second line of loop protection; the event pattern is the first. This one
    # also covers principals with no sessionIssuer, which the pattern reaches
    # only through its $or branch.
    #
    # Exact match, not a substring test. Role names are attacker-choosable: a
    # principal that merely CONTAINS the remediator's name — say a role called
    # `cloudsecops-remediator-bypass` created by someone with iam:CreateRole —
    # would be silently dropped here, which turns loop protection into an
    # evasion primitive. Compare the whole ARN against a configured value.
    if actor_arn == REMEDIATOR_ROLE_ARN:
        logger.warning("self_triggered_event_dropped", extra={"actor": actor_arn})
        return {"status": "dropped_self_trigger"}

    if operation not in TRAIL_PARAM:
        logger.info("notify_only_tier", extra={"eventName": operation})
        return {"status": "notify_only"}

    raw = detail.get("requestParameters", {}).get(TRAIL_PARAM[operation])
    targets = [raw] if isinstance(raw, str) else (raw or [])
    if not targets:
        # Fail loudly. An unrecognised request shape is a change in the event,
        # not an absence of work, and must not be reported as success.
        logger.error("no_target_resolved", extra={"eventName": operation})
        raise RuntimeError("no target resolved from requestParameters")
    if len(targets) > MAX_TARGETS_PER_EXECUTION:
        # Fail closed. One event resolving to many targets is either a bug or
        # an attempt to use the automation as an amplifier.
        logger.error("blast_radius_exceeded", extra={"count": len(targets)})
        raise RuntimeError("blast radius exceeded; refusing to act")

    dry_run = dry_run_enabled()
    results = [
        remediate(
            action={
                "account": event["account"],
                "region": event["region"],
                "trail": target,
                "operation": operation,
            },
            ctx={
                "correlation_id": event["id"],
                "actor_arn": actor_arn,
                "tier": "disruptive",
                "dry_run": dry_run,
            },
        )
        for target in targets
    ]
    return {"status": "processed", "results": results}

Four details carry the design. The idempotency key is the action tuple and only the action tuple, so redelivery of one triggering event does not produce a second StartLogging; the correlation id and actor travel in a second argument the key ignores. The trail parameter is looked up per operation, because reading name for PutEventSelectors returns nothing and produces a silent no-op that reports success. Both the empty-target and blast-radius cases raise rather than return, so failure is visible in the DLQ instead of silently partial. The dry-run flag is read from Parameter Store at invocation rather than baked into an environment variable, so on-call can flip it without a deployment.

The approval branch

Pseudocode — an Amazon States Language fragment showing an approval that defaults to deny on timeout. State names, ARNs and the payload contract are placeholders.

{
  "AwaitApproval": {
    "Type": "Task",
    "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
    "Parameters": {
      "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/approvals",
      "MessageBody": {
        "TaskToken.$": "$$.Task.Token",
        "diff.$": "$.proposed_diff",
        "target_count.$": "$.target_count",
        "inverse.$": "$.inverse",
        "confidence.$": "$.confidence_basis",
        "deadline_minutes": 30
      }
    },
    "TimeoutSeconds": 1800,
    "HeartbeatSeconds": 600,
    "Catch": [
      {
        "ErrorEquals": ["States.Timeout"],
        "Next": "DenyAndContain"
      }
    ],
    "Next": "ApplyRemediation"
  }
}

The Catch is the whole control: a timeout routes to DenyAndContain, not to ApplyRemediation. Without an explicit timeout the token waits until the one-year execution quota, which is why TimeoutSeconds is mandatory rather than optional.

Failure handling and the kill switch

Illustrative — AWS CLI commands attaching an alarm to a dead-letter queue and stopping the remediator two ways. Not executed during writing. Substitute your own ARNs; the alarm action must point at a topic on-call actually receives.

# Alarm when anything lands in the DLQ. A DLQ with no alarm is a silent failure.
aws cloudwatch put-metric-alarm \
  --alarm-name cloudsecops-remediation-dlq-nonempty \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=cloudsecops-remediation-dlq \
  --statistic Maximum --period 60 --evaluation-periods 1 \
  --threshold 0 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:security-oncall

# Kill switch. Reserved concurrency of 0 stops the function processing events
# without deleting it, without a deployment, and reversibly.
aws lambda put-function-concurrency \
  --function-name cloudsecops-cloudtrail-remediator \
  --reserved-concurrent-executions 0

# Second kill switch, one layer up: stop events reaching the target at all.
aws events disable-rule \
  --event-bus-name cloudsecops-security-bus \
  --name cloudtrail-disruption-remediate

Both switches must be exercisable by on-call without a code deploy and without a change-approval ticket. Put the exact commands in the runbook and time them during a game day: a kill switch that takes twenty minutes to locate is not one. Neither undoes actions already applied — that is layer one's job.

Triage from the DLQ using the message attributes rather than by reproducing the event. A message with no retries recorded points at the wiring, because permissions, missing targets and unresolvable addresses bypass retry entirely. EXHAUSTED_RETRY_CONDITION with a high RETRY_ATTEMPTS means the function was reached and failed repeatedly, which points at the handler or the downstream API.

The action ledger

Every automated action produces one structured record, written to a log group in the log archive account with a retention policy the workload accounts cannot change.

Illustrative — a CloudWatch Logs Insights query over the structured ledger records. Not executed. Adjust the log group and field paths to match your logger's output shape.

fields @timestamp, correlation_id, actor, operation, target, outcome, decision_path
| filter @message like /remediation_ledger/
| filter outcome != "dry_run_no_change"
| stats count(*) as actions by actor, operation, outcome
| sort actions desc
| limit 100

A note on where this store lives. AWS CloudTrail Lake is no longer open to new customers starting on 31 May 2026; existing customers continue as normal, but the service receives only critical bug fixes and security updates, and AWS recommends migrating Lake data to Amazon CloudWatch, stating that data prior to 2023 will not be migrated. CloudTrail itself, meaning trails, Insights and aggregated events, is explicitly unaffected. If you already run Lake, keep querying your ledger there. If you are building now, target CloudWatch Logs, or S3 with Athena. Guidance telling you to stand up a new event data store as your remediation evidence store is out of date.

Testing it

An untested remediation is an outage with a deployment pipeline. Test in an account you own and are authorised to test, with a change record, outside the change-freeze windows that apply to workloads sharing that account. None of these should run against a production account without that authorisation.

• Unit tests on the decision gate. Table-driven, one case per tier, asserting the outcome and that notify-tier events never reach the action path.
• Event-pattern tests. Run aws events test-event-pattern against captured events for every principal type you expect: a role session, an IAM user with static keys, and the account root. The IAM-user case is the one that fails when the $or branch is missing, and it fails silently.
• Event replay. Capture real CloudTrail events from a test account, store them as fixtures, replay them, and assert exactly one StartLogging per action. This is the idempotency test, and the one most likely to fail first because of a key that includes the event id. Include a fixture whose keyed fields are absent and assert that the handler raises rather than proceeding unprotected.
• Loop test. Deploy with the exclusion removed, trigger one StopLogging, count invocations, confirm it runs away, restore the exclusion, confirm it stops after one. Keep reserved concurrency low so the runaway costs nothing.
• Concurrency and blast-radius test. Inject several hundred synthetic events. Assert that reserved concurrency holds, that the blast-radius check raises rather than truncating, and that the excess lands in the DLQ.
• Quota-exhaustion test. If your design recreates trails, create trails in a test Region until you hit the limit of five per Region and observe the failure mode. This is the test that usually changes the design.
• Permission-revocation test. Remove the target role's permission; confirm the event reaches the DLQ with no retries and the alarm fires. Trust policies drift, so this is the failure path most likely in production.
• Kill-switch drill. Timed, during a game day, executed by whoever is actually on call rather than by the automation's author.
• Game day. Disrupt the trail; observe detection, decision, action and ledger record; then execute the inverse from the ledger. The AWS Well-Architected security pillar puts running simulations at SEC10-BP07 for the same reason.

The attacker who knows your automation exists

Assume the automation is known: its rules are in a repository, its role names follow a convention, and its behaviour is observable by anyone who can watch what happens after a triggering action. Design on that assumption, as cryptography does about algorithms. What must stay non-public is the exception register — which principals and resources are carved out of the deny, and until when.

Self-inflicted denial of service through quota exhaustion. CloudTrail permits five trails per Region and the quota cannot be increased. An automation that answers DeleteTrail by creating a replacement converts a repeated, cheap API call into permanent consumption of a scarce resource; driven far enough, the legitimate trail cannot be recreated and the automation built to preserve logging has made logging unrecoverable without support intervention. This is a reasoned failure mode derived from two documented facts, the quota and the auto-recreate design, and not an incident CloudSecOps has observed. The mitigation is to check remaining quota before creating, cap creations per account and Region per day, and route DeleteTrail to the approval tier.

Amplification. Any automation doing work proportional to attacker-controlled input is an amplifier: if one API call causes a role assumption, several API calls and several log records, a few thousand cheap calls produce a cost and rate-limit event in the security tooling account. The mitigations are the concurrency ceiling, the per-execution cap, and idempotency.

Noise as cover. An automation that generates alerts generates alert fatigue on demand; flooding the loop with low-value triggers buries the one action that mattered. Deduplicate notifications on the idempotency key, alert on rate changes rather than every action, and keep the ledger separate from the notification stream so reconstruction does not depend on what anyone read at the time.

The remediation role as a lateral path. An attacker who compromises it acquires an identity whose job is making privileged cross-account changes, so the changes generate no anomaly. Detection has to come from the role's call profile, not from the fact that it made a call.

Triggering the automation to learn it. An attacker who can cause a benign triggering event learns the response's timing, scope and identity by observation, at no risk. Not preventable, and not worth treating as a secret.

Common mistakes

Each of these appears in assessments often enough to be worth naming. All are covered in mechanism above; this is the index.

Trade-offs and limits

This design costs a Step Functions workflow, a DynamoDB table, two queues, a cross-account bus, an alarm, and ongoing maintenance of a privileged role across every account in scope. That is a real system with a real on-call burden, worth building only where prevention cannot reach and the residual matters. The ASR change log above is the honest picture of the carrying cost: monthly dependency work, plus occasional breakage when an AWS API moves underneath you.

If you are a small team. Do not build this. The first six items of the action list below, from inventory through to adjacent-asset alerts, deliver most of the risk reduction with no code and no privileged automation identity. Stop there. A three-person team that ships a cross-account remediation loop has added an availability risk and an on-call rotation it cannot staff, in exchange for a residual it may not have.

If you have no Organizations. There is no organization trail and no SCP, so prevention is unavailable and the loop carries the whole weight — with its detection path necessarily dependent on the resource it defends, the shape this article argues against. The honest recommendation is to move to Organizations and treat the loop as an interim measure with known limits.

If a managed provider runs your response. The guardrail requirements do not move; the questions do. Ask which of the eleven guardrails their platform implements, what their kill switch is, how fast you can pull it, and whether their action ledger lands in an account you control. A remediation identity you cannot revoke is worse than one you built.

What this article does not resolve. Product naming is mid-flight: the Security Hub documentation describes "AWS Security Hub Cloud Security Posture Management (AWS Security Hub CSPM)" while the product and pricing pages present a single AWS Security Hub with Essentials, Threat Analytics and Extended plans. This article reports the disagreement rather than resolving it. Confirm which plan and console surface you operate against before mapping controls.

What the research could not verify. Lambda asynchronous invocation retry defaults, AWS Config remediation exception expiry behaviour, and the enumerated values of MaximumExecutionFrequency are not quoted here because they were not confirmed from primary documentation. The MITRE ATT&CK page for the relevant impair-defences sub-technique did not render for retrieval, so no technique title is asserted. Whether an organization trail satisfies the EventBridge trail requirement for member accounts is undocumented and is flagged above as a test rather than a fact. No third-party practitioner analysis or incident reporting was consulted; the evidence base is first-party AWS documentation, standards bodies, maintainer documentation and prior CloudSecOps publications. Nothing here is a claim about industry-wide prevalence.

What will go stale first. The CloudTrail Lake migration path is an active change. The Security Hub plan naming is mid-rename. The RCP supported-service list grows, and CloudTrail could join it. The ASR version moved during the writing of this article. The Powertools decorator surface is a library API, not an AWS contract. Treat every version and quota here as stated on the verification date, and re-check before building.

Minimum viable action list

In order. Each step is useful on its own, and each makes the next cheaper.

1. Inventory every trail in every account and Region. Record which are organization trails and which are account-local.
2. Create an organization trail delivering to a bucket in a dedicated log archive account, with log file integrity validation enabled.
3. Enumerate your CloudTrail delegated administrators — up to three, each able to delete organization trails. Reduce the set and alert on registration changes.
4. Attach the deny SCP to one non-production OU. Review AccessDenied events for a full business cycle before widening. Expect IaC pipelines to surface first.
5. Apply an RCP to the log archive bucket and its KMS key. CloudTrail is not an RCP-supported service; the bucket and the key are.
6. Alert on the adjacent-asset tier: KMS key state, archive bucket policy and lifecycle, and LeaveOrganization. Do this before writing any remediation code.
7. Only now, identify the residual: account-local trails, accounts outside Organizations, the management account. If the residual is empty, stop here.
8. Build the loop in dry-run mode. Run it for a full weekly cycle and read the intended actions.
9. Add the guardrails from the table above, and write the tests before enabling enforcement.
10. Enable enforcement on one account. Run the game day. Time the kill switch.
11. Widen by OU, not by organization, reviewing the ledger at each stage.
12. Put a review date on the exception register and on this design.

References

• AWS CloudTrail User Guide — current feature set and service names.
• Logging management events with CloudTrail — recorded control-plane operations and advanced event selector syntax.
• Logging network activity events with CloudTrail — the NetworkActivity category; off by default and chargeable.
• CloudTrail Lake service availability change — closed to new customers from 31 May 2026; migration to CloudWatch.
• AWS CloudTrail Lake — event data stores and retention limits.
• Creating a trail for an organization — member accounts cannot modify an organization trail.
• CloudTrail delegated administrator — maximum of three; the capability table showing trail deletion.
• CloudTrail quotas — five trails per Region, no adjustable quotas, 256 KB to EventBridge.
• CloudTrail security best practices — log file validation and limiting AWSCloudTrail_FullAccess.
• Validating CloudTrail log file integrity — hourly digests, SHA-256 hashing and signing, chained previous signature.
• EventBridge events from AWS services — the detail-type, List/Get/Describe exclusion, trail requirement, best-effort delivery.
• EventBridge rule event delivery and retries — 24 hours, up to 185 attempts, exponential backoff and jitter.
• Using dead-letter queues with EventBridge rules — no-retry conditions, message attributes, standard queues only.
• EventBridge content-based filtering — anything-but with prefix, exists on leaf nodes, $or limits.
• GuardDuty findings and EventBridge — near-real-time new findings; six-hour default aggregation for repeats.
• GuardDuty IAM finding types — Stealth:IAMUser/CloudTrailLoggingDisabled and its default Low severity.
• Security Hub CSPM automation rules — rules update findings; they do not change resources.
• Security Hub CSPM EventBridge integration types — detail-types and the 20-finding custom action limit.
• Security Hub CSPM custom actions — 50 custom actions; create in the aggregation Region.
• What is AWS Security Hub CSPM — the documentation-side product naming.
• AWS Security Hub pricing and plans — the Essentials, Threat Analytics and Extended plan structure.
• Security Hub CSPM CloudTrail controls — control severities and benchmark mapping.
• AWS Config remediation — remediation executes SSM Automation documents.
• Setting up automatic remediation in AWS Config — the stale-snapshot warning and retry-on-failure behaviour.
• AWS Config RemediationConfiguration API reference — the MaximumAutomaticAttempts and RetryAttemptSeconds fields.
• Evaluating resources with AWS Config rules — configuration-change, periodic and hybrid triggers.
• AWS Config managed rule cloudtrail-enabled — confirms CLOUD_TRAIL_ENABLED is periodic.
• AWS Systems Manager Automation — runbooks and rate control for concurrency and error thresholds.
• The aws:approve automation action — 10 approvers, 7-day default and 30-day maximum, no multi-account support.
• Step Functions service integration patterns — .waitForTaskToken, the one-year quota and HeartbeatSeconds.
• Step Functions callback pattern example — the SQS callback shape used for approvals.
• Service control policies — SCPs grant nothing and do not reach the management account or service-linked roles.
• Service control policy examples — the index pointing to the AWS samples repository.
• aws-samples service control policy examples repository — including a directory for denying changes to security services.
• Resource control policies — supported services, the exclusions, AWS managed KMS keys, kms:RetireGrant, staged rollout.
• IAM global condition context keys — aws:PrincipalArn and aws:PrincipalTag for policy exemptions.
• Lambda function concurrency — reserved concurrency of 0 as the documented way to throttle a function.
• Lambda asynchronous invocation — the async queue and on-failure destinations.
• Powertools for AWS Lambda idempotency utility — key construction, persistence, statuses, 3,600-second default, register_lambda_context.
• Automated Security Response on AWS — the reference architecture, findings to Step Functions to an SSM runbook.
• Automated Security Response on AWS change log — v3.1.8 dated 2026-07-28, release cadence, and the SSM resource-form fix.
• AWS Security Incident Response Guide: containment — the definition of containment and its three categories.
• AWS Well-Architected security pillar: incident response preparation — SEC10-BP01 through BP07, including simulations.
• NIST SP 800-61 Revision 3 — April 2025, framed as a CSF 2.0 Community Profile.
• Automated Remediation for CloudTrail Disruption — the earlier CloudSecOps article this piece descends from.
• AWS security use cases and automations — the companion CloudSecOps article on adjacent use cases.

Validity and revision

Every AWS behavioural claim here was verified against the primary documentation linked above on 2026-08-06, with a second verification pass on 2026-08-07 that corrected the Automated Security Response version, removed a resource-control-policy service count that no longer matched the documentation, and narrowed the NIST SP 800-61 Revision 3 characterisation to what the publication record supports. The Powertools idempotency behaviour described above was exercised against version 3.31.1 outside AWS; no AWS API call in this article was executed.

Version-dependent material. The CloudTrail Lake availability change is an active migration. The Security Hub product naming is mid-rename, with documentation and product pages out of step. The RCP supported-service list changes; CloudTrail is not on it today. The CloudTrail network activity event source list grows. Automated Security Response on AWS is cited at v3.1.8 (2026-07-28) and releases roughly monthly. Every quota stated — five trails per Region, 10 approvers, 20 findings per custom action, 50 custom actions, 185 EventBridge retry attempts, 3,600-second idempotency default — is as of the verification date.

Recommended review. A targeted re-check by 2026-11-06 covering the CloudTrail Lake migration path, the Security Hub plan naming and the ASR version pin; a full review by 2027-02-06 covering the RCP service list, the network activity event sources, the Security Hub control set and the Powertools API surface.