EBS Cross-Region Encrypted Snapshot Copy fails with status error

As part of our DR strategy, I had to start copying our EBS snapshots from us-west-2 to us-east-1.

My solution looks like this:

• In us-west-2, CloudWatch Rule that calls an SNS topic when an EBS snapshot is taken (also in us-west-2)
• SNS topic has a us-east-1 Lambda Subscriber which has the code to copy the EBS snapshot from source to destination region

In order to copy a snapshot into us-east-1, the Lambda function has to be hosted in us-east-1 as well.

A KMS CMK key is being used for EBS encryption. So the EBS snapshots have a KMS CMK in the source region and another CMK in the target us-east-1 region.

While I was writing the solution in terraform, I was running as admin so everything “worked”. But when I had to run it via our CI/CD pipeline — well, that’s when I got the “status” error. Nothing more.

This was so hard to troubleshoot! For one, EBS snapshot errors do not create any CloudWatch logs. Where could I find the reason for this error?

I knew my automation ran successfully as admin so I knew it had to do with permissions. So I sat there, combing through IAM permissions till I figured it out!

Hope this helps you out one day!

"kms:CreateGrant",
"kms:DescribeKey",
"kms:Encrypt",
"kms:GetKeyPolicy",
"kms:GenerateDataKeyWithoutPlaintext"
resources = [us-east-1-kms_key_arn, snapshot_us-west-2-source_kms_key_arn]

Pro Tip: The EBS Snapshot Copy Lambda role needs permissions to the source EBS KMS encryption key in its parent region as well as to the new EBS KMS encryption key in the target disaster recovery region.