AWS, please fix: dead lambda trigger reference

Cross-account operations: console refuses to delete SNS lambda trigger


Intro

Recently, after updating our application, the client subscribed to the newly created SNS topic using a lambda function (yes, I know… however, that’s not what this post is about). It turned out that it was not yet ready to handle the new data format, so the decision was made to remove the subscription and go back to using the data from the previous source. As is often the case, things were moving fast and data was leaking, so it was decided that the operations team would make this change manually and the developers would adjust the infrastructure code in the near future and release a patch that would automatically fix the problem. And so the problem was born…

1. Starting point
Using the automatically created infrastructure, the client subscribed its lambda function to incoming messages from the specified SNS topic. This can be found in two places from a client console perspective:

(111111111111 - our app account, 999999999999 - customer account)

SNS console subscription

SNS:

console subscription

Lambda console subscription

Lambda:

console subscription

2. Manually removing the subscription & trigger
Deleting the subscription from the SNS console works fine, but it does not delete the trigger in the Lambda console. Okay, so it will be done in a separate step. Here is the outcome:

Lambda trigger deletion error

Lambda:

trigger deletion error

Error: User: arn:aws:sts::999999999999:assumed-role/someRole/someUser is not authorized to perform: SNS:ListSubscriptionsByTopic on resource: arn:aws:sns:eu-west-1:111111111111:dead-lambda-trigger-reference-topic because no resource-based policy allows the SNS:ListSubscriptionsByTopic action

Neither the subscription nor the trigger has been removed. However, how to get rid of it? We were asked to provide the SNS:ListSubscriptionsByTopic action for a customer account. After a quick check, it turned out that in such a case, all subscriptions to that topic could be checked, which is highly undesirable. In other words: the customer can subscribe, but should not care who else is subscribed. The request was denied, but somehow you have to get rid of the trigger. I decided to take a deeper look into it.

3. Programmatic trigger removal
Google was partially helpful. I was able to find a post on StackOverflow where someone had a similar problem using the SDK. I thought it would be easiest to test these commands using the AWS CLI. So I created a test setup and started typing. Here is how I was able to solve this problem:

aws lambda get-policy --function-name dead-lambda-trigger-reference-lambda

The answer has a policy key and its content is important here:

{
  "Policy": "IMPORTANT JSON, look below",
  "RevisionId": "b8c770fa-e2b2-4b60-bc51-382dddd4e901"
}
{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "sns-eu-west-1-111111111111-dead-lambda-trigger-reference-topic",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:eu-west-1:999999999999:function:dead-lambda-trigger-reference-lambda",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:sns:eu-west-1:111111111111:dead-lambda-trigger-reference-topic"
        }
      }
    }
  ]
}

The Sid key of the desired Statement array object must be taken for the next call. If unmodified after creation, it has the following format:

sns-region-accountWhereTheTopicLivesIn-topicName

Obviously, there can be many objects in the statement array, but we are interested in the one related to our SNS topic.
Now to delete the dead reference it’s necessary to call:

aws lambda remove-permission \
 --statement-id sns-eu-west-1-111111111111-dead-lambda-trigger-reference-topic \
 --function-name dead-lambda-trigger-reference-lambda

That is all. The trigger is gone and the subscription can be deleted, if not already done, through the SNS console. It is easy to see that this is about the removal of one policy statement. This could just as well be done using the IAM console, but I think it would be even less readable for the user. Especially for a non-technical one.

4. Conclusion
I have shown how to delete a cross-account Lambda trigger using the CLI, but of course it can be done through any SDK or REST API calls using the equivalents of these commands. However, these are not intuitive commands that are easy to find in the documentation. It would also be great if this could be done easily through the SNS console, or if the trigger would also be removed when the subscription is deleted.

Therefore: AWS, fix it please!