↓Skip to main content

Send AWS Config Events to ServiceNow via AWS EventBridge

·10 mins
Blog Aws Eventbridge Aws-Config Servicenow
Table of Contents

Recently our IT Service Automation team requested a real time view of our AWS Resources for our Configuration Management Database (CMDB). We leverage ServiceNow for our CMDB so tracking changes to cloud resources from a third party service was an interesting challenge. After some investigation, I felt the solutions available were overly complicated and with the push for Event Driven Architecture (EDA), I figured I could leverage Amazon EventBridge to capture and send resource change events.

AWS Config #

Enter AWS Config. The description in the AWS Config developer documentation sums up the capabilities quite well:

AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.

By leveraging EventBridge with Config, we can “watch” for any configuration changes and act on them using managed integration capabilities.

EventBridge #

EventBridge is an interesting tool. It is both simple and complex at the same time, which means it is quite tailorable to many different use cases. In our case, we have many AWS accounts in our environment. We manage all of those accounts using AWS Organizations via a “management” or “root” account. The topic of AWS Organizations is beyond the scope of this post, but it is crucial in the management and governance of multi-account AWS implementations. Since we have so many accounts, we need a way to communicate changes in one of the member accounts back to the management account. EventBridge allows this via cross-account configuration and permissions. I will address the details below.

Member Account #

First, let’s talk about capturing the AWS Config events in the Member account.

AWS Config and many other AWS services send their events to the default event bus within the account. Coming from a typical queuing background like RabbitMQ or SQS, I had the expectation of configuring a custom bus. Unfortunately this is not available for these managed services so we must work with the default.

Event Bridge Rule #

To start, we must configure the rule which will listen for the events AWS Config sends to EventBridge. These rules look at the data in the event and if it finds a match it can take an action. The rule ends up being pretty simple as this CloudFormation sample shows:

EventPattern:
  source:
    - aws.config
  detail-type:
    - Config Configuration Item Change

As you can see, we are looking for a source of aws.config which is the source service of the event. Next in the detail-type field, we are looking for a match of Config Configuration Item Change. This is the detail which AWS Config is announcing when a resource changes. This includes resource creation and termination, which is crucial for keeping our CMDB up to date. Read more about EventBridge patterns.

Event Bridge Targets #

CloudWatch #

Now that we are capturing the events we are interested in with a rule, what should we do with that data? I’m a firm believer in observability so let’s focus on logging the event to CloudWatch to make sure our rule is capturing events correctly. We will define our targets in CloudFormation like this:

Targets:
  - Id: !Sub "log-${Name}"
    Arn: !GetAtt LogGroup.Arn

This snippet is linking the log group defined in the CloudFormation template with the target. That’s pretty cool, but aren’t we missing something? Where have we defined the IAM permissions for EventBridge to write to CloudWatch? In steps a resource policy. An IAM policy grants permissions to a User or Group where Resource policies grant permissions to a resource, in this case we want EventBridge to have access to Cloudwatch. Granting the permission only needs to be done once per account, so let’s check to see if our account has an existing policy for CloudWatch. We will use the AWS CLI and the following command:

 aws logs describe-resource-policies --region us-east-1

When you run this command, you may get an empty array for resourcePolicies or you may see something like this example where a policy has been created for another resource.

{
  "resourcePolicies": [
    {
      "policyName": "AWSLogDeliveryWrite20150319",
      "policyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AWSLogDeliveryWrite\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"delivery.logs.amazonaws.com\"},\"Action\":[\"logs:CreateLogStream\",\"logs:PutLogEvents\"],\"Resource\":\"arn:aws:logs:us-east-1:123456789000:log-group:/aws/vendedlogs/pipes/SQSReDrivePipe:log-stream:*\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"123456789000\"},\"ArnLike\":{\"aws:SourceArn\":\"arn:aws:logs:us-east-1:123456789000:*\"}}}]}",
      "lastUpdatedTime": 1711392687402
    }
  ]
}

If the response doesn’t contain a policy for EventBridge you will need to upload a policy document like this one to your account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TrustEventsToStoreLogEvent",
      "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
      "Effect": "Allow",
      "Principal": {
        "Service": ["events.amazonaws.com", "delivery.logs.amazonaws.com"]
      },
      "Resource": "arn:<partition>:logs:<region>:<account>:log-group:/aws/events/*:*"
    }
  ]
}

Assuming the policy document is in a file called resourcePolicy.json, you can upload it to your account using the following command:

aws logs put-resource-policy --policy-name TrustEventsToStoreLogEvents --policy-document file://resourcePolicy.json

Note: your log group name MUST be in the /aws/events namespace when you create the log group so that the policy and the log group line up properly.

An alternative method to deploy the policy is via a CloudFormation resource such as the following. Keep in mind if you deploy this resource via CloudFormation and then later delete the CloudFormation stack, this resource policy will also be deleted and will prevent your event buses from logging to CloudWatch. If you have multiple event buses, consider deploying the policy in a centralized stack for the entire account.

You should choose only one method (AWS CLI or CloudFormation) to deploy the policy.

LogPolicy:
  Type: AWS::Logs::ResourcePolicy
  Properties:
    PolicyName: log-policy-EventBridge
    PolicyDocument: !Sub '{"Statement": [{"Action": ["logs:CreateLogStream","logs:PutLogEvents"],"Effect": "Allow","Principal": {"Service": ["events.amazonaws.com","delivery.logs.amazonaws.com"]},"Resource": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/events/*:*","Sid": "TrustEventsToStoreLogEvent"}],"Version": "2012-10-17"}'

Here is the CloudFormation I used to create my log group to capture the AWS Config events captured by the rule.

LogGroup:
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: !Sub "/aws/events/eventbridge/producer-${Name}"
    RetentionInDays: 7
    Tags:
      - Key: Name
        Value: !Sub "eventbridge-producer-${Name}"
Cross Account EventBridge Destination #

Since we have a multi account configuration, we need a place to aggregate the events from all of the accounts by sending the event to an EventBridge bus in our management account. We do that by setting the Arn of a custom event bus in the management account and specifying a role. We will discuss this custom bus a little later in this post.

Targets:
  - Id: !Sub "root-account-${Name}"
    Arn: !Ref DestinationEventBusArn
    RoleArn: !GetAtt ProducerRole.Arn

This role has both a trust for EventBridge and a policy for putting events to the DestinationEventBusArn, which in the context of the CloudFormation template is passed in as a parameter.

ProducerRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: !Sub "role-producer-${Name}"
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service: events.amazonaws.com
          Action: sts:AssumeRole
    Policies:
      - PolicyName: AllowEventsToEventBridge
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            Effect: Allow
            Action:
              - events:PutEvents
            Resource: !Ref DestinationEventBusArn

Management Account #

Next, we need to discuss setting up our management account bus to receive messages from all accounts in the organization.

Since the management account “is an account too”, we need to capture config events here as well from the default event bus. The CloudFormation resources are defined in the same way as the Producer account: rule, logging, policies, and roles as we’ve defined above. The special nature of the management account lies in the custom event bus we create to gather the events together so they can be sent to ServiceNow.

Custom Event Bus #

The custom event bus is actually very simple to configure, it just needs a name.

EventBus:
  Type: AWS::Events::EventBus
  Properties:
    Name: !Sub "eventbus-${Name}"

Event Bus Policy #

Since the event bus is custom, we need to specify permissions as to which resources can send it events.

EventBusPolicy:
  Type: AWS::Events::EventBusPolicy
  Properties:
    EventBusName: !Ref EventBus
    StatementId: AllowAllAccountsInOrganizationToPutEvents
    Action: "events:PutEvents"
    Principal: "*"
    Condition:
      Key: aws:PrincipalOrgID
      Type: StringEquals
      Value: !Ref OrganizationId

This policy is similar to an IAM policy in its structure with a StatementId, Action, Principal, and Condition. In this case, we are allowing EventBridge to PutEvents from any Principal as long as the Principal is in the specified AWS Organization via the OrganizationId. The OrganizationId is passed in as a parameter to the CloudFormation template.

API Rule #

Now that we have the event bus configured, we need a rule to do something with those events. Let’s walk through the APIRule resource.

ApiRule:
  Type: AWS::Events::Rule
  Properties:
    Name: !Sub "api-rule-${Name}"
    Description: Rule to send events to API destination
    EventBusName: !GetAtt EventBus.Arn
    EventPattern:
      source:
        - aws.config
    State: "ENABLED"
    Targets:
      - Id: !Sub "apitarget-${Name}"
        Arn: !GetAtt ApiDestination.Arn
        RoleArn: !GetAtt ApiRole.Arn
      - Id: !Sub "log-${Name}"
        Arn: !GetAtt ConsumerLogGroup.Arn

Similar to the Producer and Consumer rules attached to the default busses, this rule is looking for aws.config event sources in the pattern and we have a logging target. The difference with this rule is, instead of pointing the target at another event bus, we are pointing it at an API Destination.

API Destination #

API destinations are defined as:

EventBridge API destinations are HTTPS endpoints that you can invoke as the target of an event bus rule, or pipe, similar to how you invoke an AWS service or resource as a target. Using API destinations, you can route events between AWS services, integrated software as a service (SaaS) applications, and public or private applications by using API calls.

This makes using API destinations the perfect choice for sending events to ServiceNow’s API. Here is how the resource is defined.

ApiDestination:
  Type: AWS::Events::ApiDestination
  Properties:
    Name: !Sub "destination-${Name}"
    ConnectionArn: !GetAtt Connection.Arn
    Description: !Sub "destination-${Name}"
    HttpMethod: !Ref HttpMethod
    InvocationEndpoint: !Sub "{{resolve:secretsmanager:${ApiSecret}:SecretString:ApiEndpoint}}"
    InvocationRateLimitPerSecond: !Ref InvocationLimit

AWS has abstracted much of the complexity for us when connecting to an HTTP endpoint, so the definition is pretty simple. The HTTP Method and Invocation Rate Limits are passed into the template as parameters so they are customizable. The configuration of the endpoint I’ve chosen to keep in a Secrets Manager secret and is resolved at deploy time by the CloudFormation service using the resolve intrinsic function. As you can see, there is a ConnectionArn property which we will discuss next.

Connection #

The Connection defines how EventBridge will authenticate with the external service. Authentication methods include Basic, API Key, and OAuth. I do not recommend Basic for any type of workload. Basic authentication credentials are easily intercepted by a malicious actor. I’m showing an API Key example here where the name and value pair is stored securely in Secrets Manager. Implementing OAuth would be ideal, as the tokens are exchanged in real time and are temporary. While possible to use OAuth with ServiceNow, we chose API Key for the first version of this integration to help reduce complexity.

Connection:
  Type: AWS::Events::Connection
  Properties:
    AuthorizationType: API_KEY
    AuthParameters:
      ApiKeyAuthParameters:
        ApiKeyName: !Sub "{{resolve:secretsmanager:${ApiSecret}:SecretString:ApiKeyName}}"
        ApiKeyValue: !Sub "{{resolve:secretsmanager:${ApiSecret}:SecretString:ApiKeyValue}}"
    Description: !Sub "connection-${Name}"
    Name: !Sub "connection-${Name}"

So to summarize, the event bus targets the API Destination which contains the HTTPS Rest endpoint, and the API Destination uses the Connection for the authentication values needed for secure data exchange.

API Destination Target Role #

In order for this resource chain to properly work, we need to circle back to the ApiRule resource and discus the IAM Role for the API Destination target.

ApiRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: !Sub "role-apidestination-${Name}"
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service: events.amazonaws.com
          Action: sts:AssumeRole
    Policies:
      - PolicyName: !Sub "policy-apidestination-${Name}"
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action: events:InvokeApiDestination
              Resource: !GetAtt ApiDestination.Arn

Walking through this resource we see we have a trust relationship with EventBridge, and the policy sets the events:InvokeApiDestination action for the API Destination resource. Fairly straightforward, but crucial to ensure proper execution.

Final Thoughts #

I want to mention these snippets I’ve shown you here have been simplified for clarity. Any organization larger than one person should give thought to and implement a tagging strategy. At my organization, we have a tagging strategy and we require certain tags on all resources. I’ve removed those tags from the resources discussed to simplify the examples. You can find the two sample CloudFormation templates here: Producer used in the member accounts and Consumer used in the management account.

In addition, in order to deploy the stacks in all of the member accounts, we use CloudFormation StackSets. StackSets are beyond the scope of this post and you can learn more about StackSets here. Because the member account stacks rely on information from the management account stack, you need to deploy the management stack first, and then the member account StackSet.

As you can see EventBridge simplifies the collection of AWS Config events across multiple accounts. There is no “programming” necessary from the standpoint of Lambdas and no queueing to handle message volume; all of these concerns are handled by EventBridge and the associated resources.