Table of Contents generated with DocToc
- Overview
- Prerequisites for this setup
- Quick start: Create AWS CloudTrail
- Third party visualized reporting tools
- The CloudTrail and Spunk integration
- Reference: what is CloudTrail
This document describe how to use a script to setup AWS Cloudtail services to audit API calls, and how to setup SpunkAppForAWS app to generate reports in Splunk.
All the setups can be done through AWS console, but we use a script for CloudTrail setup to ensure naming schema consistency cross all regions. All AWS services (Cloudtrail, SNS, SQS) created will have a prefix 'accountId-'
[AWSCLI] (https://github.com/aws/aws-cli) command line tool is used to create Cloudtrail. To install (or upgrade) the package:
$ brew install awscli jq
This will install aws command under /usr/local/bin. There are three ways to setup AWS CLI AWS credentials. The examples here assumes you run the Cloudtrail creation code on an on-premise system and use a configuration file for key id and key secret. If you run it on EC2, you need to create an IAM role and åthe aws cli can use role-based token automatically.
To configure AWSCLI with an IAM user that can create CloudTrail, SNS, SQS, and S3 bucket services
$ aws --profile <profile> configure
The above command will generate two files, for example:
$ cat /etc/.aws/awscli.conf
# For AWS CLI
[profile mylab-cloudtrail]
region = us-west-2
...
$cat /etc/.aws/credentials
[mylab-cloudtrail]
aws_access_key_id = <key id>
aws_secret_access_key = <secrect>
For security purpose, we monitor all regions that CloudTrail service is available. These are the current supported regions:
- us-east-1
- us-west-1
- us-west-2
- eu-west-1
- sa-east-1
- ap-northeast-1
- ap-southeast-1
- ap-southeast-2
To enable CloudTrail on these regions, download and run cloudtrail-admnin.sh. This script calls awscli cloudtrail command to create or destroy cloudtrail setup. For setup, it performs the following functinons:
- Turn on Cloudtrail service in all supported region to send reports to s3://
<accountId>-cloudtrail S3 bucket - Create trail with name s3://
<accountId>-cloudtrail-<region> - Create Simple Notification Service (SNS) topic for the Cloudtrail service in each region
- (Optional) Create
<accountId>-cloudtrail SQS in one region to receive CloudTrail report notification. You only need this for Splunk.
All CloudTrails reports are sent into one S3 bucket default to s3://<accountId>-cloudtrail. Global events - generated by global services such as Identity and Access Management (IAM) - will be logged in the trail in the region given at the command line.
For help:
$ ./cloudtrail-admin.sh -h
create-cloudtrail -a <action> -p <profile> [-b <bucket>] -r region [-n] [-y]
-a <create|show|delete>: action. create, show or delete cloudtrails setup by this tool.
-p <aws profile>: authenticate as this profile.
-b <bucket>: optional. bucket name to get all trail reports.
-r <region>: region to get AWS global events, e.g. IAM
-y : non-interative mode. Answer to yes to all default values.
-n : dryrun. print out the commands
-h : Help
To create Cloudtrails, SNS and SQS:
./cloudtrail-admin.sh -a create -p <aws profile> -r <region to receive global event>
To show Cloudtrails, SNS and SQS:
./cloudtrail-admin.sh -a show -p <aws profile>
To tear down the resources created:
./cloudtrail-admin.sh -a delete -p <aws profile> -r <region to receive global event>
NOTE: The Cloudtrail bucket will not be deleted since it may contain past audit logs.
The cloudtrail now is enabled and you can go to AWS CloudTrail service console to view logs or through API calls.
Many tools are available to generate visualized reports using the CloudTrail files stored in S3 bucket. Here are listed AWS partners. This documentation describes how to use SplunkAppforAWS to consume Cloudtrail data and generate reports.
Traditionally system administrators monitor a system's integrity using intrusion detection tools such as Tripwire. System logs are usually sent to a central log server for auditing and security analysis.
For services running on AWS, another important operation and security related auditing is to monitor API calls that can change services and environments. Use cases enabled by CloudTrail service:
- Security Analysis
- Track Changes to AWS Resources
- Troubleshoot Operational Issues
- Compliance Aid
Here is an example of API call recorded by CloudTrail and how Spunk reports it (what, when, who, where, etc.)
{ [-]
awsRegion: us-west-2
eventID: 7ad60379-0e2e-4b6c-8a6f-100f00fc5df1
eventName: ModifyReservedInstances
eventSource: ec2.amazonaws.com
eventTime: 2014-07-14T19:29:42Z
eventVersion: 1.01
requestID: 900f0797-0f2d-43dd-bcf5-41e1cfbc5c9f
requestParameters: { [-]
clientToken: fa8494bb-8546-4fc4-9bb8-8dc84e7d0015
reservedInstancesSet: { [+]
}
targetConfigurationSet: { [+]
}
}
responseElements: { [-]
reservedInstancesModificationId: rimod-f0713af8-0984-42ff-8016-3dfea813b209
}
sourceIPAddress: 171.66.221.130
userAgent: console.ec2.amazonaws.com
userIdentity: { [-]
accessKeyId: XXXXXXXXXXXXXXXXXXX
accountId: XXXXXXXXXXXXX
arn: arn:aws:sts::XXXXXXXXXXX:assumed-role/admin-saml/john
principalId: XXXXXXXXXXXXX:jonh
sessionContext: { [+]
}
type: AssumedRole
}
}
You should create an IAM user with the minimum access privileges needed.
Fexample, you can create an IAM user cloudtrail-splunkapp which has permission to read SQS, delete message from SQS, and get data from S3 bucket. The following polices should work if you use canned AWS policy generator:
- Readonly access to S3 bucket
- Readonly access to CloudTrail
- Full access to the SQS (it deletes messages after read stuff from the message queue)
In this integration, we create CloudTrail service for each region and Simple Notification Service (SNS) topic for each CloudTrail service. The reports from all regions are aggregated to one S3 bucket. One Simple Queue Service (SQS) is subscribed to all the SNS topics.
The SQS named <accountNumber>-cloudtrail is created by cloudtrail-admin.sh script. It is easiest to use AWS's SQS service console to subscribe the queue to each SNS cloudtrail SNS topics because AWS service will generate proper policy for the queue to allow messages from the SNS topics.
SQS is needed in Splunk AWS app configuration. Splunk AWS app runs at defined interval to retrieve messages from AWS SQS service. The message body contains the S3 bucket location for Cloudtrail reports. Splunk then calls S3 API to get Cloudtrail reports from the S3 bucket.
The application has two major report categories - Cloudtrail monitoring for API activities, and Billing and Usage that helps budgeting and cost analysis.
This module requires to turn on Detailed Billing Reports. AWS will publish detailed billing reports in CSV format to the S3 bucket that you specify. If you have consolidated billing setup, ask the payer to create an IAM user that has read access to the S3 bucket that collects billing csv files. If you want to monitor the subaccounts's running status (e.g. the break downs of running on-demand instances, reserved instances, type of instances etc.), you need to create an IAM user in each sub-account that has access to check EC2 status in that subaccount.
To make the Billing and Usage work, you need the following:
- If you use consolidated billing, ask the payer to create an IAM user that have read only access to the S3 bucket's billing CSV file
- If you need to pull dynamic EC2 information, create an IAM user for each account/subaccount with the following policies:
- Amazon EC2 Reports Access
- Amazon EC2 Read Only Access
- opt/splunk/etc/apps/SplunkAppforAWS/lookupsinstancetype.csv This file should contain all instances types that you use in your account. The list from the SplinkAppforAWS may not be complete.
- aws.conf file
The file contains the setup for Billing and Usage. Here an example:
# more /opt/splunk/etc/apps/SplunkAppforAWS/local/aws.conf
# All this three stanzas are required, in order to run AWS App.
[keys]
# Format :
# <accountno> = <id> <access-key> <secret> <limit>
#
# Where:
# id = company/group name without space
# access-key = AWS Access Key
# secret = AWS Secret Key
# limit = Monthly spending limit for this account
# If you have this
012345678901 = <rollup-paying account> <IAM billing policy user key> <IAM billing policy key secrete> 200000
# Accounts/subaccounts. Information can be found in the detailed billing reports
<account-no1> = <account-name1> <IAM EC2 policy user key> <IAM EC2 policy key secrect> 100000
<account-no2> = <account-name2> '' '' 100000
[regions]
rgn1 = eu-west-1
rgn2 = sa-east-1
rgn3 = us-east-1
rgn4 = ap-northeast-1
rgn5 = us-west-2
rgn6 = us-west-1
rgn7 = ap-southeast-1
rgn8 = ap-southeast-2
[misc]
# Format :
# corpkey = <id> <access-key> <secret>
#
# Where:
# id = company name/corp account name without space
# access-key = AWS Access Key
# secret = AWS Secret Key
# Same as you specified in [key] section
corpkey = <rollup-paying account name> <IAM billoing user key> <IAM billing user key secret>
# acno is corp account number for AWS, same as in [key] section
acno = 01234567891
# s3bucket is bucket name where AWS bill csv files will be dropped
s3bucket = <builling bucket>
5 . /opt/splunk/etc/apps/SplunkAppforAWS/lookups
This file contains account and name mapping. Put accounts/subaccounts found in the billing CSV file here:
# more accountNames.csv
accountId,accountName
012345678901,<name>
123456789012,<name>
.....
The SplunkAppForAWS needs to be installed on Splunk search head and indexers. The data input only needs to be configured on indexers where the app will retrieve audit reports and build the indexes. Splunk search head does search against the logindex servers.
You can setup data input on Splunk Settings->Data Input console:
- Go to "Settings->Data Inputs"
- Create a new data input
- Fill out the CloudTrail IAM user's key, secret, SQS name, and the region, created by create-cloudtrail.sh script.
- Set script interval to 300 seconds. Cloudtrail delivers report within 15 minutes of API calls to S3 bucket, and within 5 minutes to SNS after the report is posted. You have up to 20 minutes to see the the latest API calls to show up in Splunk.
- In the "Script iterations" box, use a very large number, otherwise, the app script will stop running after 2048 runs.
- Click "More Advanced Setting", check Source Type to Manual, and "Source" to aws-cloudtrail
It will generate a inputs.conf file, that can be managed by a configuration tool such as Puppet. Here is an input.conf file example:
splunkidexer:/opt/splunk/etc/apps/launcher/local# more inputs.conf
exclude_describe_events = 0
index = aws-cloudtrail
interval = 30
remove_files_when_done = 0
key_id = <access key>
secret_key = <access key secret>
sqs_queue = <accountNumber>-cloudtrail
sqs_queue_region = us-west-2
sourcetype = aws-cloudtrail
script_iterations = 1000000
-
First update to the latest boto library. The package comes with it boto installation under /opt/splunk/etc/apps/SplunkAppforAWS/bin directory. To update:
mkdir /tmp/boto pip install --install-option="--prefix=/tmp/boto" boto cd /tmp/boto/lib/python2.7/site-packages tar cvf /tmp/botolib.tar boto cd /opt/splunk/etc/apps/SplunkAppforAWS/bin mv boto boto.old && tar xvf /tmp/botolib.tar -
/var/log/splunk/splunkd.log file contain splunk logs. Grep 'AWS' keywords to find out if the scripts are running and if there are any problems.
-
$SPLUNK_HOME/etc/apps/SplunkAppforAWS/lookups directory should contain billing data pulled from S3 bucket. subaccounts.* files are generated from the get_bill.py program each time a pulling occurs.
-
$SPLUNK_HOME/etc/apps/SplunkAppforAWS/log contains logs generated by various scripts that pulls billing, instance usage information. You need access key and key secret (create an IAM user for it) for the accounts you want to pull the information.
[AWS CloudTrail] (http://aws.amazon.com/cloudtrail/) is an AWS service. When enabled, it captures AWS API calls made by or on behalf of an AWS account and delivers log files to an Amazon S3 bucket.