A small handler for AWS Cloudwatch
This small log handler is designed to send events to AWS Cloudwatch. It is useful when operating infrastructure outside of the AWS, when AWS does not log automatically (EC2) or when you want to separate logs. If you are logging on Serverless Infrastructure (e.g. Lambdas or ECS) you might find it easier to just let AWS Handle the logs automatically.
We originally developed this to be used on dedicated servers (on and off EC2) and chose to create something new because we wanted to:
If you already have a codebase that is using python's logger, you only need minor modifications to send your logs to AWS. In fact, you only need to change code at the logger creation. If you haven't done any logging before, I recommend you look at a basic tutorial on python's logging module. There are plenty of resources out there.
We are on PyPi, so you can install via pip
pip install cloudwatch
Or if you prefer to customise and (hopefully) feedback on improvements and bugs
git clone https://github.com/labrixdigital/cloudwatch
This module is designed to fit right into your existing logging code, so you only need to replace (or add) a handler and the same logger will send events to cloudwatch.
import logging
#Create the logger
logger = logging.getLogger('my_logger')
#Create the formatter
formatter = logging.Formatter('%(asctime)s : %(levelname)s - %(message)s')
#Import cloudwath and create the new handler
from cloudwatch import cloudwatch
handler = cloudwatch.CloudwatchHandler(log_group = 'my_log_group')
#Pass the formater to the handler
handler.setFormatter(formatter)
#Set the level
logger.setLevel(logging.WARNING)
#Add the handler to the logger
logger.addHandler(handler)
#USE IT!
logger.warning("Watch out! Something happened!")
If you dont add credentials when creating the handler, it uses the default AWS credentials that you set-up on the CLI, or that you passed on the invokation (if using on EC2, Lambda, ECS), this is in line with the boto3 configuration Expained here. However, you can also specify the credentials like this:
handler = cloudwatch.CloudwatchHandler(
log_group = 'my_log_group',
access_id = 'AWS_ACCESS_KEY_ID',
access_key = 'AWS_SECRET_ACCESS_KEY'
)
Likewise, you can specify a region where the logger will be with the parameter: region = 'us-east-1'
If you dont add a log stream, the logger will create one for you with the timestamp and a random number. This is useful when you have multiple processes logging to the same log group without colliding. If you want, you can specify the log stream like this:
handler = cloudwatch.CloudwatchHandler(
log_group = 'my_log_group',
log_stream = 'my_log_stream'
)
AWS CloudWatch Logs takes a maximum event size of 256 KB (reference). This means that large messages can result in an error if not handled correctly. We have included 3 options in the parameters:
One caveat is that because the truncate and split options are based on bytes and not characters (because its size limited), its possible that non-ASCII characters will end up being split or truncated.
#Specify truncate
handler = cloudwatch.CloudwatchHandler(
log_group = 'my_log_group',
overflow = 'truncate'
)
#Specify split
```python
handler = cloudwatch.CloudwatchHandler(
log_group = 'my_log_group',
overflow = 'split'
)
Positional order | Keyword argument | Required | Default | Description |
---|---|---|---|---|
0 | access_id |
No | Taken from the AWS Configuration File or Role | The AWS Access Key ID that you want to use to interact with your AWS Account. Usually a 20 character alphanumeric. |
1 | access_key |
No | Taken from the AWS Configuration File or Role | The corresponding AWS Secret to the above parameter |
2 | region |
No | Taken from the AWS Configuration File or Role | The AWS Region name (e.g. us-east-1 ) |
3 | log_group |
Yes | The name of the log group. If it already exists, it writes to it, otherwise it creates it. | |
4 | log_stream |
No | Datetime in the format %Y%m%d%H%M%S%f and 3 random digits |
The name of the log stream. If it already exists, it writes to it, otherwise it creates it. |
5 | overflow |
No | Defines the behaviour when a message is too large to send in one API call. Either 'error' , 'truncate' or 'split' . Default is 'error' |
We much prefer keyword arguments, and encourage you to use them. However, if you really want to avoid some typing, the order of the positional arguments work as follows:
handler = cloudwatch.CloudwatchHandler(
'AWS_ACCESS_KEY_ID',
'AWS_SECRET_ACCESS_KEY',
'REGION',
'LOG_GROUP',
'LOG_STREAM',
'OVERFLOW'
)