How to Send Log Files to AWS CloudWatch

6 min readJun 7, 2021

AWS CloudWatch allows you to collect logs from your AWS EC2 instances as well as on-premise servers. Files such as the Apache2 access and error logs, your application logs, mysql logs or mongodb logs that are commonly found on servers or any log file. This is especially useful if you have a scaling group of instances behind a load balancer. Rather than connecting to each instance and manually searching the logs with grep, CloudWatch centralises the logs into one log stream, allowing you to search all your log files from one place.

For example, we have a few EC2 instances behind a Load Balancer to run our application. They all run Apache2 so we send the contents of the Apache2 error log to CloudWatch. Even with just a few servers it’s much easier than logging into each one individually and searching each file with grep commands. In our infrastructure has over 20–30 servers and would be practically impossible to manage through the command line alone. If an issue occurs we’re able to see all the logs in the AWS Console without wasting time logging onto everything.

To get CloudWatch set-up on Ubuntu, you need to complete the following:

Create a new IAM role (one time only) or you can also use Access key and Secret key
Attach the IAM role to an EC2 instance (If you are going with access key and secret key skip this step)
Install and configure the CloudWatch agent

Create a New IAM Role

On the “Select type of trusted entity” page, select “EC2” as the service to be associated with the new role. Click the “Next: Permissions” button to proceed.

On the “Attach permissions policies” page, select the “CloudWatchAgentServerPolicy”. Click “Next: Tags” to proceed.

On the “Add tags” page, add tags if required (optional). Click “Next: Review” to proceed.
On the “Review” page, enter a name for the new role. Click “Create role” to proceed and create the new role.

Attach the IAM Role

To attach the IAM Role to the EC2 instance, you can either do it through the AWS console or via the AWS Command Line Interface (CLI):

1. Using the AWS Console

Go to the EC2 Dashboard, select Instances from the menu and check the checkbox next to the EC2 instance you want to stream the logs from. To attach the IAM Role, click the Actions dropdown and select Security > Attach/Replace/modify IAM Role:

2. Using the AWS CLI

This command was added to the AWS CLI in version 1.11.46, so make sure you have the correct version (you can check the version with aws --version).

$ aws ec2 associate-iam-instance-profile --instance-id YourInstanceId --iam-instance-profile Name=CloudWatchAgentServerRole

Or

Use AWS Access key and secret key

Click your username in the top right corner of the navigation bar and select the “My Security Credentials” menu item from the resulting drop-down menu.
On the “My security credentials” page, click the “Create access key” button.

In the resulting dialog, note the new AWS access key ID and corresponding secret access key
Create an AWS credentials file with the AWS access key ID and shared access key at /etc/.aws/credentials with the following content. Replace the AWS-ACCESS-KEY-ID and AWS-SECRET-ACCESS-KEY placeholders with the keys obtained in the previous step:

[default]
  aws_access_key_id=AWS-ACCESS-KEY-ID
  aws_secret_access_key=AWS-SECRET-ACCESS-KEY

Edit the common configuration file for the Amazon CloudWatch agent at /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml and specify the path to the credentials file created in the previous step.

sudo vi /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Update the file with this content:

[credentials]
  shared_credential_file = "/etc/.aws/credentials"

Note : You can use either attach a role with EC2 instance or using aws credentials

Install CloudWatch Agent on your server

Now everything is in place, connect to the EC2 instance you want to log data from and run through the following few commands. The first is to download the CloudWatch Agent from S3 (the following is for AMD64 Ubuntu, if you want a download for Centos, Debian, etc, see a full list here):

$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

Install the Agent with:

$ sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Now it’s installed it needs to be configured before it can be started. There are two ways to do this, using the wizard which will ask you a series of questions and generate a config file for you, or you can manually add a config file. As we’re only interested in logging one file to CloudWatch and it’s the same on each EC2 instance, it’s easy to manually add a config file:

1. Manually create a config.json

The Log Agent uses a config file located at /opt/aws/amazon-cloudwatch-agent/bin/config.json. Use your favourite editor (e.g. vim,vi,nano) to create and edit a file with the following content, e.g.

sudo vim /opt/aws/amazon-cloudwatch-agent/bin/config.json{
     "agent": {
         "run_as_user": "root"
     },
     "logs": {
         "logs_collected": {
             "files": {
                 "collect_list": [
                     {
                         "file_path": "/var/log/apache2/error.log",
                         "log_group_name": "/aws/apache2-error-log",
                         "log_stream_name": "{instance_id}"
                     },
                     {
                         "file_path": "/var/log/apache2/access.log",
                         "log_group_name": "/aws/apache2-access-log",
                         "log_stream_name": "{instance_id}"
                     }                 ]
             }
         }
     }
 }

The most important part of the config file is file_path. This is the path to the log file on the server that you want to collect data from. /var/log/apache2/error.log is the default error log for Apache on Ubuntu. The log_group_name and log_stream_name options are just used for naming the Log Group and Log Streams respectively in CloudWatch. I’d recommend keeping {instance_id} for the log_stream_name as this helps identify which EC2 instance sent the log data.

2. Wizard

To start the wizard run:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

You’ll be asked a series of questions. The Log Agent can be used to collect system-level metrics, so you will be asked about them. But you can ignore those questions as they’re not related to collecting logs. If you do use the wizard, you can always take the config file that’s generated and then manually add that (following the step above) to any additional instances.

Start the Agent

Run the following command to run the agent. The CloudWatch agent is integrated with systemd so it will start automatically after a reboot:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

Check that the agent is running with the following command:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

The steps above will also automatically configure the Amazon CloudWatch agent to automatically start on server reboot.

View Logs

Once the log file you are watching has data written to it, you’ll be able to find it in CloudWatch. Go to the CloudWatch Overview and select Logs from the menu. You should see the label for the Log Group you used in the config (e.g. apache-error-log).

Click on the log group name to see the log streams. Each log stream uses the EC2 instance ID, so you know which EC2 instance logged the data:

TIP: If the Amazon CloudWatch agent fails to start, you can debug the error by reviewing the Amazon CloudWatch log files. Learn more about the log files and other troubleshooting steps in the official documentation.

Conclusion

Want me to write more!
Hit claps if you liked it. It will encourage me to write more. Follow me for more interesting posts. Comment below if you have any other questions and inputs.