Sending Linux Application Logs to AWS CloudWatch

by

Amazon Web Services (AWS) CloudWatch is a comprehensive monitoring service designed to provide detailed insights into server performance and resource utilization. By default, AWS CloudWatch offers monitoring for server metrics like CPU Utilization, Network Traffic, and more. However, beyond these default parameters, users can configure CloudWatch for custom monitoring of metrics such as disk and memory (RAM) utilization. This flexibility allows for a tailored approach to monitoring system resources efficiently.

CloudWatch also extends its capabilities to monitor application or server logs, though this is not a default feature. Users need to configure it to push the desired logs for monitoring based on specific requirements.

Beyond AWS Resources

While an AWS-native service, CloudWatch is not restricted to AWS environments. It can be set up to receive logs from virtual machines (VMs) hosted on other cloud platforms or even from physical servers, providing a unified monitoring solution across diverse infrastructures.

Sending Logs to AWS CloudWatch

To forward application or server logs to AWS CloudWatch, you must install the CloudWatch agent on the server. This guide will demonstrate the process using an AWS EC2 instance.

Setup Steps

  1. Attach a suitable IAM role to the instance to enable communication with CloudWatch.
  2. Install the AWS CloudWatch agent.
  3. Configure the AWS CloudWatch agent.
  4. Verify logs in the AWS CloudWatch portal.

Configuring the IAM Role

Attaching a role with the necessary permissions to the EC2 instance allows it to interact with CloudWatch. The policies required include:

  • CreateLogStream
  • DescribeLogStream
  • CreateLogGroup
  • PutLogEvents

Follow these steps to create an IAM role:

  1. Navigate to the AWS console and select the “IAM” service.
  2. Create a new policy by selecting “Policy” in the IAM dashboard and then “Create Policy”.
  3. In the “Create Policy” tab, select “JSON” and input the following policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:*:*:*"
      ]
    }
  ]
}

After setting up the policy, name it and proceed to create a role by selecting “EC2” under Amazon Services. Attach the previously created policy to the role and provide a name for easy reference. The role is now ready to be attached to your EC2 instance.

Installing and Configuring the AWS CloudWatch Agent

With the role configured, proceed with the agent installation:

For Amazon Linux Instances

  1. SSH into your EC2 instance.
  2. Run system updates with sudo yum update -y.
  3. Install the awslogs package with sudo yum install -y awslogs.
  4. Edit /etc/awslogs/awscli.conf to set the correct region.
  5. Configure the logs to monitor in /etc/awslogs/awslogs.conf:
[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /var/log/messages
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = Amazon-Linux-2

Restart the awslog service using sudo service awslogsd start.

For RHEL or CentOS Instances

  1. Download the agent setup using: curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
  2. Run the setup script: python ./awslogs-agent-setup.py --region your-region
  3. Edit the resulting config file /var/awslogs/etc/awslogs.conf as necessary.
  4. Start the service with systemctl start awslogs.

Verifying the Configuration

  • Log into the AWS Management Console.
  • Select CloudWatch from the Services menu.
  • Navigate to “Log Groups”.
  • Search for the log stream name configured in your awslogs.conf.
  • Verify that logs are being sent to CloudWatch.

Conclusion

AWS CloudWatch offers a robust solution for server and log monitoring across various environments. Whether your infrastructure is on AWS, another cloud, or on-premises, AWS CloudWatch can provide centralized monitoring insights.

Frequently Asked Questions (FAQ)

  • What types of logs can I send to AWS CloudWatch?You can send application logs, server logs, or any custom logs that you configure the CloudWatch agent to monitor.
  • Can I monitor non-AWS servers?Yes, CloudWatch can be configured to monitor servers from other cloud providers or on-premises servers.
  • How frequently are the logs updated in CloudWatch?By default, logs are updated every 5 seconds, but this can be configured by adjusting the buffer_duration parameter.
  • Do I need an AWS account to use CloudWatch?Yes, an AWS account is necessary to configure and use AWS CloudWatch services.