【Introduction to Containers】Tips for handling ECS container logs in AWS - PrismScaler

Hello! We are a writer team from Definer Inc.
In this issue, you are wondering about some tips for handling ECS container logs in AWS.
Let's take a look at the actual screens and resources to explain in detail.

This article is written for the purpose of properly managing container logs.
It is a collection of information and practices that can be helpful to IT professionals when they want to properly implement log management when using AWS to operate ECS containers.

We will consider how to handle logs in container operations with ECS.
 
Since ECS Docker containers are used and discarded each time they are deployed, application logs cannot be stored in the container.
Therefore, application logs must be managed using external storage or external services.
 
The following is a summary of some options for log management for ECS containers.
 
(1) Cloudwatch Logs
The most popular method is to use Cloudwatch Logs.
 
The advantages are as follows.
・It is a managed service of AWS and easy to integrate with ECS.
・Cost reduction is possible by rotating to S3.
・Visualization is easy with ElasticSearch.
 
On the other hand, there are some disadvantages as follows.
・Container logs must be exported to standard output to be transferred to Cloudwatch Logs.
・Log stream units are difficult to see (it is important to be able to identify which container the logs originated from, for example, by including the startup date and time in the host name, etc.)
 
(2) Log collection service
Another option is to use a log collection and analysis service such as fluentd.
The advantage is that flexible customization is possible.
The disadvantages are the service fee and the cost of implementing integration with AWS.

Let's start with the Cloudwatch Logs configuration for the ECS task definition.
As a precondition, we assume that the application logs in the container are spit out to standard output (STDOUT).
 
(1) Preparation of Json files as inputs
This time, the following input file is used.
The key point is lines 17 through 25, where "awslogs" is specified as the log driver.
 
(2) Update ECS task definition
Go to the ECS console, "Task Definition" → Task Details screen → "Create New Revision" → "Create New Revision Using JSON".
Copy and paste the JSON file that will be the input and save it.
 
After a fresh start of the container, you can see it logging to Cloudwatch Logs!

{
    "family": "test",
    "containerDefinitions": [
        {
            "name": "test",
            "image": "${account number}.dkr.ecr.ap-northeast-1.amazonaws.com/test:610",
            "cpu": 512,
            "memory": 1024,
            "portMappings": [
                {
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "ecs/test",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "test"
                }
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::${account number}:role/${IAM Task Role Name}",
    "executionRoleArn": "arn:aws:iam::${account number}:role/${IAM task execution role name}",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "EC2",
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "1024",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    }
}                

Share CloudWatch to another account and put it all together in one account ....
Log output from Aqua Enterprise to CloudWatch Logs ...
The concept of multiple accounts in a startup and AWS Control ...

・PrismScaler is a web service that enables the construction of multi-cloud infrastructures such as AWS, Azure, and GCP in just three steps, without requiring development and operation.
・PrismScaler is a web service that enables multi-cloud infrastructure construction such as AWS, Azure, GCP, etc. in just 3 steps without development and operation.
・The solution is designed for a wide range of usage scenarios such as cloud infrastructure construction/cloud migration, cloud maintenance and operation, and cost optimization, and can easily realize more than several hundred high-quality general-purpose cloud infrastructures by appropriately combining IaaS and PaaS.
 

This article provides useful introductory information free of charge. For consultation and inquiries, please contact "Definer Inc".

・Definer Inc. provides one-stop solutions from upstream to downstream of IT.
・We are committed to providing integrated support for advanced IT technologies such as AI and cloud IT infrastructure, from consulting to requirement definition/design development/implementation, and maintenance and operation.
・We are committed to providing integrated support for advanced IT technologies such as AI and cloud IT infrastructure, from consulting to requirement definition, design development, implementation, maintenance, and operation.
・PrismScaler is a high-quality, rapid, "auto-configuration," "auto-monitoring," "problem detection," and "configuration visualization" for multi-cloud/IT infrastructure such as AWS, Azure, and GCP.