1. Introduction
Hello! We are a writer team from Definer Inc.
In this issue, you are wondering about the use of the minimal monitoring configuration required for a serverless environment in AWS.
Let's take a look at the actual screens and resources to explain in detail.
2. Purpose/Use Cases
Monitoring gives us the ability to understand system health, resolve performance and availability issues based on data. Monitoring is essential for any system, including serverless architectures. However, the monitoring requirements for serverless architectures may be different compared to traditional server-based architectures. Here are a few reasons why minimal monitoring is often recommended for serverless:
・ Auto-scaling nature: Serverless platforms, such as AWS Lambda or Azure Functions, automatically scale the resources based on the incoming request load. This dynamic scaling reduces the need for monitoring resource utilization and capacity planning. The cloud provider takes care of scaling resources up or down as required, ensuring optimal performance and resource allocation.
・Limited infrastructure management: Serverless architectures abstract away the underlying infrastructure, allowing developers to focus more on writing code rather than managing servers or virtual machines. This means there is less need to monitor hardware or operating system metrics, as it becomes the responsibility of the cloud provider.
・Event-driven nature: Serverless functions are typically triggered by events, such as HTTP requests, database updates, or messages from message queues. Monitoring the event sources and the associated function invocations becomes crucial to ensure the system's correct functioning. However, monitoring other system components like the underlying server instances is unnecessary, as those are abstracted away.
・Billing and cost optimization: Serverless architectures often charge based on the resources consumed during function execution. Monitoring helps track usage patterns, identify potential cost optimizations, and understand the correlation between usage and associated costs. Minimal monitoring can be sufficient to gain insights into the cost aspects of the system.
This article summarizes information and practices that can be helpful when you want to design the minimal monitoring items required for AWS serverless environments such as Fargate and Lambda.
3. Minimal monitoring configuration required for serverless environment
First, let's assume that we are assuming the following AWS serverless application
・We want to ensure a minimum level of security while keeping costs low.
・There is no IaaS such as EC2 that needs to be managed.
・Error handling is not real-time, but can be done on the next business day.
・We are operating a serverless application with the following configuration: API Gateway + Lambda, ELB + Fargate, etc.
We have picked up the necessary monitoring items.
Security monitoring:
Security monitoring is the automated process of collecting and analyzing indicators of potential security threats, then triaging these threats with appropriate action. Due to the pervasive, unavoidable nature of security risks, fast response times are essential to maintaining system security, and continuous, automated security monitoring is key to quick threat detection and response.
・Enable CloudTrail
・Periodic implementation of Trusted Advisor (~1 per month)
・Enable GuardDuty
・Cloudwatch alert notification in Personal Health Dashboard
Resource Monitoring:
Resource monitoring is the practice of continuously tracking and measuring the utilization, performance, and availability of various computing resources within a system or infrastructure. It involves monitoring and collecting metrics related to CPU usage, memory consumption, disk I/O, network traffic, and other relevant parameters. On AWS, we can check those metrics on Cloudwatch:
・CPU monitoring (Cloudwatch metrics)
・Memory monitoring (Cloudwatch metrics)
・Disk monitoring (Cloudwatch metrics)
Resource monitoring empowers you to optimize resource utilization, enhance system performance, control costs, meet compliance requirements, and proactively manage issues. It helps you make data-driven decisions, improve operational efficiency, and ensure the reliability and stability of your system.
Log monitoring:
Log monitoring refers to the practice of collecting, analyzing, and interpreting log data generated by various components of a system or application. Logs capture valuable information about system events, activities, errors, and performance metrics. Log monitoring involves actively monitoring these logs in real-time or near real-time to gain insights into the health, performance, and security of the system.
・Application logs (Cloudwatch Logs)
・OS logs (Cloudwatch Logs)
4. Enable CloudTrail and GuardDuty
(1) Enabling CloudTrail
CloudTrail is enabled by default for your AWS account. You can use Event history in the CloudTrail console to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure. This includes activity made through the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.
For an ongoing record of events in your AWS account, create a trail. A trail enables CloudTrail to deliver log files to an Amazon S3 bucket.
It's important to note that enabling CloudTrail for Lambda functions will log events related to function invocations, function updates, and other Lambda-related management activities. However, it won't provide detailed logs for the function's internal execution. For that, you might need to implement additional logging within your Lambda function code using a logging framework like AWS CloudWatch Logs or a third-party solution.
Next, let's actually enable CloudTrail and create a trail for your serverless system.
First, we will configure CloudTrail. Before doing that, you will need an S3 bucket for log storage.
Please use the following commands to create and confirm the settings.
Note:
The
--is-multi-region-trail flag enables the trail to capture events from all AWS regions.
The --include-global-service-events flag includes global service events in the trail logs. ## Enabling CloudTrail
aws cloudtrail create-trail \
--name cloudtrail-test \
--is-multi-region-trail \
--enable-log-file-validation \
--s3-bucket-name ${S3 bucket name} \
--include-global-service-events
## Creating CloudTrail for Lambda function
aws cloudtrail create-trail \
--name ${trail name} \
--s3-bucket-name ${S3 bucket name} \
--is-multi-region-trail \
--include-global-service-events \
--event-selectors "ReadWriteType=All,IncludeManagementEvents=true,DataResources=[{Type=AWS::Lambda::Function,Resources=[${lambda-function-arn}]}]"
## Creating CloudTrail for API Gateway
aws cloudtrail create-trail \
--name ${trail-name} \
--s3-bucket-name ${S3 bucket name} \
--is-multi-region-trail \
--include-global-service-events \
--event-selectors "ReadWriteType=All,IncludeManagementEvents=true,DataResources=[{Type=AWS::ApiGateway::RestApi,Resources=[${api-gateway-id}]}]"
## Confirmation of CloudTrail activation
aws cloudtrail describe-trails
(2) Enable GuardDuty
・Intelligent threat detection: GuardDuty analyzes various data sources such as AWS CloudTrail logs, VPC Flow Logs, and DNS logs to identify suspicious and malicious activity. It uses machine learning models to detect patterns of behavior associated with known attack techniques and anomalous behavior.
・Continuous monitoring: GuardDuty provides continuous monitoring of your AWS accounts and workloads, automatically analyzing and processing data to detect threats. It doesn't require any manual configuration or additional software installation.
・Integrated threat intelligence: GuardDuty integrates with various threat intelligence feeds and continuously updates its knowledge base to identify known malicious IP addresses, domains, and other indicators of compromise. This helps to enhance detection accuracy and reduce false positives.
Next, we will configure GuardDuty.
After setting up CloudTrail, please configure GuardDuty by following commands:
Amazon GuardDuty is a threat detection service that continuously monitors your AWS accounts and workloads for malicious activity and delivers detailed security findings for visibility and remediation.
Here are some key features and benefits of Amazon GuardDuty:
・Intelligent threat detection: GuardDuty analyzes various data sources such as AWS CloudTrail logs, VPC Flow Logs, and DNS logs to identify suspicious and malicious activity. It uses machine learning models to detect patterns of behavior associated with known attack techniques and anomalous behavior.
・Continuous monitoring: GuardDuty provides continuous monitoring of your AWS accounts and workloads, automatically analyzing and processing data to detect threats. It doesn't require any manual configuration or additional software installation.
・Integrated threat intelligence: GuardDuty integrates with various threat intelligence feeds and continuously updates its knowledge base to identify known malicious IP addresses, domains, and other indicators of compromise. This helps to enhance detection accuracy and reduce false positives.
Next, we will configure GuardDuty.
After setting up CloudTrail, please configure GuardDuty by following commands:
## Enable GuardDuty
aws guardduty create-detector \
--finding-publishing-frequency FIFTEEN_MINUTES \
--region ap-northeast-1 \
--enable
## Confirmation of GuardDuty activation
aws guardduty list-detectors \
--region ap-northeast-1 5. Serverless Monitoring Tools
1. Datadog
2. Epsagon
3. Site24x7
4. Amazon CloudWatch
5. New Relic
6. Thundra
7. SignalFx
8. Dashbird.io
9. Lumigo
10. Rookout
6. Cited/Referenced Articles
7. About the proprietary solution "PrismScaler"
・PrismScaler is a web service that enables the construction of multi-cloud infrastructures such as AWS, Azure, and GCP in just three steps, without requiring development and operation.
・PrismScaler is a web service that enables multi-cloud infrastructure construction such as AWS, Azure, GCP, etc. in just 3 steps without development and operation.
・The solution is designed for a wide range of usage scenarios such as cloud infrastructure construction/cloud migration, cloud maintenance and operation, and cost optimization, and can easily realize more than several hundred high-quality general-purpose cloud infrastructures by appropriately combining IaaS and PaaS.
8. Contact us
This article provides useful introductory information free of charge. For consultation and inquiries, please contact "Definer Inc".
9. Regarding Definer
・Definer Inc. provides one-stop solutions from upstream to downstream of IT.
・We are committed to providing integrated support for advanced IT technologies such as AI and cloud IT infrastructure, from consulting to requirement definition/design development/implementation, and maintenance and operation.
・We are committed to providing integrated support for advanced IT technologies such as AI and cloud IT infrastructure, from consulting to requirement definition, design development, implementation, maintenance, and operation.
・PrismScaler is a high-quality, rapid, "auto-configuration," "auto-monitoring," "problem detection," and "configuration visualization" for multi-cloud/IT infrastructure such as AWS, Azure, and GCP.