Logging Solutions

• Obs: May be necessary attach the Cloud Watch IAM policy inside the EKS NodeGroup

FluentD + CloudWatch + ElasticSearch

Configure FluentD

Change this values inside the fluentd.yml file

      containers:
      - name: fluentd-cloudwatch
        image: fluent/fluentd-kubernetes-daemonset:v1.14-debian-cloudwatch-1
        env:
          - name: REGION
            value: eu-west-1
          - name: CLUSTER_NAME
            value: EKS-Workshop

Apply fluentd.yml.

kubectl apply -f fluentd/fluentd.yml

Logs can take up to 5 minutes before showing inside CloudWatch

AWS -> CloudWatch -> Logs -> Log Groups -> /eks/EKS-Workshop/containers

Better way to visualize

1. Enter in the Log Events
2. Actions
3. Expamd all rows

Create Elastic Search Stack

-Under Construction-

Configure CloudWatch

• Create a new role (IAM)

• Roles

• Create Role
• Select Lambda
• Next: Permissions
• Attach AmazonESFullAccess

• rolename -> lambda_es

• Go to cloudwatch /eks/EKS-Workshop/containers

• Actions

• Create Amazon OpenSearch Service subscription filter
• Select Amazon ES cluster
• Select Lambda IAM execution Role (lambda_es)
• Log format -> Common Log Format
• Subscription filter name -> [^:]

• Start Streaming

• Open Kibana

• Auth

• Security
• Roles
• all_access
• Mapped users
• Manage mapping
• Add user (edit arn as needed):
• arn:aws:iam::111111111111:role/lambda_es
• Add backend roles

• arn:aws:iam::111111111111:role/lambda_es

• Index

• Dashboard
• Create Index Pattern
• cwl-*
• Next step
• @timestamp
• Create Index pattern

Demo

kubectl apply -f log-generator/log-generator.yml