ELK Stack - Container - Docker Compose - AWS ECS

9/12/2021 - Zeroska, It could be use to just deploy a normal ELK Stack on premise not necessary must be on AWS Cloud

The Current Situation

Our company needs a centralized log system, among all the open source and free logging systems I choose ELK Stack.

We currenly don't have ML nodes

When I started I have to research which architecture would be suitable for our needs. Luckily I found this https://www.elastic.co/pdf/architecture-best-practices.pdf (with no background in system design, I think this is a great material)

Update 16/6/2022: The ELK Project is now scaled up into ELK Stack on AWS ECS with code pipeline and ba unch of AWS services

Update 2/7/2022: The Elasticsearch node when deploying to ECS while using Fargate is seem impossible because the Elasticsearch node in the docker container needs a lot of Virtual Memory and Fargate doesn't support configuring the vm.max_map_count= 261244 so the Docker Compose file I used no longer work

AWS Model:

ELK AWS Model

The Process of Making ELK Stack

At first, I use docker-compose to extend my laziness.

One of the core intentions here is to keep the Docker Compose file generic, not hard-coding variables, allowing it to be environment agnostic (i.e. development/testing/production)

Because of the updated 2/7/2022, I can't deploy it using docker-compose anymore (or so I thought), I am try to find another way to deploy ELK Stack on AWS so there are 2 ideas flashing in my head:

1. Manually configure the EC2 instance (😐😐)

2. Build the image and push the image to ECR and then manually config the ECS to use EC2 as a Launch type

Build Image and Push to ECR and then use ECS Launch type EC2

Before we build the image of Elasticsearch, Logstash, and Kibana we must know that when pushing to the AWS environment it is very different from normal on-premise

First is that you have to install these plugins in your Elasticsearch container for it to run on the AWS Cloud

• discovery-ec2 (https://www.elastic.co/guide/en/elasticsearch/plugins/8.3/discovery-ec2.html) currently the 8.x version has still supported this plugin

• repository-s3 (https://www.elastic.co/guide/en/elasticsearch/reference/8.3/repository-s3.html)

The Second is to have AWS CLI installed on your machine and the right IAM policy (the privilege to use ECS, ECR,...)

Here is the GitHub repo that I use during the time, it includes the Dockerfile and script and guidance for building ELK Stack and deploying it to ECS

ECS Task definition and Auto Scaling Group and other AWS related

Important Notes

Some tips and finding that I stumble when configuring ELK Stack:

LogstashElasticsearchTroubleshooting and Problems

After the cluster forms successfully for the first time, remove the cluster.initial_master_nodes setting from each node's configuration. Do not use this setting when restarting a cluster or adding a new node to an existing cluster. (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html)

Config TLS for Container (Pain)

Important Configuration

• https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config.html#dev-vs-prod

• https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster (cái này nói về scaling ES cluster)

Refs

• https://medium.com/@devfire/deploying-the-elk-stack-on-amazon-ecs-dd97d671df06 (good series on how to deploy elk stack on Amazon ECS)

• https://medium.com/opseco-technologies/what-is-an-elasticsearch-cluster-dce98ed3edc0 (I and my college learn how to set vm.max_map for ECS using launch type EC2 by reading this blog)