{"id":34,"date":"2014-10-08T02:42:00","date_gmt":"2014-10-08T02:42:00","guid":{"rendered":"http:\/\/www.asianux.org.vn\/index.php\/2014\/10\/08\/centralized-logs-management-with-logtash-elasticsearch-and-redis\/"},"modified":"2014-10-08T02:42:00","modified_gmt":"2014-10-08T02:42:00","slug":"centralized-logs-management-with-logtash-elasticsearch-and-redis","status":"publish","type":"post","link":"https:\/\/www.asianux.org.vn\/index.php\/2014\/10\/08\/centralized-logs-management-with-logtash-elasticsearch-and-redis\/","title":{"rendered":"Centralized Logs Management with Logtash, ElasticSearch, and Redis"},"content":{"rendered":"

Deploying a <\/span><\/span>Centralized Logs Management System<\/b> <\/span>seems very easy these days with such these great tools:<\/span>

+ <\/span><\/span>Logtash<\/b>: collect logs, index logs, process logs, and ship logs<\/span>
+ <\/span><\/span>Redis<\/b>: receive logs from logs shippers<\/span>
+ <\/span><\/span>ElasticSearch<\/b>: store logs<\/span>
+ <\/span><\/span>Kibana<\/b>: web interface with graphs, tables…<\/span>

We will implement the logs management system as the following architecture:<\/span><\/p>\n

\"\"<\/a><\/div>\n



<\/p>\n

(based on this article: <\/span>http:\/\/michael.bouvy.net\/blog\/en\/2013\/11\/19\/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana\/<\/a>)<\/i><\/span><\/div>\n


In  this tutorial, I only deploy <\/span><\/span>one shipper<\/b> <\/span>(nginx logs of my Django app) on one machine, and <\/span><\/span>one server<\/b> <\/span>to play as logs indexer (redis, logstash, elasticsearch, kibana):<\/span>


1. <\/span><\/b>On the <\/span>indexer server, i<\/span><\/b>nstall and run <\/span>Redis<\/span>: <\/b>

http:\/\/iambusychangingtheworld.blogspot.com\/2013\/11\/install-redis-and-run-as-service.html<\/a>

2. O<\/b>n the <\/span>indexer server, i<\/span><\/b>nstall and run <\/span>ElasticSearch<\/span><\/span>:<\/b>

$ sudo aptitude install openjdk-6-jre<\/b><\/i><\/span>
$ wget https:\/\/download.elasticsearch.org\/elasticsearch\/elasticsearch\/elasticsearch-0.90.7.deb<\/b><\/i><\/span>
$ sudo dpkg -i elasticsearch<\/b><\/i><\/span>


3. O<\/b>n the <\/span>indexer server, <\/span><\/span><\/b>download, create config and run <\/span>Logtash<\/span> <\/span>to get log from <\/span>Redis<\/span> <\/span>and store them to <\/span>ElasticSearch<\/span>:<\/b>

+ Download Logtash:<\/span>

$ sudo mkdir \/opt\/logstash \/etc\/logstash<\/b><\/i><\/span>
$ sudo cd \/opt\/logstash<\/b><\/i><\/span>
$ sudo wget https:\/\/download.elasticsearch.org\/logstash\/logstash\/logstash-1.2.2-flatjar.jar<\/b><\/i><\/span>


+ Create Logtash config file <\/span><\/span>\/etc\/logstash\/logstash-indexer.conf<\/b> <\/span>with the following content:<\/span>

input {<\/i><\/span>
        redis {<\/i><\/span>
                host => “127.0.0.1”<\/i><\/span>
                data_type => “list”<\/i><\/span>
                key => “logstash”<\/i><\/span>
                codec => json<\/i><\/span>
        }<\/i><\/span>
}<\/i><\/span>
output {<\/i><\/span>
        elasticsearch {<\/i><\/span>
                embedded => true<\/i><\/span>
        }<\/i><\/span>
}<\/i><\/span>


+ Run Logstash, this will also activate the Kibana web interface on port 9292:<\/span>

$ java -jar \/opt\/logstash\/logstash-1.2.2-flatjar.jar agent -f \/etc\/logstash\/logstash-indexer.conf — web<\/b><\/i><\/span>


 4. On the <\/span>shipper machine<\/span> <\/span>(my computer), download Logstash, and create config file for Logtash to copy my Django app’s logs to the <\/span>indexer server<\/span>:<\/b>

+ Download Logstash:<\/span>

$ sudo mkdir \/opt\/logstash \/etc\/logstash<\/b><\/i><\/span>
$ sudo cd \/opt\/logstash<\/b><\/i><\/span>
$ sudo wget https:\/\/download.elasticsearch.org\/logstash\/logstash\/logstash-1.2.2-flatjar.jar<\/b><\/i><\/span>

+ Create a config file at <\/span><\/span>\/etc\/logstash\/logstash-shipper.conf <\/span><\/b>for Logstash to copy logs file redis at the indexer server:<\/span>

input {
        file {
                path => “\/home\/projects\/logs\/*ecap.log”
                type => “nginx”
        }
}
output {
        redis {
                host => “indexer.server.ip”
                data_type => “list”
                key => “logstash”
        }
}<\/i><\/span>


+ Run Logstash:<\/span>

$ java -jar \/opt\/logstash\/logstash-1.2.2-flatjar.jar agent -f \/etc\/logstash\/logstash-shipper.conf<\/b><\/i><\/span>


<\/b><\/span>5. From a random machine on my network, open browser to access the <\/span>kibana<\/span> <\/span>web interface to manage all the logs:<\/b>

<\/p>\n

\"\"<\/a><\/div>\n




From now on, If I want to monitor any services’s logs, I just need to run a Logstash instance on the server which runs that service.<\/span>


But, there is one annoying thing<\/b><\/span>: the CPU usages on the indexer server is very high. It’s because I’m running all the services (logstash, redis, elasticsearch, kibana) on a same server, and the java processes consume a lot of CPU. Look at the following htop screenshots and you will see:<\/span>

<\/p>\n