I used to complain about how slow Filebeat is after restarting. I did not want to restart Filebeat because my ELK stack would take at least one hour and a half to be back to normal.

Also, it was very annoying that doing live tutorials to Data Engineers and Developers regarding the ELK stack was almost a gamble. I knew logs will arrive to ElasticSearch, the problem was that I didn’t know exactly when. The bottleneck was Filebeat but I did not know how to fix it.

The solution is this: keep Filebeat fast by removing unwanted logs. When Filebeat is restarted, he starts to crawl again all the logs of all the inputs that you have specified in the filebeat.yml. Some of these logs are useless, some are already stored in a safe place, some are duplicates, some are pure garbage! (For example, I crawl a “message” queue from rsyslog and receive a lot of unwanted information). Just as an illustration, the other day my rsyslog “message” queue size was 24gb, out of which only 1.5gb were useful logs. No wonder Filebeat took to much to start, it had to crawl at least 24gb of lines every time it was restarted!

If you just create a script that periodically gets rid of old unwanted logs, you can speed Filebeat. In the case that you want to keep your logs, you can try the Logrotate tool as well.