Logstash
Poor man log parser
Tricks Worth Reading
https://www.elastic.co/elasticon/conf/2016/sf/dive-deep-with-logstash-from-pipelines-to-persistent-queues (one of the most detail slides about Logstash, I would recommend reading this for a better understanding)
https://www.elastic.co/blog/a-practical-introduction-to-logstash (read the hold thing and maybe you won't need to read the rest of this blog)
https://github.com/hellosign/logstash-fundamentals/tree/master/examples (God damn, why am I not found this sooner)
Research about Grok (Parsing Logs like a God)
Check all these refs
Everybody going to tell you if you are facing unstructured data and you need to structure it in a way to process it then Grok is the way. I'm not going to talk about the syntax and other fundamental stuff
Grok is dead simple and you just need to know that, so what are the questions that most people going to ask
How to write good grok patterns
Multiline??
Write good grok patterns
Ground rule
_grokparsefailure is very costly to performance -> try not to fail at parsing or fail faster (Why this is a thing, you should read this https://www.elastic.co/blog/do-you-grok-grok)
Avoid writing grok pattern from specific to general since grok match in order, grok miss is very expensive -> Destroy your performance
Using Anchor only match those patterns against the whole string from start to finish, and nothing else. (^ and $)
Don't be afraid to use conditional (also don't be like me, I'm has a habit that doesn't plan the conditional statement ahead and later ending up with a complex logic that is very hard to maintain, I'm trying to fix that problem)
Grok Example
How to write your own custom grok patterns
My thought on Logstash
Logstash is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite "stash.". Logstash has over 200 plugins, and you can write your own very easily as well.
Date: March 6, 2022
-----
I think it is a wonderful piece of software, it is simple at a high level and easy to use. I don't know about the performance (Using JVM) of a lot of plugins (Honestly I just use the default plugin but soon maybe I will try another or write one myself)
Date: Aug 13, 2023
-----
Writing plugins for logstash is much easier than writing SC4S custom parser.
Using Input, Filter, and Output
Logstash pipeline is fairly simple, it has 3 part which is input, filter, and output
Input
Logstash has various input sources (the documentation has all the supported sources). all the famous are up there. You could have multiple input source
So what happens when you declare an Input in logstash?
It creates a thread and
Filter
Output
Compex Logstash Configuration
Writing a Logstash pipeline to filter and normalize log is one of my favorites
Logstash Plugins
Logstash Configuration tips and tricks
Last updated
Was this helpful?