Here are the repo contain an example of a custom filter I made for this Windows Event Log in Syslog Format
Why am I sending Windows Event Log in Syslog Format? π
Because I can, long story short, my use case or company policy or requirement doesn't want me to use Splunk UF to act as an intermediate forwarder to forward Windows Event Logs but wants Windows Machine to send logs using nxlog and deliver logs in Syslog format. So here we are sending Windows Event Log in Syslog formatπ.
Overview Topology
This is the same as the Splunk Test Lab I made, so you can see the SC4S container and the NXLog
The setup
I'm using NXLog CE to forward logs from the Windows endpoint
By using NXLog I convert Windows Event Logs into JSON after that I wrap it with Syslog Format
About the nxlog configuration file, you have to read the documentation for configuration on your own. I use this configuration line to convert logs to JSON and wrap it with Syslog format:
What is SC4S and how to create a custom Parser/Filter?
SC4S is Splunk Connect for Syslog, a Syslog intermediate for Splunk that parses and indexes some known vendor products using Syslog.
π The question is if you have a custom application that is using Syslog and you want to support it then now what, or does your SC4S keep sending you event sc4s:fallback
sc4s:fallback is when you don't have a parser or a filter for that type of log, product, or vendor, and most likely you will have to develop or create one for that
Where to create an SC4S custom parser?
You navigate to /opt/sc4s/config/app_parser/syslog/
This folder is where SC4S stores your custom parser configuration
Or Xml Configration file which I'll never get to use
How do I approach this?
In the beginning, I didn't know how to do this because the documentation was quite limited, but luckily the open source SC4S so I can take a look at their code and find out how they create a parser and hopefully I can't reverse engineer the process and create one of my own.
Reading Reddit post and their GitHub repo helps a lot -> found out create a custom parser is a normal thing in SC4S -> now I know I can do it
Copy other parser configurations and testing them using the local lab is super helpful -> I create a guide repo for you to create an SC4S test lab above
Takeaway and my to-do
Because the NXLog Windows Event Log Schema is different from Splunk UF and I also create different source types for the Windows Event Log in Json via Syslog
There are like 3 field that are different
Because I named the sourcetype is os_win_syslog -> the log I forwarded is not CIM compatible (Not only the sourcetype is the only problem)-> So it is not good because in Splunk world CIM πCompatible is everything (it normalize everything and make a standard in Splunk)
Also sending windows event log in JSON is fine but if you want to leverage the CIM, I think I should send it in XML rather than JSON (Only support in NXLog EE)
Aug 11 2023
---
NXLog has a different schema -> changing the source type will not help and sending it using XML format will also not help because of NXLog, the only way is that you have a transformer between the NXLog and the Splunk (right now is sc4s -> but it is not going to do a great job to act as a Transformer)
The alternative solution could be Cribl Stream as an SC4S replacement (because it can parse and transform logs) or you buy the NXLog EE -> which will allow you to do Splunk Windows Event Log schema
About Syslog Format
There are 3 type of Syslog Format:
BSD
IETF
Snare
Conclusion
Fun ride, please future developer or god of technology please make the all the format in one or create a god parser or something π
block parser app-syslog-winevent-json() {
channel {
# In the sc4s documentation don't mention this at all you need to read the GitHub repo to know
# This exist: json-parser
parser {
json-parser(
prefix('.values.')
);
};
rewrite {
#set defaults these values can be overridden at run time by splunk_metadata.csv
r_set_splunk_dest_default(
index("main")
source("os_win_syslog")
sourcetype('os_win_syslog')
#this value is used to lookup runtime settings such as index from splunk_metadata.csv
vendor("Microsoft")
product("Windows")
template("t_msg_only")
);
};
};
};
application app-syslog-winevent-json[sc4s-syslog] {
parser { app-syslog-winevent-json(); };
};
block parser app-syslog-winevent-xml() {
channel {
# In the sc4s documentation don't mention this at all you need to read the GitHub repo to know
# This exist: json-parser also xml (parser)
parser {
xml(
prefix('.values.')
);
};
rewrite {
#set defaults these values can be overridden at run time by splunk_metadata.csv
r_set_splunk_dest_default(
index("main")
source("os_win_syslog")
sourcetype('os_win_syslog')
#this value is used to lookup runtime settings such as index from splunk_metadata.csv
vendor("Microsoft")
product("Windows")
template("t_msg_only")
);
};
};
};
application app-syslog-winevent-xml[sc4s-syslog] {
parser { app-syslog-winevent-xml(); };
};
