Fix erroneous DM Splunk Missing Forwarders alert

For some time now Splunk has been alerting me to “missing” forwarders even though all of those forwarders are working perfectly fine. It turns out to be a glitch in the Deployment MonitorĀ app. After much digging I found this Splunk article which explains it:

https://answers.splunk.com/answers/188784/after-update-to-splunk-enterprise-62-why-does-the.html

The fix is fairly simple, thankfully. You have to edit the macros.conf of the Deployment Monitor app to add this small snippet right before the first pipe:

NOT eventType=*

The default path for this configuration file is:

/opt/splunk/etc/apps/SplunkDeploymentMonitor/default/macros.conf

The relevant stanza in my macros.conf is below:

[forwarder_metrics]
definition = index="_internal" source="*metrics.log" group=tcpin_connections NOT eventType=* | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | eval connectionType=case(fwdType=="uf","universal forwarder", fwdType=="lwf", "lightweight forwarder",fwdType=="full", "heavy forwarder", connectionType=="cooked" or connectionType=="cookedSSL","Splunk forwarder", connectionType=="raw" or connectionType=="rawSSL","legacy forwarder")| eval build=if(isnull(build),"n/a",build) | eval version=if(isnull(version),"pre 4.2",version) | eval guid=if(isnull(guid),sourceHost,guid) | eval os=if(isnull(os),"n/a",os)| eval arch=if(isnull(arch),"n/a",arch) | fields connectionType sourceIp sourceHost sourcePort destPort kb tcp_eps tcp_Kprocessed tcp_KBps splunk_server build version os arch guid

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.