Sending events to ElasticSearch¶
This profile configures Filebeat to read events from suricata events files and send them to an ElasticSearch cluster.
The profile has the following best-practices enabled:
Filebeat is configured to send each event type to a different ElasticSearch Index, those indexes are:
suricata-alerts-%{+yyyy.MM.dd}: this index will contains the Suricata Alerts.
suricata-nsm-%{+yyyy.MM.dd}: this index will contains the NSM records.
suricata-stats-%{+yyyy.MM.dd}: contains the Suricata stats.
suricata-service-logs-%{+yyyy.MM.dd}: contains the suricata service logs.
Filebeat is configured to automatically load index templates to the ElasticSearch cluster so that the alerts/events are stored correctly.
The profile allows you to easily configure authentication if needed.
Note
Loading the index template can be turned off with the Enable Index Template Setup setting, for example when your ElasticSearch cluster manages its own index templates. Filebeat still starts and ships events with it disabled, but ElasticSearch then indexes them using its own dynamic mapping defaults: the raised field limits this profile sets are not applied, and the events are written to regular indices instead of data streams.
“no matching index template found for data stream”¶
Filebeat has stopped shipping, nothing new arrives in ElasticSearch, and /var/log/filebeat/filebeat-*.ndjson on the Suricata host repeats:
no matching index template found for data stream [idstower-suricata]
This is the signature of a cluster still running version 2.0.0 of this profile. That version names the index template idstower-suricata but gives it the index pattern suricata-*. Filebeat 8 and 9 create a data stream named after the template name, so ElasticSearch looks for a template whose pattern matches idstower-suricata, suricata-* does not match it, and the output connection stays failed, no events are indexed. Nothing in the IDSTower interface shows this, the message only appears in Filebeat’s own log on the host.
Version 3.0.0 of the profile fixes it: the template is named suricata with the pattern suricata*, which does match. Upgrade the cluster’s Filebeat configuration profile to 3.0.0 — a cluster with the upgrade pending shows a Profile Upgrade Required banner, and the procedure, including the two checks the Filebeat profile upgrade needs before you deploy it, is in step 8 of Upgrade Guide from v2.9.x to v3.0.1.
If you have customised the profile and cannot take the upgrade yet, the same symptom clears by editing your profile’s Filebeat template so that setup.template.name is matched by setup.template.pattern (name suricata with pattern suricata*), then redeploying the configuration to the hosts.