Forwarding Events to a SIEM

IDSTower does not ship SIEM-specific connectors. What it does ship and manage on every Suricata host is a logshipper (Filebeat), with two seeded configuration profiles - “Send events to ElasticSearch” and “Send events to Logstash” - and either one can feed a SIEM: use Logstash (or any Beats-compatible input) as the relay, or let a SIEM that reads from ElasticSearch consume the indices directly. Beyond those two profiles, the Filebeat configuration itself is fully customizable, so Filebeat can also deliver the events in the form your SIEM ingests - see Route C - customize the Filebeat configuration - and you can also run your own logshipper on the hosts instead - see Route D - use your own logshipper.

If you would like IDSTower to support your SIEM, or to deploy and manage a logshipper other than Filebeat, email support@idstower.com and tell us which one - these requests decide which integrations we build next.

The profile is selected, and its output hosts set, on the Logshipper settings page - see Set Logshipper (Filebeat) Settings.

What the logshipper sends

Filebeat tails Suricata’s EVE JSON output on each host - one event per line - and parses it before shipping, so the receiver gets the EVE fields as top-level fields, not an escaped JSON string. The files it follows are:

/var/log/suricata/instance-*/suricata-alerts*.json
/var/log/suricata/instance-*/suricata-nsm*.json
/var/log/suricata/instance-*/suricata-stats*.json
/var/log/suricata/instance-*/suricata.log.json

To every event, Filebeat adds:

  • data_type - one of suricata-alerts, suricata-nsm, suricata-stats or suricata-service-logs; use it to route or filter each stream separately.

  • suricata.instance - the instance directory name (eg: instance-eth0), so events stay attributable to the IDS instance that produced them.

  • host.* - the sensor’s hostname and OS details, plus cloud, Docker and Kubernetes metadata where applicable.

Route A - via Logstash

This is the recommended route for ArcSight, Rapid7 InsightIDR, QRadar, Splunk and any other SIEM with a syslog, TCP/JSON or Beats input.

  1. Select the “Send events to Logstash” configuration profile for the Cluster’s logshipper.

  2. Set “Logstash Hosts” to your Logstash host(s), one per line, in host:port form (eg: ls1.example.local:5044); port 5044 is used when none is given.

  3. Set “Enable TLS” according to your Logstash Beats input, and deploy the settings to the Cluster.

  4. On the Logstash side, use a pipeline along these lines - the hosts, ports and stream selection below are placeholders, adapt them to your environment:

input {
  beats {
    port => 5044
    # ssl_enabled => true, plus the certificate settings, if TLS was enabled above
  }
}

filter {
  # The EVE fields are already top-level fields - no json filter is needed.
  # if [data_type] != "suricata-alerts" { drop {} }   # alerts only, for example
}

output {
  # Generic JSON over TCP - accepted by most SIEM collectors
  tcp {
    host => "siem-collector.example.local"
    port => 5140
    codec => json_lines
  }
}

  1. For a SIEM that expects CEF over syslog (ArcSight), replace the output with:

output {
  syslog {
    host => "arcsight-connector.example.local"
    port => 514
    protocol => "tcp"
    codec => cef
  }
}

Note

Logstash’s cef codec “can be used to decode CEF events into the Elastic Common Schema, or to encode ECS-compatible events into CEF” - encoding expects ECS-compatible events, so map the EVE fields you care about to ECS field names (with mutate/rename) before the output, or set the codec’s fields option explicitly.

Route B - via ElasticSearch

Select the “Send events to ElasticSearch” profile instead, set “ElasticSearch Hosts” (default port 9200), “Use HTTPS/TLS” and the authentication settings. Filebeat then writes one daily index per stream - suricata-alerts-YYYY.MM.dd, suricata-nsm-YYYY.MM.dd, suricata-stats-YYYY.MM.dd and suricata-service-logs-YYYY.MM.dd. Elastic SIEM, or any SIEM that reads from ElasticSearch, consumes these with the index pattern suricata-*.

Route C - customize the Filebeat configuration

When your SIEM can take the events straight from Filebeat - for example from a Kafka topic, a Redis list, or a file its own agent collects - you can change the logshipper’s configuration instead of adding a relay. The Filebeat configuration is fully customizable: on the Cluster’s configuration page, open the Advanced tab and edit filebeat.yml template under “Filebeat Configuration Files Templates” (see Cluster Configs Tab). IDSTower renders it for every host and writes it to /etc/filebeat/filebeat.yml when the configuration is deployed. Your edits are versioned, so you can revert to an earlier version at any time.

Use the template of the profile your Cluster already runs as the base for your edits. It already defines everything described in What the logshipper sends - the four inputs that follow Suricata’s EVE files, the parsing that turns the EVE fields into top-level fields, and the data_type, suricata.instance and host.* fields - so keep those sections and change only what your SIEM needs:

  • The output - replace the output.logstash (or output.elasticsearch) section with the output your SIEM ingests. Filebeat can send to ElasticSearch, Logstash, Kafka, Redis, a local file or the console, one output at a time; see the “Configure the output” pages of the Filebeat documentation for each one’s settings.

  • The event format - Filebeat’s processors (rename, drop_fields, add_fields, script and others) reshape each event before it is sent, and the Kafka, Redis, file and console outputs accept a codec.format string that writes each event in exactly the layout your SIEM expects.

For example, to publish each stream to its own Kafka topic, replace the output section with the following - the host is a placeholder:

output.kafka:
  hosts: ["kafka1.example.local:9092"]
  topic: "%{[data_type]}"
  codec.json:
    pretty: false

The shipped output section reads its hosts and TLS setting from the Logshipper settings page through template variables (logshipper_outputHosts and logshipper_tls_enabled); an output you write yourself does not use those settings unless you keep the variables in it. The template is Jinja2 with IDSTower’s delimiters - {{ ... }} for values and [[% ... %]] for statements such as loops - and Filebeat’s own %{...} references pass through unchanged.

After deploying, check the result on a host: sudo filebeat test config -c /etc/filebeat/filebeat.yml validates the rendered file, and sudo filebeat test output -c /etc/filebeat/filebeat.yml checks that the configured output is reachable.

Note

Switching the Cluster to another Filebeat configuration profile, or upgrading it to a newer version of the profile, starts from that profile’s own template and does not carry your edits over. Keep a copy of your changes - the template’s version history holds them - and apply them again after such a change.

Route D - use your own logshipper

If your SIEM comes with its own collection agent, or your organization standardizes on a logshipper other than Filebeat, install it on the Suricata hosts and point it at the files Suricata writes. You do not need IDSTower’s logshipper for this: a Cluster can run without one, and for a Cluster that has none, the Cluster Actions menu offers Add Logshipper should you want it later. If IDSTower’s logshipper is installed as well, both read the same files independently, so make sure they do not deliver the same events to the same place.

IDSTower configures every Suricata instance to log into a directory of its own, /var/log/suricata/instance-<instance name>/, where it writes:

  • suricata-alerts-<YYYYMMDD-HHMMSS>.json - the alerts.

  • suricata-nsm-<YYYYMMDD-HHMMSS>.json - the network security monitoring events: the protocol and anomaly events enabled in the Cluster’s configuration (DNS, HTTP, TLS, files and others).

  • suricata-stats-<YYYYMMDD-HHMMSS>.json - Suricata’s performance statistics.

  • suricata.log.json - Suricata’s own service log as JSON; suricata.log holds the same log as text.

Point your logshipper at these patterns, the same ones IDSTower’s own Filebeat follows:

/var/log/suricata/instance-*/suricata-alerts*.json
/var/log/suricata/instance-*/suricata-nsm*.json
/var/log/suricata/instance-*/suricata-stats*.json
/var/log/suricata/instance-*/suricata.log.json

What your logshipper has to handle:

  • One JSON object per line - the files are in Suricata’s EVE format, and each event’s event_type field says what it is (alert, dns, stats and so on).

  • A new file every hour - Suricata starts a new file each hour with the start time in its name, so follow the patterns above rather than fixed file names, and make sure new files are picked up.

  • Old files are deleted - when the Cluster’s Log Management is enabled, IDSTower removes old files according to its retention and disk usage settings (see Management Capabilities), so the logshipper has to keep up with the data rather than rely on files staying on disk.

  • The instance name is in the path, not in the events - take it from the directory name (instance-<name>) if you need to know which instance produced an event; IDSTower’s Filebeat does the same for its suricata.instance field.

  • Several instances per host - each one has its own directory, which is why the patterns use instance-*.

On a Cluster whose Suricata configuration IDSTower does not manage (an existing deployment onboarded without configuration management), Suricata writes wherever the host’s own configuration says, so use those paths instead.

If you would rather have IDSTower deploy and manage the logshipper you use, email support@idstower.com with its name.

Vendor notes

  • Rapid7 InsightIDR - its documentation does not mention Suricata, but its Custom Logs event source ingests arbitrary logs through a Collector and “Rapid7 recommends using JSON or key-value pair (KVP) format”, received either by a Collector “listening on a network port for syslog data” or by watching/tailing a file. Point the Logstash output of Route A at that Collector.

  • ArcSight - there is no IDSTower-side ArcSight integration; the supported path is Logstash’s cef codec emitting CEF over syslog to a SmartConnector, as in step 5 above.

For any other product, consult your SIEM’s documentation for its Beats, syslog or JSON/TCP input and adapt the Logstash output accordingly, or, if it ingests from Kafka, Redis or files, adapt the Filebeat output itself as in Route C - customize the Filebeat configuration, or use its own agent as in Route D - use your own logshipper. If your SIEM is not covered here and you would like it to be, email support@idstower.com and tell us which one.

Verifying the pipeline

  1. On a Suricata host, confirm Suricata is producing events:

$ sudo tail -n 1 /var/log/suricata/instance-*/suricata-alerts*.json

  1. Confirm the logshipper is running - the Filebeat service state is shown per host on the Cluster’s Hosts page, and on the host itself:

$ sudo systemctl status filebeat
$ sudo tail -n 20 /var/log/filebeat/filebeat

  1. On the receiving side, count the events that arrived over the same period; if Logstash is in the path, add a temporary stdout { codec => rubydebug } output to see what it receives.