How Rules and Indicators Reach Your Hosts

Changes you make in IDSTower are not pushed to your Suricata hosts the moment you save them, instead each host pulls its own content from IDSTower on a schedule.

Two small update scripts run on every managed host, they ask IDSTower for the rules and the indicators that belong to that host, compare what they receive with what is already on disk, and apply the new content only when something actually changed.

So, when does a change take effect?

  • A rules change (enabling, disabling or editing a rule, importing rules from a file, or a rules feed update) is picked up by the next rules update run, every 5 minutes.

  • An indicators change (IP Reputation categories and indicators, FQDN and SHA256 datasets) is picked up by the next indicators update run, also every 5 minutes, scheduled two minutes after the rules run so the two never start together.

  • Each run begins with a random delay of up to 2 minutes, this spreads a large cluster’s pulls over the interval instead of having every host hit IDSTower in the same second, so in practice a change lands within 5 to 7 minutes.

  • Once the new content is written to disk it is applied to the running Suricata with a live rule reload, without dropping traffic, when a live reload is not possible the script restarts the Suricata service instead so the change still takes effect.

Nothing has to be clicked in IDSTower to start this, and on the IDSTower side there is no extra delay either: the rules and indicator files are compiled once and cached, and a change marks the cached copy stale immediately, so the next host pull is served the new content. The GeneratedPayload settings in the Configuration Guide control that cache.

You can watch this happen for every Suricata instance in the Cluster Hosts Tab: Rules Updated and IOCs Updated show how long ago that instance last pulled its rules and its indicators from IDSTower, hover over either one for the exact time.

What Runs on Each Host

IDSTower installs its scheduled jobs in the root crontab of every managed host. Each job is written with the #Ansible: <job name> marker comment that identifies it as IDSTower-managed, so sudo crontab -l on a host shows four jobs over eight lines:

#Ansible: IDSTower Host Heartbeat
* * * * * bash /etc/idstower/hostHealthHeartbeat.sh
#Ansible: IDSTower Suricata Rules Update
*/5 * * * * bash /etc/suricata/suricata_rules_update.sh
#Ansible: IDSTower Suricata Indicators Update
2-59/5 * * * * bash /etc/suricata/suricata_indicators_update.sh
#Ansible: IDSTower Suricata logs deletion
*/15 * * * * bash /etc/suricata/suricata_logs_deletion.sh

Leave those marker comments in place, they are how IDSTower finds its own jobs when it re-applies them or removes them again.

  • Host heartbeat, every minute, this is what keeps each host’s metrics and health current in the Cluster Hosts tab, it never touches rules or indicators.

  • Rules update, at minute 0, 5, 10, … of every hour.

  • Indicators update, at minute 2, 7, 12, …, half an interval after the rules update. Both scripts take turns through a shared lock, so starting them in the same minute would only make one of them wait.

  • Logs deletion, unrelated to rules and indicators, it enforces the log retention and disk utilization thresholds from the cluster’s Log Management settings (every 15 minutes by default, and configurable there).

What a Run Does

  1. Skips the run entirely, before waiting for anything, if an IDSTower operation (a configuration deployment, a service start/stop/restart) is in progress on the host, since restarting Suricata underneath a running operation would make that operation fail and roll back. This is logged, and the next scheduled run applies the pending change.

  2. Waits out its random start delay of up to 2 minutes.

  3. Takes the lock it shares with the other update script. The lock is taken after the delay on purpose: holding it across the delay would keep it busy for up to two minutes every run and make IDSTower operations wait for nothing.

  4. Checks for a running IDSTower operation a second time, since one may have started while the run was waiting, and skips the run if it finds one.

  5. Requests the host’s compiled content from the IDSTower API, once for each Suricata instance on the host. If IDSTower is still compiling the ruleset it answers 202 Accepted with a Retry-After header, and the script waits and retries (up to three times) instead of giving up until the next cron run.

  6. Compares the checksum of what it received with the file already on disk, an identical file means there is nothing to do and the run ends without touching Suricata, unless an earlier run had to postpone loading its content (see below).

  7. Writes the new file and applies it, either with a live rule reload over Suricata’s unix command socket, or by restarting the Suricata service when no usable socket is available. A live reload that fails is deliberately not followed by a restart: an invalid ruleset would then take the IDS down entirely, whereas a failed reload leaves the previous ruleset running. A failed reload is not repeated on later runs either, since a ruleset Suricata rejects would be rejected again every few minutes. Its error is in the script log, and the next change from IDSTower, or running suricata_rules_update.sh by hand with --force (see below), tries again. --force on the indicators script also rewrites the IP Reputation categories, which restarts Suricata.

A restart is checked against the operation marker one last time immediately before it happens. If an operation started in the meantime the service is left alone, the new content stays on disk, and the script records that the instance still needs a reload or restart in /var/run/idstower/detection-content-pending/instance-<name>. A later run makes it once the operation has finished, even though its files are up to date by then, and logs “Applying the update postponed by an earlier run”: a reload by the next run of either script, a restart by the next indicators update.

Updating the IP Reputation categories file is the one change that always requires a restart, Suricata reads that file only at startup.

Applying a Change Immediately

If you do not want to wait for the next scheduled run, run the same script by hand on the host, --skip-delay drops the random start delay:

sudo bash /etc/suricata/suricata_rules_update.sh --skip-delay
sudo bash /etc/suricata/suricata_indicators_update.sh --skip-delay

sudo is required, the scripts read the host’s API credentials from /etc/idstower/idstower.conf (root-owned, mode 0600), write into the Suricata configuration directory, and manage the Suricata service.

Add --force to re-download and re-apply the content even when it has not changed, --debug to see what the script is doing, and --help for the full list of options.

Once the script has pulled the content, the instance’s Rules Updated or IOCs Updated time in the Cluster Hosts tab resets to a few seconds ago.

Changing the Update Interval

The 5 minute interval is fixed in this version, it is not exposed in the IDSTower interface or in appsettings.json, it lives in the automation that configures the host.

Editing the crontab on a host by hand does not survive either, IDSTower re-applies both cron entries every time the configuration is deployed to that host, overwriting your change. A configurable interval is a known request from several customers, if the default does not suit your deployment please raise it with support so it can be prioritized.

When a Change Did Not Arrive

  • The last pull, the Cluster Hosts tab shows for every Suricata instance how long ago it last pulled its rules (Rules Updated) and its indicators (IOCs Updated). A recent time means the host is fetching on schedule, so a change that is still not active was received but not applied, and the script log below says why. A time that reads never, or turns yellow after 15 minutes without a pull, means the host has stopped fetching, check the schedule and the host’s heartbeat below.

  • The script log, /var/log/idstower/rules_updates.log and /var/log/idstower/indicators_updates.log, every run records whether the local file was already up to date, what it downloaded, and whether Suricata was reloaded or restarted.

  • A postponed load, a file in /var/run/idstower/detection-content-pending/ names an instance whose new content is on disk but not yet loaded, and holds reload or restart. Once no IDSTower operation is in progress, the next run makes it and removes the file (a restart, the next indicators update). A reboot empties the directory, since Suricata loads all its content when it starts. A reload refused because another instance holds this instance’s unix-command socket stays pending, with a warning in the log on every run, until each instance has a socket of its own again.

  • The schedule, sudo crontab -l, if the entries are missing, deploy the configuration to the host again from IDSTower to re-create them.

  • The Suricata service, sudo systemctl status suricata-instance-<name>.service and sudo journalctl -ex -u suricata-instance-<name>.service show why Suricata refused a new ruleset.

  • The host in IDSTower, the Cluster Hosts Tab shows each host’s health and its Suricata service state, a host whose heartbeat has stopped is not applying updates either.

Next, we will see features offered by the IDS Rules Management interface