Feed Settings, Schedule and Failures

This page is the reference for the custom feed types listed in Importing Threat Intelligence Feeds. It explains the options in the Add New Feed dialog and on each feed’s settings card, and what IDSTower does with a feed once it is added. For the steps to add a feed, see the page for its format.

Authentication

The Authentication Type setting has four options:

Type

What IDSTower sends

None

No credentials.

Secret Code

No header. Every %secret-code% in the feed URL is replaced with the code when the feed is downloaded, for example https://feeds.example.com/%secret-code%/iocs.txt.

Basic Http Authentication

The username and password, as an HTTP Basic Authorization header.

API Key

The key as the whole value of the Authorization header, exactly as entered. No scheme such as Bearer is added.

Important

A Secret Code is only used if the URL contains %secret-code%. IDSTower does not check for the placeholder, so without it the code is silently ignored and the feed is downloaded without it.

Note

Currently, an API key can only contain letters, digits and the characters ! # $ % & ' * + - . ^ _ | ~. Verify Connection rejects a key with any other character, such as a space (Bearer <token>), /, =, :, @ or ,, with the error The format of value ‘…’ is invalid. A key changed to such a value on the feed’s card makes every update of the feed fail the same way.

Some feed types restrict the choice:

  • STIX/TAXII 2.0/2.1 feeds support None and Basic Http Authentication only. IDSTower sends HTTP Basic credentials on every TAXII request (empty ones with None); a Secret Code or an API Key is not used.

  • MISP API feeds require API Key. The dialog selects it when you choose this type, and Verify Connection refuses any other type.

  • Text, CSV, JSON and MISP Feed accept all four types.

Import settings

Ignore TLS Errors is set with the URL, in the first step of the Add New Feed dialog; the other settings are in its second step. All of them can be changed later on the feed’s settings card. Which ones are shown depends on the feed type.

Setting

Values

Default

Feed types

Ignore TLS Errors

Yes, No

No

All

Update Frequency

Every Day, Every Hour

Every Day

All

Assigned Indicators Score

1 (Very Low) to 10 (Critical)

5

Text, CSV, JSON, TAXII

Expire Indicators After

1, 2, 4, 6, 8, 10 or 12 weeks, or Never Expire

6 weeks

All

Initial Download Range

0 to 3650 days

30 days

MISP API, TAXII

Items Per Page

100 to 10000

10000

MISP API, TAXII

  • Ignore TLS Errors skips certificate validation for an https:// URL. Enable it only for a server you trust whose certificate IDSTower cannot verify, such as a self-signed one.

  • Update Frequency sets how often the feed is downloaded, see How updates are scheduled.

  • Assigned Indicators Score is the score given to the indicators the feed imports. MISP feeds take the score from the MISP event instead, see Indicator score.

  • Expire Indicators After sets when the indicators imported from the feed expire. Each time the feed is imported and still lists an indicator, its expiry date moves to that interval from now if that is later than the date it has, and is never moved earlier: an indicator you set by hand to Never Expire, or to a later date, keeps that expiry (see Setting an Expiry Date). Lowering this setting therefore does not shorten the expiry of indicators the feed has already imported.

  • Initial Download Range sets how far back the first download goes. 0 downloads no history, only items added from the first download on.

  • Items Per Page sets how many items each request downloads. A TAXII server can impose a lower limit, which IDSTower then uses.

Changing Initial Download Range or Items Per Page on an existing MISP API or TAXII feed makes its next update start the download again from the beginning of the range.

How updates are scheduled

  • IDSTower checks every minute for enabled feeds that are due.

  • A feed is due when its last successful update is older than its Update Frequency (24 hours or 1 hour). A new feed has no successful update yet, so its first download starts within about a minute of adding it.

  • Text, CSV, JSON and MISP Feed downloads are compared with the previous download. When the content is unchanged, the scheduled update skips the import.

  • MISP API and TAXII feeds download incrementally: the first update downloads the Initial Download Range, later updates only the newer items.

  • Scheduled updates do not run while the IDSTower license is missing or invalid.

Clicking Update on a feed’s card runs an update immediately:

  • For Text, CSV, JSON and MISP Feed, it imports the downloaded content even if it is unchanged.

  • For MISP API and TAXII, it runs the next incremental update.

  • It runs even if the feed is paused after failures, see Failures and pausing.

  • It is not available while the card has unsaved changes; click Save Changes first.

  • If an update of the feed is already running, it shows a notice instead and the running update carries on.

Failures and pausing

An update fails when the feed cannot be downloaded, parsed or imported. IDSTower records the error and tries again at the next check, a minute later.

  • After 3 consecutive failures the feed is paused for 1 hour. Scheduled updates skip it until the hour has passed, then try once more; each further failure pauses it for another hour.

  • MISP API and TAXII feeds set to Every Hour have a further limit: after 10 consecutive failures, scheduled updates stop until a day has passed since the feed’s last successful update.

  • A successful update clears the failure count and the pause.

To resume a paused feed, do one of the following:

  • Wait. The feed is retried automatically when the pause ends, or, for a feed over the 10-failure limit, once a day has passed since its last successful update.

  • Click Update on the feed’s card. It runs regardless of the pause; if it fails, it counts as a further failure and the pause starts again for another hour.

  • Disable the feed, then enable it again. This clears the failure count and the pause, and also resets the feed’s download state, see Disabling and deleting a feed.

Saving the feed’s settings does not lift a pause.

Note

The feed card does not currently show that a feed is paused or how often it has failed. To find out why a feed is not updating, click Update and read the error it reports, or check the IDSTower application log (in /var/log/idstower/ by default, see Configuration Guide), which records each failure and each pause.

Disabling and deleting a feed

Disabling a feed (the switch on its card) stops its updates and resets its download state: the time of its last update, the fingerprint of its last download, the MISP API and TAXII download position, the failure count and the pause. When you enable it again, it is downloaded and imported from scratch within about a minute, and a MISP API or TAXII feed starts again from its Initial Download Range.

Deleting a feed (the Delete button, shown for custom feeds only) removes the feed and its settings. The indicators it imported, or the rules for an IDS Rules feed, are not deleted with it and stay in IDSTower.

What each feed type imports

Feed type

What is imported

Text, CSV, JSON

Each value is trimmed and lower-cased, then recognised as an IPv4 or IPv6 address (optionally with a port, as in 203.0.113.5:443), a domain name or a SHA-256 hash. Any other value, such as an MD5 or SHA-1 hash, a URL, an email address or a CIDR range, is skipped without an error.

MISP Feed, MISP API

Attributes with to_ids set, in the categories Payload delivery, Artifacts dropped, Payload installation or Network activity, of type ip-src, ip-dst, ip-src|port, ip-dst|port, hostname, domain, domain|ip, hostname|port or sha256. For domain|ip and hostname|port only the domain is kept.

STIX/TAXII 2.0/2.1

Indicator objects whose pattern is a single comparison on ipv4-addr:value, ipv6-addr:value, domain-name:value or file:hashes.'SHA-256', for example [ipv4-addr:value = '198.51.100.7']. Patterns that combine several comparisons, IP ranges and other hash types are skipped. IDSTower reads every collection on the server that is marked as readable.

Indicator score

  • Text, CSV, JSON and TAXII feeds give each indicator the feed’s Assigned Indicators Score. If the indicator already exists in IDSTower with a higher score, for example from another feed, the higher score is kept.

  • MISP Feed and MISP API take the score from the threat level of the MISP event: High is 10, Medium 5, Low 2 and Undefined 1. This score replaces the score of an indicator that already exists, even a higher one.

What Verify Connection checks

Verify Connection first checks that no other feed uses the same name or URL, then:

Feed type

Check

Text, CSV, JSON

Downloads the file with the chosen authentication.

MISP Feed

Downloads and reads manifest.json. The event files are not downloaded.

MISP API

Sends one search request, limited to 10 attributes.

STIX/TAXII 2.0/2.1

Runs TAXII server discovery and detects the version. The collections are not read.

Verify Connection does not parse the content. A wrong CSV column name or index, a JSONPath expression that matches nothing, or a file without any supported indicators only shows once the feed is imported: as an update error (click Update to see it), or as a feed that imports no indicators.

URL rules

  • The URL must start with http:// or https://, and must not point to localhost or 127.0.0.1.

  • No two feeds can share a name (compared case-insensitively) or a URL.

  • Text, CSV, JSON: the URL of the file. For Text and CSV feeds, a URL ending in .tar.gz or .tgz is unpacked, and the first .txt, .text, .csv, .rules or .rule file in the archive is imported. Other compressed formats are not unpacked.

  • MISP Feed: the URL of the folder that holds manifest.json, for example https://feeds.example.com/misp/ for https://feeds.example.com/misp/manifest.json. IDSTower adds manifest.json and the event file names to it.

  • MISP API: the base URL of the MISP instance, for example https://misp.example.com. IDSTower sends its searches to /attributes/restSearch under it.

  • STIX/TAXII 2.0/2.1: the base URL of the TAXII server, without a discovery path. IDSTower tries /taxii2/ (TAXII 2.1) and then /taxii/ (TAXII 2.0) under it.