idstools package

Submodules

idstools.maps module

Provide mappings from ID’s to descriptions.

Includes mapping classes for event ID messages and classification information.

class idstools.maps.ClassificationMap(fileobj=None)[source]

Bases: object

ClassificationMap maps classification IDs and names to a dict object describing a classification.

Parameters:fileobj – (Optional) A file like object to load classifications from on initialization.

The classification dicts stored in the map have the following fields:

  • name (string)
  • description (string)
  • priority (int)

Example:

>>> from idstools import maps
>>> classmap = maps.ClassificationMap()
>>> classmap.load_from_file(open("tests/classification.config"))

>>> classmap.get(3)
{'priority': 2, 'name': 'bad-unknown', 'description': 'Potentially Bad Traffic'}
>>> classmap.get_by_name("bad-unknown")
{'priority': 2, 'name': 'bad-unknown', 'description': 'Potentially Bad Traffic'}
add(classification)[source]

Add a classification to the map.

get(class_id)[source]

Get a classification by ID.

Parameters:class_id – The classification ID to get.
Returns:A dict describing the classification or None.
get_by_name(name)[source]

Get a classification by name.

Parameters:name – The name of the classification
Returns:A dict describing the classification or None.
load_from_file(fileobj)[source]

Load classifications from a Snort style classification.config file object.

size()[source]
class idstools.maps.SignatureMap[source]

Bases: object

SignatureMap maps signature IDs to a signature info dict.

The signature map can be build up from classification.config, gen-msg.map, and new and old-style sid-msg.map files.

The dict’s in the map will have at a minimum the following fields:

  • gid (int)
  • sid (int)
  • msg (string)
  • refs (list of strings)

Signatures loaded from a new style sid-msg.map file will also have rev, classification and priority fields.

Example:

>>> from idstools import maps
>>> sigmap = maps.SignatureMap()
>>> sigmap.load_generator_map(open("tests/gen-msg.map"))
>>> sigmap.load_signature_map(open("tests/sid-msg-v2.map"))
>>> print(sigmap.get(1, 2495))
{'classification': 'misc-attack', 'rev': 8, 'priority': 0, 'gid': 1,
'sid': 2495,
'msg': 'GPL NETBIOS SMB DCEPRC ORPCThis request flood attempt',
'ref': ['bugtraq,8811', 'cve,2003-0813', 'nessus,12206',
'url,www.microsoft.com/technet/security/bulletin/MS04-011.mspx']}
get(generator_id, signature_id)[source]

Get signature info by generator_id and signature_id.

Parameters:
  • generator_id – The generator id of the signature to lookup.
  • signature_id – The signature id of the signature to lookup.

For convenience, if the generator_id is 3 and the signature is not found, a second lookup will be done using a generator_id of 1.

load_generator_map(fileobj)[source]

Load the generator message map (gen-msg.map) from a file-like object.

load_signature_map(fileobj, defaultgid=1)[source]

Load signature message map (sid-msg.map) from a file-like object.

size()[source]

idstools.net module

Module for network related operations.

idstools.net.get(url, fileobj, progress_hook=None)[source]

Perform a GET request against a URL writing the contents into the provideded file like object.

Parameters:
  • url – The URL to fetch
  • fileobj – The fileobj to write the content to
  • progress_hook – The function to call with progress updates
Returns:

Returns a tuple containing the number of bytes read and the result of the info() function from urllib2.urlopen().

Raises:

Exceptions from urllib2.urlopen() and writing to the provided fileobj may occur.

idstools.packet module

Provides basic packet decoding.

idstools.packet.decode_ethernet(pkt)[source]

Decode an ethernet packet.

idstools.packet.decode_icmp(pkt)[source]

Decode an ICMP packet.

idstools.packet.decode_icmp6(pkt)[source]

Decode an ICMPv6 packet.

idstools.packet.decode_ip(pkt)[source]

Decode an IP packet.

idstools.packet.decode_ip6(pkt)[source]

Decode an IPv6 packet.

idstools.packet.decode_tcp(pkt)[source]

Decode a TCP packet.

idstools.packet.decode_udp(pkt)[source]

Decode a UDP packet.

idstools.packet.printable_ethernet_addr(addr)[source]

Return a formatted ethernet address from its raw form.

idstools.rule module

Module for parsing Snort-like rules.

Parsing is done using regular expressions and the job of this module is to do its best at parsing out fields of interest from the rule rather than perform a sanity check.

The methods that parse multiple rules for a provided input (parse_file, parse_fileobj) return a list of rules instead of dict keyed by ID as its not the job of this module to detect or deal with duplicate signature IDs.

class idstools.rule.FlowbitResolver[source]

Bases: object

get_required_flowbits(rules)[source]
get_required_rules(rulemap, flowbits, include_enabled=False)[source]

Returns a list of rules that need to be enabled in order to satisfy the list of required flowbits.

getters = ['isset', 'isnotset']
parse_flowbit(flowbit)[source]
resolve(rules)[source]
set_required_flowbits(rules, required)[source]
setters = ['set', 'setx', 'unset', 'toggle']
class idstools.rule.Rule(enabled=None, action=None, group=None)[source]

Bases: dict

Class representing a rule.

The Rule class is a class that also acts like a dictionary.

Dictionary fields:

  • group: The group the rule belongs to, typically the filename.

  • enabled: True if rule is enabled (uncommented), False is

    disabled (commented)

  • action: The action of the rule (alert, pass, etc) as a

    string

  • direction: The direction string of the rule.

  • gid: The gid of the rule as an integer

  • sid: The sid of the rule as an integer

  • rev: The revision of the rule as an integer

  • msg: The rule message as a string

  • flowbits: List of flowbit options in the rule

  • metadata: Metadata values as a list

  • references: References as a list

  • classtype: The classification type

  • priority: The rule priority, 0 if not provided

  • raw: The raw rule as read from the file or buffer

Parameters:
  • enabled – Optional parameter to set the enabled state of the rule
  • action – Optional parameter to set the action of the rule
brief()[source]

A brief description of the rule.

Returns:A brief description of the rule
Return type:string
id

The ID of the rule.

Returns:A tuple (gid, sid) representing the ID of the rule
Return type:A tuple of 2 ints
idstr
idstools.rule.enable_flowbit_dependencies(rulemap)[source]

Helper function to resolve flowbits, wrapping the FlowbitResolver class.

idstools.rule.format_sidmsgmap(rule)[source]

Format a rule as a sid-msg.map entry.

idstools.rule.format_sidmsgmap_v2(rule)[source]

Format a rule as a v2 sid-msg.map entry.

eg: gid || sid || rev || classification || priority || msg || ref0 || refN

idstools.rule.parse(buf, group=None)[source]

Parse a single rule for a string buffer.

Parameters:buf – A string buffer containing a single Snort-like rule
Returns:An instance of of Rule representing the parsed rule
idstools.rule.parse_file(filename, group=None)[source]

Parse multiple rules from the provided filename.

Parameters:filename – Name of file to parse rules from
Returns:A list of Rule instances, one for each rule parsed
idstools.rule.parse_fileobj(fileobj, group=None)[source]

Parse multiple rules from a file like object.

Note: At this time rules must exist on one line.

Parameters:fileobj – A file like object to parse rules from.
Returns:A list of Rule instances, one for each rule parsed

idstools.snort module

class idstools.snort.SnortApp(config=None, path=None, os=None, dynamic_engine_lib=None)[source]

Bases: object

Snort represents the Snort application.

Parameters:
  • config – A dictionary configuration object. The dictionary can contain the same fields as the following parameters. Parameters take precedence over the config dictionary.
  • path – The path to the Snort binary.
dump_dynamic_rules(dynamic_detection_lib_dir, verbose=False)[source]
exists()[source]
find_dynamic_detection_lib_dir(prefix)[source]

Find the dynamic SO rule directory in prefix based on what we know about Snort.

get_arch()[source]
set_dynamic_engine_lib(dynamic_engine_lib, config)[source]
version()[source]

idstools.suricata module

class idstools.suricata.SuricataVersion(major, minor, patch, full, short, raw)

Bases: tuple

full

Alias for field number 3

major

Alias for field number 0

minor

Alias for field number 1

patch

Alias for field number 2

raw

Alias for field number 5

short

Alias for field number 4

idstools.suricata.get_path(program='suricata')[source]

Find Suricata in the shell path.

idstools.suricata.get_version(path=None)[source]

Get a SuricataVersion named tuple describing the version.

If no path argument is found, the envionment PATH will be searched.

idstools.unified2 module

Unified2 record and event reading.

Unified2 is a file format used by the Snort and Suricata IDS engines for logging events.

For more information on the unified2 file format see:

usage: from idstools import unified2
class idstools.unified2.AbstractDecoder(fields)[source]

Bases: object

Base class for decoders.

class idstools.unified2.Aggregator[source]

Bases: object

A class implementing something like the aggregator pattern to aggregate records until an event can be built.

add(record)[source]

Add a new record to aggregator.

Parameters:record – The decoded unified2 record to add.
Returns:If adding a new record allows an event to be completed, an Event will be returned.
flush()[source]

Flush the queue. This converts the records in the queue into an Event.

If using the Aggregator directly, you’ll want to call flush after adding all your records to get the final event.

Returns:An Event or None if there are no records.
class idstools.unified2.Event(event)[source]

Bases: dict

Event represents a unified2 event record with a dict-like interface.

Fields:

  • sensor-id
  • event-id
  • event-second
  • event-microsecond
  • signature-id
  • generator-id
  • signature-revision
  • classification-id
  • priority
  • source-ip
  • destination-ip
  • sport-itype
  • dport-icode
  • protocol
  • impact-flag
  • impact
  • blocked
  • mpls-label
  • vlan-id

Methods that return events rather than single records will also populate the fields packets and extra-data. These fields are lists of the Packet and ExtraData records associated with the event.

class idstools.unified2.EventDecoder(fields)[source]

Bases: idstools.unified2.AbstractDecoder

Decoder for event type records.

decode(buf)[source]

Decodes a buffer into an Event object.

decode_ip(addr)[source]
class idstools.unified2.ExtraData(*fields, **kwargs)[source]

Bases: dict

ExtraData represents a unified2 extra-data record with a dict like interface.

Fields:

  • event-type
  • event-length
  • sensor-id
  • event-id
  • event-second
  • type
  • data-type
  • data-length
  • data
class idstools.unified2.ExtraDataDecoder(fields)[source]

Bases: idstools.unified2.AbstractDecoder

Decoder for extra data type records.

decode(buf)[source]

Decodes a buffer into an ExtraData object.

class idstools.unified2.Field(name, length, fmt=None)[source]

Bases: object

A class to represent a field in a unified2 record. Used for building the decoders.

fmt

Builds a format string for struct.unpack.

class idstools.unified2.FileEventReader(*files)[source]

Bases: object

FileEventReader reads records from one or more filenames and aggregates them into events.

Parameters:files... – One or more files to read events from.

Example:

reader = unified2.FileEventReader("unified2.log.1382627941",
                                  "unified2.log.1382627966)
for event in reader:
    print(event)
next()[source]

Return the next Event or None if EOF.

class idstools.unified2.FileRecordReader(*files)[source]

Bases: object

FileRecordReader reads and decodes unified2 records from one or more files supplied by filename.

Parameters:files... – One or more filenames to read records from.

Example:

reader = unified2.RecordReader("unified2.log.1382627941",
                               "unified2.log.1382627966)
for record in reader:
    print(record)
next()[source]

Return the next record or None if EOF.

Records returned will be one of the types Event, Packet, ExtraData or Unknown if the record is of an unknown type.

tell()[source]

Returns the current filename and offset.

class idstools.unified2.Packet(*fields, **kwargs)[source]

Bases: dict

Packet represents a unified2 packet record with a dict-like interface.

Fields:

  • sensor-id
  • event-id
  • event-second
  • packet-second
  • packet-microsecond
  • linktype
  • length
  • data
class idstools.unified2.PacketDecoder(fields)[source]

Bases: idstools.unified2.AbstractDecoder

Decoder for packet type records.

decode(buf)[source]

Decodes a buffer into a Packet object.

class idstools.unified2.RecordReader(fileobj)[source]

Bases: object

RecordReader reads and decodes unified2 records from a file-like object.

Parameters:fileobj – The file-like object to read from.

Example:

fileobj = open("/var/log/snort/merged.log.1382627987", "rb")
reader = RecordReader(fileobj):
for record in reader:
    print(record)
next()[source]

Return the next record or None if EOF.

Records returned will be one of the types Event, Packet, ExtraData or Unknown if the record is of an unknown type.

tell()[source]

Get the current offset in the underlying file object.

class idstools.unified2.SpoolEventReader(directory, prefix, follow=False, delete=False, bookmark=False)[source]

Bases: object

SpoolEventReader reads records from a unified2 spool directory and aggregates them into events.

Required parameters:

Parameters:
  • directory – Path to unified2 spool directory.
  • prefix – Filename prefix for unified2 log files.

Optional parameters:

Parameters:
  • follow – Set to true to follow the log files. Reading will wait until an event is available before returning.
  • delete – If True, unified2 files will be deleted when reading has moved onto the next one.
  • bookmark – If True, the reader will remember its location and start reading from the bookmarked location on initialization.

Example:

reader = unified2.SpoolEventReader("/var/log/snort", "unified2.log")
for event in reader:
    print(event)
next()[source]

Return the next Event.

If in follow mode and EOF is head, this method will sleep and and try again.

rollover_hook(closed, opened)[source]
tell()[source]

See SpoolRecordReader.tell().

class idstools.unified2.SpoolRecordReader(directory, prefix, init_filename=None, init_offset=None, follow=False, rollover_hook=None)[source]

Bases: object

SpoolRecordReader reads and decodes records from a unified2 spool directory.

Required parameters:

Parameters:
  • directory – Path to unified2 spool directory.
  • prefix – Filename prefix for unified2 log files.

Optional parameters:

Parameters:
  • init_filename – Filename open on initialization.
  • init_offset – Offset to seek to on initialization.
  • follow – Set to true if reading should wait for the next record to become available.
  • rollover_hook – Function to call on rollover of log file, the first parameter being the filename being closed, the second being the filename being opened.

Example with following and rollover deletion:

def rollover_hook(closed, opened):
    os.unlink(closed)

reader = unified2.SpoolRecordReader("/var/log/snort",
    "unified2.log", rollover_hook = rollover_hook,
    follow = True)
for record in reader:
    print(record)
get_filenames()[source]

Return the filenames (sorted) from the spool directory.

next()[source]

Return the next record or None if EOF.

If in follow mode and EOF, this method will sleep and and try again.

Returns:A record of type Event, Packet, ExtraData or Unknown if the record is of an unknown type.
open_file(filename)[source]
open_next()[source]

Open the next available file. If a new file is opened its filename will be returned, otherwise None will be returned.

tell()[source]

Return a tuple containing the filename and offset of the file currently being processed.

class idstools.unified2.Unified2Bookmark(directory=None, prefix=None, filename=None)[source]

Bases: object

Class to represent a “bookmark” for unified2 spool directories.

get()[source]

Get the current bookmark.

Returns a tuple of filename and offset.

update(filename, offset)[source]

Update the bookmark with the given filename and offset.

class idstools.unified2.Unknown(record_type, buf)[source]

Bases: object

Class to represent an unknown record type.

In the unlikely case that a record is of an unknown type, an instance of Unknown will be used to hold the record type and buffer.

idstools.unified2.decode_record(record_type, buf)[source]

Decodes a raw record into an object representing the record.

Parameters:
  • record_type – The type of record.
  • buf – Buffer containing the raw record.
Returns:

The decoded record as a Event, Packet, ExtraData or Unknown if the record is of an unknown type.

idstools.unified2.read_record(fileobj)[source]

Reads a unified2 record from the provided file object.

Parameters:fileobj – The file like object to read from. Currently this object needs to support read, seek and tell.
Returns:If a complete record is read a Record will be returned, otherwise None will be returned.

If some data is read, but not enough for a whole record, the location of the file object will be reset and a EOFError exception will be raised.

idstools.util module

Module for utility functions that don’t really fit anywhere else.

idstools.util.archive_to_dict(filename, include='*')[source]

Convert an archive file (eg: .tar.gz) to a dict of filenames and their content.

Useful for working with rules.

idstools.util.decode_inet_addr(addr)[source]
idstools.util.md5_hexdigest(filename)[source]

Compute the MD5 checksum for the contents of the provided filename.

Parameters:filename – Filename to computer MD5 checksum of.
Returns:A string representing the hex value of the computed MD5.
idstools.util.mktempdir(delete_on_exit=True)[source]

Create a temporary directory that is removed on exit.

Module contents