idstools.unified2 module

Unified2 record and event reading.

Unified2 is a file format used by the Snort and Suricata IDS engines for logging events.

For more information on the unified2 file format see:

http://manual.snort.org/node44.html

usage: from idstools import unified2

class idstools.unified2.AbstractDecoder(fields)

Bases: object

Base class for decoders.

class idstools.unified2.Aggregator

Bases: object

Deprecated: Applications requiring the aggregation of packets and extra data with an event should implement their own aggregation logic.

A class implementing something like the aggregator pattern to aggregate records until an event can be built.

add(record)

Add a new record to aggregator.

Parameters:record – The decoded unified2 record to add. Returns:If adding a new record allows an event to be completed, an Event will be returned.

flush()

Flush the queue. This converts the records in the queue into an Event.

If using the Aggregator directly, you’ll want to call flush after adding all your records to get the final event.

Returns:An Event or None if there are no records.

class idstools.unified2.Event(event)

Bases: dict

Event represents a unified2 event record with a dict-like interface. The unified2 file format specifies multiple types of event records, idstools normalizes them into a single type.

Fields:

• sensor-id
• event-id
• event-second
• event-microsecond
• signature-id
• generator-id
• signature-revision
• classification-id
• priority
• source-ip
• destination-ip
• sport-itype
• dport-icode
• protocol
• impact-flag
• impact
• blocked
• mpls-label
• vlan-id

Deprecated: Methods that return events rather than single records will also populate the fields packets and extra-data. These fields are lists of the Packet and ExtraData records associated with the event.

class idstools.unified2.EventDecoder(fields)

Bases: idstools.unified2.AbstractDecoder

Decoder for event type records.

decode(buf)

Decodes a buffer into an Event object.

decode_ip(addr)

decode_ip_v3(addr, vers)

class idstools.unified2.ExtraData(*fields, **kwargs)

Bases: dict

ExtraData represents a unified2 extra-data record with a dict like interface.

Fields:

• event-type
• event-length
• sensor-id
• event-id
• event-second
• type
• data-type
• data-length
• data

class idstools.unified2.ExtraDataDecoder(fields)

Bases: idstools.unified2.AbstractDecoder

Decoder for extra data type records.

decode(buf)

Decodes a buffer into an ExtraData object.

class idstools.unified2.Field(name, length, fmt=None)

Bases: object

A class to represent a field in a unified2 record. Used for building the decoders.

fmt

Builds a format string for struct.unpack.

class idstools.unified2.FileEventReader(*files)

Bases: object

Deprecated: Event readers have been deprecated due to the deprecation of the Aggregator.

FileEventReader reads records from one or more filenames and aggregates them into events.

Parameters:files... – One or more files to read events from.

Example:

reader = unified2.FileEventReader("unified2.log.1382627941",
                                  "unified2.log.1382627966)
for event in reader:
    print(event)

next()

Return the next Event or None if EOF.

class idstools.unified2.FileRecordReader(*files)

Bases: object

FileRecordReader reads and decodes unified2 records from one or more files supplied by filename.

Parameters:files... – One or more filenames to read records from.

Example:

reader = unified2.RecordReader("unified2.log.1382627941",
                               "unified2.log.1382627966)
for record in reader:
    print(record)

next()

Return the next record or None if EOF.

Records returned will be one of the types Event, Packet, ExtraData or Unknown if the record is of an unknown type.

tell()

Returns the current filename and offset.

class idstools.unified2.Packet(*fields, **kwargs)

Bases: dict

Packet represents a unified2 packet record with a dict-like interface.

Fields:

• sensor-id
• event-id
• event-second
• packet-second
• packet-microsecond
• linktype
• length
• data

class idstools.unified2.PacketDecoder(fields)

Bases: idstools.unified2.AbstractDecoder

Decoder for packet type records.

decode(buf)

Decodes a buffer into a Packet object.

class idstools.unified2.RecordReader(fileobj)

Bases: object

RecordReader reads and decodes unified2 records from a file-like object.

Parameters:fileobj – The file-like object to read from.

Example:

fileobj = open("/var/log/snort/merged.log.1382627987", "rb")
reader = RecordReader(fileobj):
for record in reader:
    print(record)

next()

Return the next record or None if EOF.

Records returned will be one of the types Event, Packet, ExtraData or Unknown if the record is of an unknown type.

tell()

Get the current offset in the underlying file object.

class idstools.unified2.SpoolEventReader(directory, prefix, follow=False, delete=False, bookmark=False)

Bases: object

Deprecated: Event readers have been deprecated due to the deprecation of the Aggregator.

SpoolEventReader reads records from a unified2 spool directory and aggregates them into events.

Required parameters:

Parameters:

• directory – Path to unified2 spool directory.
• prefix – Filename prefix for unified2 log files.

Optional parameters:

Parameters:

• follow – Set to true to follow the log files. Reading will wait until an event is available before returning.
• delete – If True, unified2 files will be deleted when reading has moved onto the next one.
• bookmark – If True, the reader will remember its location and start reading from the bookmarked location on initialization.

Example:

reader = unified2.SpoolEventReader("/var/log/snort", "unified2.log")
for event in reader:
    print(event)

next()

Return the next Event.

If in follow mode and EOF is head, this method will sleep and and try again.

rollover_hook(closed, opened)

tell()

See SpoolRecordReader.tell().

class idstools.unified2.SpoolRecordReader(directory, prefix, init_filename=None, init_offset=None, follow=False, rollover_hook=None)

Bases: object

SpoolRecordReader reads and decodes records from a unified2 spool directory.

Required parameters:

Parameters:

• directory – Path to unified2 spool directory.
• prefix – Filename prefix for unified2 log files.

Optional parameters:

Parameters:

• init_filename – Filename open on initialization.
• init_offset – Offset to seek to on initialization.
• follow – Set to true if reading should wait for the next record to become available.
• rollover_hook – Function to call on rollover of log file, the first parameter being the filename being closed, the second being the filename being opened.

Example with following and rollover deletion:

def rollover_hook(closed, opened):
    os.unlink(closed)

reader = unified2.SpoolRecordReader("/var/log/snort",
    "unified2.log", rollover_hook = rollover_hook,
    follow = True)
for record in reader:
    print(record)

get_filenames()

Return the filenames (sorted) from the spool directory.

iter_next()

Return the next record or None if EOF.

If in follow mode and EOF, this method will sleep and and try again.

Returns:A record of type Event, Packet, ExtraData or Unknown if the record is of an unknown type.

next()

Return the next decoded unified2 record from the spool directory.

open_file(filename)

open_next()

Open the next available file. If a new file is opened its filename will be returned, otherwise None will be returned.

tell()

Return a tuple containing the filename and offset of the file currently being processed.

class idstools.unified2.Unified2Bookmark(directory=None, prefix=None, filename=None)

Bases: object

Class to represent a “bookmark” for unified2 spool directories.

get()

Get the current bookmark.

Returns a tuple of filename and offset.

update(filename, offset)

Update the bookmark with the given filename and offset.

class idstools.unified2.Unknown(record_type, buf)

Bases: object

Class to represent an unknown record type.

In the unlikely case that a record is of an unknown type, an instance of Unknown will be used to hold the record type and buffer.

exception idstools.unified2.UnknownRecordType(record_type)

Bases: Exception

idstools.unified2.decode_record(record_type, buf)

Decodes a raw record into an object representing the record.

Parameters:

• record_type – The type of record.
• buf – Buffer containing the raw record.

Returns:

The decoded record as a Event, Packet, ExtraData or Unknown if the record is of an unknown type.

idstools.unified2.read_record(fileobj)

Reads a unified2 record from the provided file object.

Parameters:fileobj – The file like object to read from. Currently this object needs to support read, seek and tell. Returns:If a complete record is read a Record will be returned, otherwise None will be returned.

If some data is read, but not enough for a whole record, the location of the file object will be reset and a EOFError exception will be raised.