Unified2 File Reading

idstools provides unified2 readers for reading individual records as well as aggregating records into events.

Reader Objects

Unified2 file reading and decoding is done with a reader objects. Different reader objects exist for where you are reading from and whether you want to read individual records, or have records aggregated into events.

RecordReader

class idstools.unified2.RecordReader(fileobj)[source]

RecordReader reads and decodes unified2 records from a file-like object.

Parameters:fileobj – The file-like object to read from.

Example:

fileobj = open("/var/log/snort/merged.log.1382627987", "rb")
reader = RecordReader(fileobj):
for record in reader:
    print(record)
next()[source]

Return the next record or None if EOF.

Records returned will be one of the types Event, Packet, ExtraData or Unknown if the record is of an unknown type.

tell()[source]

Get the current offset in the underlying file object.

FileRecordReader

class idstools.unified2.FileRecordReader(*files)[source]

FileRecordReader reads and decodes unified2 records from one or more files supplied by filename.

Parameters:files... – One or more filenames to read records from.

Example:

reader = unified2.RecordReader("unified2.log.1382627941",
                               "unified2.log.1382627966)
for record in reader:
    print(record)
next()[source]

Return the next record or None if EOF.

Records returned will be one of the types Event, Packet, ExtraData or Unknown if the record is of an unknown type.

tell()[source]

Returns the current filename and offset.

FileEventReader

class idstools.unified2.FileEventReader(*files)[source]

FileEventReader reads records from one or more filenames and aggregates them into events.

Parameters:files... – One or more files to read events from.

Example:

reader = unified2.FileEventReader("unified2.log.1382627941",
                                  "unified2.log.1382627966)
for event in reader:
    print(event)
next()[source]

Return the next Event or None if EOF.

SpoolRecordReader

class idstools.unified2.SpoolRecordReader(directory, prefix, init_filename=None, init_offset=None, follow=False, rollover_hook=None)[source]

SpoolRecordReader reads and decodes records from a unified2 spool directory.

Required parameters:

Parameters:
  • directory – Path to unified2 spool directory.
  • prefix – Filename prefix for unified2 log files.

Optional parameters:

Parameters:
  • init_filename – Filename open on initialization.
  • init_offset – Offset to seek to on initialization.
  • follow – Set to true if reading should wait for the next record to become available.
  • rollover_hook – Function to call on rollover of log file, the first parameter being the filename being closed, the second being the filename being opened.

Example with following and rollover deletion:

def rollover_hook(closed, opened):
    os.unlink(closed)

reader = unified2.SpoolRecordReader("/var/log/snort",
    "unified2.log", rollover_hook = rollover_hook,
    follow = True)
for record in reader:
    print(record)
next()[source]

Return the next record or None if EOF.

If in follow mode and EOF, this method will sleep and and try again.

Returns:A record of type Event, Packet, ExtraData or Unknown if the record is of an unknown type.
tell()[source]

Return a tuple containing the filename and offset of the file currently being processed.

SpoolEventReader

class idstools.unified2.SpoolEventReader(directory, prefix, follow=False, delete=False, bookmark=False)[source]

SpoolEventReader reads records from a unified2 spool directory and aggregates them into events.

Required parameters:

Parameters:
  • directory – Path to unified2 spool directory.
  • prefix – Filename prefix for unified2 log files.

Optional parameters:

Parameters:
  • follow – Set to true to follow the log files. Reading will wait until an event is available before returning.
  • delete – If True, unified2 files will be deleted when reading has moved onto the next one.
  • bookmark – If True, the reader will remember its location and start reading from the bookmarked location on initialization.

Example:

reader = unified2.SpoolEventReader("/var/log/snort", "unified2.log")
for event in reader:
    print(event)
next()[source]

Return the next Event.

If in follow mode and EOF is head, this method will sleep and and try again.

tell()[source]

See SpoolRecordReader.tell().

Record Types

A Unified2 log file is composed records of different types. A IDS event is composed of multiple records, generally a single Event record followed by one or more Packet records and sometimes one or more ExtraData records.

Record readers like SpoolRecordReader return individual records while event readers like SpoolEventReader return Event records with the associated Packet and ExtraData records as part of the event.

For most purposes the following record types look and feel like a Python dict.

Event

class idstools.unified2.Event(event)[source]

Event represents a unified2 event record with a dict-like interface.

Fields:

  • sensor-id
  • event-id
  • event-second
  • event-microsecond
  • signature-id
  • generator-id
  • signature-revision
  • classification-id
  • priority
  • source-ip
  • destination-ip
  • sport-itype
  • dport-icode
  • protocol
  • impact-flag
  • impact
  • blocked
  • mpls-label
  • vlan-id

Methods that return events rather than single records will also populate the fields packets and extra-data. These fields are lists of the Packet and ExtraData records associated with the event.

Packet

class idstools.unified2.Packet(*fields, **kwargs)[source]

Packet represents a unified2 packet record with a dict-like interface.

Fields:

  • sensor-id
  • event-id
  • event-second
  • packet-second
  • packet-microsecond
  • linktype
  • length
  • data

ExtraData

class idstools.unified2.ExtraData(*fields, **kwargs)[source]

ExtraData represents a unified2 extra-data record with a dict like interface.

Fields:

  • event-type
  • event-length
  • sensor-id
  • event-id
  • event-second
  • type
  • data-type
  • data-length
  • data