[netflow-tools] Announce: flowd-0.9

[Damien Miller]

Hi,

This is to announce a new release of the flowd NetFlow collector,       
flowd-0.9. This release includes some major functionality and           
performance improvements. Please note that this release also changes    
the on-disk flow log format (it is possible to convert old flowd logs   
to the new format using the flowd-reader tool).                         

The new version is available from http://www.mindrot.org/flowd.html

SHA1 (flowd-0.9.tar.gz) = 54cb8ecaaa36d2f25105156170680842123b965d

Thanks to the many people who suggested improvements, reported bugs and
tested patches or snapshots.

Some of the higlights of this release are:

Storage format
--------------

The on-disk storage format has been improved and given a new major
number (version 3). The new format is faster to read from disk and
will be more graceful to extend in the future. In particular, it is
possible to add fields to this new format whilst retaining backwards
compatibility.

To convert logs from previous versions of flowd, please use
flowd-reader's "-L" option. For example, "flowd-reader -Lqo new.log
old.log" will convert the flows in "old.log" to the new format and store
them in "new.log".

The new format also supports some additional fields and extends the
widths of some existing ones:

  - Receive time is now recoded as seconds and microseconds
  - Add Netflow V.9 source_id field
  - Interface indices (if_ndx_in/out), engine_type, engine_id and the 
    source and destination AS numbers are now 32-bits wide

Performance enhancements
------------------------

Flowd has had several improvements to improve its performance on busy
networks or when confronted with sudden bursts of flows. The improvements
are:

   - Addition of a basic input queue to flowd, so it is now able to cluster
     network packets reads
   - Addition of an output buffer, so every flow received doesn't end up as a
     tiny write to the filesystem
   - Increased the UDP socket receive buffer size, so the kernel can buffer 
     more flow packets during bursts
   - Shrink the UDP socket send buffer size on the listening socket, because
     flowd never sends packets from it

Python API
----------

The flowd Python API has been rewritten in 100% C, improving its 
performance by an order of magnitude. This rewrite changes the API and
adds several new interfaces, including an iterator interface to flow logs.
Reading every flow in a flow log is now as simple as:

    import flowd

    flow_log = flowd.FlowLog("flows.log", "rb")
    for flow in flow_log:
            print flow.format()

Other additions to the Python API include the addition of a Flow.has_field() 
function as well as interval_time() and iso_time() time conversion functions

flowd-reader
------------

Added the ability to read and convert flow logs from pre-0.9 versions   
of flowd ("flowd-reader -L").                                           

Changed flowd-reader's output ("-o") behaviour to *overwrite* existing
log files instead of *appending* to them. Please take care if you were
depending on the previous behaviour.

Added a "head" mode to flowd-reader. E.g. "flowd-reader -H 1024
flowd.log" will display only the first 1024 flow records.

Live flow reporting
-------------------

Addes support for relaying serialised flows to a local Unix domain
datagram socket in realtime. This allows real-time monitoring of flows
by a tiny application. A sample Python client that receives and prints
the flows is provided as tools/sockclient.py.

Please see the "logsock" option in flowd.conf(5) for more details. Note
that this option is still considered experimental.

Flow Filtering
--------------

It is now possible to filter by date/time. For example:

    accept tag 1 after date 200601010000 before date 200601072359

Other fixes and improvements
----------------------------

Added RPM spec and init files for SuSE Linux contributed by alshu AT
tut.by

Allow reading from standard input in samply Python statistics program
(tools/stats.py)

Fix byte swapping of src/dst AS and interface indices, spotted and fix
tested by Gijs Molenaar

Fix bug that broke filtering on address family in flows, spotted by Gijs
Molenaar

Support devices that send multiple templates in a single template
packet section. Thanks again to Gijs Molenaar for packet dumps that
demonstrated this