[netflow-tools] softflowctl statistics

[Damien Miller]

On Sun, 7 Dec 2008, Michael W. Lucas wrote:

> Hi,
>
> (Please pardon the nitpicking questions. My new book includes a
> section on softflowd. While I've used softflowd happily for years,
> this means that I'm now poking into corners that I've never looked at
> before.)

Excellent!

> I'm taking a close look at "softflowctl statistics," and something
> doesn't seem quite right to me. Here's output from 0.9.8 on a FreeBSD
> 6.4 test box:
>
> softflowd[61946]: Accumulated statistics: Number of active flows: 16
> Packets processed: 11898 Fragments: 0 Ignored packets: 46 (46 non-IP,
> 0 too short) Flows expired: 759 (0 forced) Flows exported: 784 in
> 67 packets (0 failures) Packets received by libpcap: 12156 Packets
> dropped by libpcap: 0 Packets dropped by interface: 3217012028
>
> I would expect "packets processed" + "ignored packets" = "packets
> received by libpcap". This rarely seems to be the case. Does softflowd
> lose packets?

Hopefully not :) The "Packets received|dropped by ..." are statistics
that are provided by libpcap and the accounting there might be slightly
different. In particular, it might be that there are a few packets
sitting in a pcap buffer waiting to be sent to softflowd. It's also
possible that I made a math error somewhere in softflowd.

softflowd can lose packets when it is too busy to process incoming ones.
It is possible to replicate this by running "softflowctl expire-all" in
a tight loop.

> Also, we've expired 759 flows, but exported 784. Where did the extra
> 25 flows come from?

This is some terminology confusion that is entirely my fault: the
"flows" in the "expired" statistic are softflowd's internal flow
representation, which are bidirectional. The "flows" in the "exported"
statistic are NetFlow flows, which are unidirectional. If every flow
had bidirectional traffic the latter should be 2 x the former, but this
is not always the case (e.g. broadcast packets, packets dropped by
firewalls).

> What makes softflowd "force" expiration of a flow? "softflowctl
> expire-all" doesn't seem to increment this counter.

This is when flows are "forced out" before their timeout expires by
new flows. Forcible expiry is performed on LRU (least recently used)
flows. You should be able to demonstrate forcible expiry by setting an
unreasonably low number of maximum flows (-m 16 or somesuch).

> Last, why would an interface on a machine with very little traffic
> show 3217012028 packets dropped by the interface?

That value comes from libpcap, but it looks like it was either
uninitialised or has underflowed an unsigned integer.

-d

PS. sorry for the slow reply - December was a mad month...